NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding
Authors: Ming Hu
Lin Wang
Siyuan Yan
Don Ma
Qingli Ren
Peng Xia
Wei Feng
Peibo Duan
Lie Ju
Zongyuan Ge
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Activity understanding; Action recognition; Nursing procedure; Standard level grading
TL;DR: NurViD, a large video dataset with expert-level annotation for nursing procedure activity understanding.

Abstract: The application of deep learning to nursing procedure activity understanding has the potential to greatly enhance the quality and safety of nurse-patient interactions. By utilizing the technique, we can facilitate training and education, improve quality control, and enable operational compliance monitoring. However, the development of automatic recognition systems in this field is currently hindered by the scarcity of appropriately labeled datasets. The existing video datasets pose several limitations: 1) these datasets are small-scale in size to support comprehensive investigations of nursing activity; 2) they primarily focus on single procedures, lacking expert-level annotations for various nursing procedures and action steps; and 3) they lack temporally localized annotations, which prevents the effective localization of targeted actions within longer video sequences. To mitigate these limitations, we propose NurViD, a large video dataset with expert-level annotation for nursing procedure activity understanding. NurViD consists of over 1.5k videos totaling 144 hours, making it approximately four times longer than the existing largest nursing activity datasets. Notably, it encompasses 51 distinct nursing procedures and 177 action steps, providing a much more comprehensive coverage compared to existing datasets that primarily focus on limited procedures. To evaluate the efficacy of current deep learning methods on nursing activity understanding, we establish three benchmarks on NurViD: procedure recognition on untrimmed videos, procedure and action recognition on trimmed videos, and action detection. Our benchmark and code will be available at https://github.com/minghu0830/NurViD-benchmark.

NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding
Ming Hu 1233+ Lin Wang 1224+
Siyuan Yan 1,3+ Don Ma
Qingli Ren Peng Xia3.2.3 Wei Feng1.2,5
Peibo Duan Lie 1.23 Zongyuan Ge1.2,3
AIM Lab, Faculty of IT, Monash University, Australia AArdoc--ooaan Monash Research, Airdoc, China Faculty of Engineering, Monash University, Australia
*College of Intelligent Systems Science and Engineering, Harbin Engineering University, China NNussnng College, Shanxi Medical University, China
Abstract
The application of deep learning nursing procedure activity understanding has the potential to greatly enhance the quality and safety of nurse-patient interactions. By utilizing the technique, we can facilitate training and education, improve quality control, and enable operational compliance monitoring. However, the development of automatic recognition systems in this field is currently hindered by the scarcity of appropriately labeled datasets. The existing video datasets pose several limitations: these datasets are small-scale in size support comprehensive investigations of nursing activity; 2) they primarily focus single procedures, lacking expert- level annotations for various nursing procedures and action steps; and 3) they lack temporally localized annotations, which prevents the effective localization targeted actions within longer video sequences. To mitigate these limitations, we propose NurViD, a large video dataset with expert-level annotation for nursing procedure activity understanding. NurViD consists of over 1.5k videos totaling 144 hours, making it approximately four times longer than the existing largest nursing activity datasets. Notably, it encompasses 51 distinct nursing procedures and 177 action steps, providing much more comprehensive coverage compared to existing datasets that primarily focus on limited procedures. To evaluate the efficacy current deep learning methods nursing activity understanding, benchmarks on ViD: procedure recognition on untrimmed videos, procedure and action recognition trimmed videos, and action detection. Our benchmark and code will be available at https:// github. https::/gitluu.com//
establish three
Introduction
The application of deep learning (DL) in understanding nursing procedure activities has the potential to greatly enhance the quality of nurse-patient interactions, while also playing crucial role in preventing medical disputes, minimizing missed nursing procedures, and reducing nursing errors [32, 45, 35, 21, 11]. DL-based automatic approaches offer several key benefits for nurses, including: (1) Reliable: it enables objective, precise, and consistent assessments of nursing skills, eliminating the subjectivity inherent in traditional evaluations by experts [27]. (2) Real-time guidance: it facilitates immediate feedback on nurses' performance, empowering them to identify areas for improvement in a
timely manner [30]. (3) Cost-effective: it can alleviate the burden of manual observations, evaluations.
and training by experts [39].
Equal Contribution
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

However, the development of automatic nursing recognition systems is currently hindered by the absence of suitably labeled datasets for nursing activities. Existing public video datasets for action recognition primarily focus on generic daily activities or specific sports, with minimal attention given to nursing or healthcare scenarios [23, 13, 36, 16, 22, 24, 17]. For example, the Kinetics 700 dataset [23] with 700 category labels includes only two labels related to nursing activities. Although some initiatives have attempted to create nursing activity understanding video datasets (e.g.. hundwashing-specitfe datasets [15, 7, 48, 42, 4, 26]). limitations arise due to the gap between them and real-world clinical settings, which is summarized as follows: (1) Limited procedure and action variery: Expensive annotation cost causes existing datasets only have single nursing procedure, whereas real clinical environments involve a wide range of complex procedures. (2) Simple scenes only: The recorded videos typically captured in controlled settings such instructional or laboratory environments, which do not accurately reflect the complexity and variability of nursing procedures in actual clinical practice. (3) Un- tn-professional laheling: They suffer from non-professional labeling or a lack of adherence to standard guidelines, leading to errors or inconsistencies. video sequence: They primarily consist of short action clips, which do not facilitate understanding long-term activities and their context.
(4) Short
To mitigate these limitations, we proposed NurViD, large-scale video benchmark for nursing procedure activity understanding. Compared to existing datasets, Vur incorporates characteristics from the following aspects: (1) Diverse procedure and action: NurViD comprises 144 hours of annotated videos, which is approximately four times longer than the largest existing nursing activity datasets. It also contains 1,538 videos depicting 51 nursing procedure categories, covering the majority of common procedures, along with 177 action steps, providing much more comprehensive coverage, compared to previous datasets that primarily focus single procedures with limited action steps. (2) Real-world clinic settings: Videos in Vur were captured from over ten real clinical environments according to our statistics, including hospitals, clinics, and nursing homes. This diverse range of settings ensures that the models trained on VurViD are applicable in real-world clinical scenarios. (3) Expert-level annotations: ViD was labeled by professionals with high expertise and knowledge in nursing. The procedure and action annotation process follows the guideline of Training Outline for Newly Employed Nurses issued by the National Health Conmission of China [2], ensuring consistency and accuracy of the annotations, (4) Support multiple recognition and detection tasks: We have established two different classification tasks and an action temporal localization benchmark specifically targeting the long-tail distribution of the dataset.
We further compare VuD dataset with the other existing nursing activity video datasets and summarize the key difference in Table 1. The contributions of Nur ViD are summarized as follows:
NurViD is the most diverse video benchmark to date for nursing procedure activity understanding tasks. It has been meticulously annotated by nursing professionals, and the annotation process follows standardized nursing procedure guidelines or protocols. NurViD exhibits competitive video size and much more comprehensive coverage of procedures and action categories compared to existing datasets.
In response to the practical needs of nursing and machine-learning communities, such as education and training, automatic action detection, and long-tail distribution, we establish three different recognition and localization benchmarks on NurViD.
Leng-term retention and availability of dataset. NurViD has been sourced from YouTube and follows the CC BY 4.0 license agreement [1].
2
Related Work
Research aimed at enabling machines to understand human behavior and activities has led to advance- ments in various practical applications. However, building such systems comes with challenges that require appropriate datasets for training and evaluation. In recent years, numerous datasets have been created to support research in human behavior understanding [34, 13, 49, 36]. While these datasets have been valuable for general activity recognition, only a limited number cater to the specific needs of nursing professionals.
Sequential action prediction. Standardized datasets have been meticulously designed for the purpose of evaluating the performance of algorithms in understanding and accurately recognizing intricate

---

Dutuset Properties
Tasks
Datasets
Puhlicty
Available"
Expert- Level
Annotationss
Settings.
No. of Videos
Segmens
Procedures
No. of Actions
Total Duration
Procedure Recognition
Recognition
Detoction
No. of
Action
Action
Real
No.
Llorca et al. [15]
XXXXX
8
7
4.2min
X
Ameling et al. [7]
24
XXXXXX
Zhong et [48]
1.400
X
Wang [42]
280
2,760
8
Kaggle [4]
292
3,504
12
23.3h
Lulla et al. [26]
3,185
6,689
38
NurViD (Ours)
The comparison among existing nursing procedure activity video datasets. Compared to other datasets, NurViD annotates the procedures and actions by following expert-level standards, focuses on more comprehensive coverage of various nursing procedure categories, collects large number of videos, totaling 144 hours, and also enables action detection tasks.
Table I:
activities within real-world scenarios [22, 16]. These datasets are used in various fields, including computer vision, robotics, natural language processing, and surveillance systems. Focusing on specific sets of actions carried out in well-defined order, sequential action understanding datasets differ from those that encompass a broader range contexts [49]. many fields, it is essential to adhere to strict sequence of steps to ensure optimal results. For example, in the medical field, following specific order of steps is crucial in procedures such as administering medication, where any deviation from the established sequence could result in serious consequences for the patient. Standardized datasets can help ensure that the actions performed in real-world situations are accurately represented and provide benchmark for evaluating the performance of algorithms.
Nursing procedure video dataset. Online learning has gained significant popularity, particularly through the utilization of instructional videos that provide step-hy-step guidance, serving valuable resources for teaching and learning specific tasks. Within the medical field, instructional videos have proven to be highly effective in conveying essential information using visual and verbal communica- tion, thus benefiting learners [17]. On the other side, the absence of comprehensive and standardized nursing procedure video datasets presents challenge in developing effective algorithms within the healthcare industry. The current datasets suffer from limited coverage, focusing only on nursing procedures relevant specific healthcare settings patient populations. Additionally, the quality of annotation plays a critical role in algorithm accuracy. The process of annotation involves identifying and labeling specifie actions and events in the videos, which be time-consuming and challenging task. Annotation errors may arise due to factors such as human error, task ambiguity, or the absence of standardized protocols.
3 Building NurViD Dataset
In this section, describe the process of building NurViD, from selecting the nursing procedures to acquiring, filtering, and annotating the video data. We leveraged the extensive collection of medical instructional videos available on YouTube [6] and carefully selected and filtered our video collection to ensure the quality and relevance of the dataset. We also developed standardized labeling scheme for the actions performed in each video, providing a valuable resource for developing and evaluating algorithms that recognize and understand nursing procedures.
3.1 Procedure and Action Definition
The selection of nursing procedures and corresponding action steps is crucial for maintaining the relevance and usefulness of ViD within the nursing profession and the broader healthcare commu- nity. The procedure selection adheres to widely accepted nursing taxonomy, taking into account frequency of use and expert guidance. Then the actions involved in these procedures are gathered and standardized.
3

---

Start:
08:15
1310
check
landwasiing
blood sumple
hundwashing
dicunent
solect vdD
skindectskin
veliunccuuee
Rb-b
80-35
Wbch
12.35
check
messure the lentt of PICT catheter
cleck and warree willdran taling Intraduove shrath
didndoc
blish starlic 20ne
PICC Insetion
Figure I: The examples for the annotated target action boundaries for Intravenous Blood Sampling and Modified Seldinger Technique with Ultrasound for PICC Placement procedures. The frames marked in colored boxes denote the annotated temporal boundaries for the target action steps.
Procedure selection. We compiled various nursing procedures from Nurselabs website [3] and nursing procedure books [8, 29, 31, 20]. With expert consultation, identified 51 commonly employed nursing procedures that align with the requirements of most nursing scenarios. To validate their relevance and prevalence, searched for corresponding videos YouTube. Table 5 provides the full names and summarized abbreviations of these 51 procedures.
Action definition. We developed action steps for various nursing procedures based college tutorials, and a nursing lecturer summarized the appropriate action labels by analyzing the action descriptions and video content. This necessary for three main reasons: (1) There variations in actions performed during specific nursing procedures, even within patients or instances of the same procedure, that require accurate representation of nuances. (2) The existence of diverse nursing procedure standards across countries and regions highlights the importance of establishing a unified standard. (3) Dealing with fine-grained video data from real-world nursing procedures requires rearranging action tags for the precise depiction of procedure nuances.
3.2 Online Video Crawling
Our objective in this stage was to gather sufficient videos demonstrating the pre-selected nurs- ing procedures. By utilizing the extensive collection of medical instructional videos available on YouTube [6, 17], we acquire wide range of videos without the need for third-party video production. To accomplish this, we queried YouTube using text-based searches for each procedure and obtained videos whose titles included the desired procedure keywords. To expand the video collection, we enhanced the search queries by including synonyms of each procedure. For example, Subcutaneous Injection Insulin can also be called Subcutaneous Insulin Administration, Subcutaneous Insulin Therapy, or abbreviated as SCII. Each video was downloaded at the highest resolution available. During video retrieval, we prioritized videos shorter than 20 minutes to limit the total storage.
3.3 Localization Annotation and Quality Control
In the NurViD dataset, each video is divided into multiple temporal segments, each of which contains only one action. Each action annotated with its starting and ending timestamps as well as its frame position in the video. The annotation process is performed by undergraduates with medical and nursing backgrounds to ensure the accuracy and consistency of the annotations.
Employing nursing professionals. Data curation is an expensive process that typically involves extensive manual annotation. some cases, datasets have employed a semi-automatic crowdsourcing approach for collection and annotation [12, 18, 40, 13]. For tasks that require greater reliability,

---

No. of Segments
15"
10
PC
PRC
ADM
18h
sciu
Cs-PPS
CSPOR-
OC-UT
POIN-
NiT
VSA
MN
SF
MRR MARU-WIC iice
OWD
PST
JRVCU
1M0m)
CSET
NGG
ABS
sctog-laa ONS- ESD
NCM
DDIG
sayc
ONS DNS--CN
ON
AT
CP3- PER-000
DV
IFF
(VV)
CEM
USC
1.38
W777
13C
Procedure
Figure 2: The average, maximum, and minimum number of action segments for each procedure.
1200
100
1000
Count
80
Count
800
600
400
200
200
400
600
800
1000
50
100
150
200
Vidco Duration (seconds)
Segment Length (seconds)
(a) Video Duration Distribution
(b) Segment Duration Distribution
Figure 3: NurViD dataset duration statistics.
certain datasets rely on domain experts for annotation, albeit at a higher cost. In our study, we formed medical team 26 individuals, consisting of nursing lecturer and 25 nursing majors from medical college, to perform labeling. Over half of the students have at least three years of undergraduate education, possess extensive practical experience nursing procedures, and have successfully completed the university's s standardized nursing procedure assessment.
Invalid video filtering. Additionally, certain videos may contain irrelevant content due to inaccuracies in text-based retrieval. Thus, in the first stage labeling, we excluded videos that fall into the following categories: 1) showcasing unrealistic environments (e.g., movies, animation). 2) providing only verbal descriptions instead of visual demonstrations, 3) featuring static images instead of continuous videos, and 4) lacking the specified procedure.
Action boundaries annotation. To ensure localization annotation quality, we followed a three-round annotation process: (1) Each annotator and tasked with filtering out inappropriate videos, (2) After filtering, the action segments of each procedure video were annotated by three members, (3) Finally, cross-checking of annotation results between every two groups was conducted to identify and rectify errors and omissions. This process resulted in minimum of three annotated action boundaries for each video. To ensure reliable annotations, we employed the complete linkage algorithm [10] to cluster and merge various temporal boundaries into stable boundaries that received multiple agreements. It is important mention that single video may feature multiple separate instances of the target action, leading to multiple boundary definitions. Several examples of annotated target action boundaries shown in Figure.
assigned 2-3 nursing procedures based video count
3.4 VurViD Statisties
Our NurViD dataset a comprehensive collection of nursing procedure videos that includes 1,538 videos (144 hours) demonstrating 51 different nursing procedures and 177 actions for recognition and detection tasks, which are summarized Table. 5 and Table. in the supplementary material. To facilitate the development and evaluation of algorithms, we trimmed the videas based annotated action boundaries, resulting in 5,608 trimmed video instances that totaled hours. The trimmed videos have average duration of 32 seconds, while the untrimmed videos have an average duration of 337 seconds. Over 74% videos have HD resolutions 1280 x 720 pixels or higher. We observed long-tailed distribution in the number of collected videos for both procedure and action.
5

---

Experimental Results
In our study, we focus three tasks using the NurViD dataset: (1) procedure classification in untrimmed videos, (2) procedure and action classification on trimmed videos, and (3) action detection on untrimmed videos. To establish reliable baselines for these classification and detection tasks, employ state-of-the-arr models that have demonstrated effectiveness in human action recognition and detection. We provide a comprehensive analysis of the baseline models, taking into consideration the specific challenges posed by the VurViD dataset, such as long-tailed class distributions. For each task, we formulate the problem in detail and evaluate the performance of the baseline models. The results of our analysis can guide the development of more accurate and robust models for fine-grained action recognition and detection in healthcare applications.
4.1 Procedure Classification on Untrimmed Videos
This task involves identifying nursing procedures from untrimmed videos, which typically consist of multiple standardized action steps that must be executed in a specifie order and other unrelated parts. The goal is to explore the effectiveness of DL technology in retrieving specific nursing procedures from large video library. By accurately classifying nursing procedures, can provide healthcare professionals with powerful tool for quickly accessing relevant videos.
Data settings. The examples from each procedure category are randomly divided into three sets: 70% for training. 10% for validation, and 20% for testing, resulting in 1,054 training, 173 validation, and 311 testing videos, respectively.
Procedure Classification Medium
Baselines
Many
Few
All
10
22
18
50
SlowFast [14]
9.9
7.5
0.1
7.4
C3D [38]
10.7
5.1
1.8
7.7
Class splits. To account for the long- tailed nature [47, 43] of the NurViD dataset, we divided the procedure classes into three splits: many, medium, and few, based the number of videos for each procedure. The accuracy of each split the average accuracy of the included procedures within that split. Specifically, the many category includes the top 19.6% most frequent classes, the medium cate- gory includes the middle 43. 1% classes, and the few category includes the remaining 37.3% classes. The number of classes per split is presented in Table 2.
I3D [9]
9.9
9.0
2.8
8.7
SlowFast*
19.9
10.2
5.0
13.5
C
21.5
11.3
5.8
14.8
I3D*
19.8
12.5
5.6
13.1
Table 2: Per-class Top-1 accuracy for procedure prediction on untrimmed videos. The best performance for each split has been highlighted in bold.
We compare the performance of SlowFast [14], I3D 191, and C3D [38] models on this task. These models evaluated in two versions: 1) Random initialization training and 2) Pre-training with weights from Kinetics 400 [23], which a human action recognition dataset.
Baselines.
Results.
The results for per-class accuracy are summarized in Table 2. We find that C3D [38] is able to achieve competitive results for all the splits. However, the best top-1 per-class accuracy is 14.8% indicating that there is significant room for improvement in this challenging task. We also observe that transfer learning from the model that is pre-trained on Kinetics 400 [23] improves the classification accuracy for all splits. With the C3D model, this corresponds to a per-class accuracy gain from 10.7% to 21.5% for the many category and from 5.1% to 11.3% for the medium category. Despite this improvement, accurately predicting procedure categories remains significant challenge.
Discussions. Based on the results of the classification benchmarks established on NurViD, we found that even models pre-trained on Kinetics 400 [23] cannot achieve satisfactory classification performance. We speculate that this may be due to several reasons: (1) videos are not always exclusively focused on nursing procedure activities and may contain other unrelated content, such as verbal instructions and brief introductions to devices; (2) some actions, such as handwashing, disinfection, and document, are commonly used in various procedures, which may cause the model to obtain similar features in different procedural videos, making it difficult to classify them. Overall, focusing on the main procedure actions in the video remains challenging task.

---

Procedure Classification
Action Classification Medium
Joint Classification Medium
Baselines
Many
Medium
Few
All
Many
Few
All
Many
Few
All
13
21
17
51
9
66
87
162
17
78
224
302
SlowFast [14)
68.9
50.0
33.0
63.0
25.7
10.2
3.2
17.1
12.5
7.2
3.3
7.5
C3D [38]
70.1
48.8
33,0
63.9
22.9
9.3
2.9
15.9
13.8
7.3
3.5
7.7
B3D [9]
67.6
49.9
32,9
62.9
26.3
9.8
4.1
17.9
12.7
7.9
4.0
7.9
SlowFast*
71.2
61.8
39.0
68.9
29.8
15.5
7.9
21.1
21.2
9.4
5.6
12.8
C3D*
73.2
60.0
39.6
71.2
28.1
14.6
7.3
22.8
21.8
10.8
5.6
13.1
L3D*
70.7
60.4
40.9
70.0
31.3
14.8
8.2
21.5
19.5
9.9
4.7
12.5
Table 3: Per-class Top-1 accuracy (%) for the procedure, action, and their joint prediction on trimmed videos. denotes the initialization from the model pre-trained on Kinetics 400 [23]. The best performance for each split has been highlighted in bold.
4.2 Procedure and Action Classification Trimmed Videas
This task focuses on classifying both the primary action that occurs in a trimmed video and their associated procedures. By accurately classifying procedures and actions, building automated systems can automate the monitoring of each step in the nursing process, thereby helping identify potentially missed diagnoses, nursing errors, and other issues. These systems can also help doctors and nurses quickly find useful nursing procedure videos, thereby improving their learning and work efficiency.
Data settings. The dataset was randomly partitioned, ensuring a balanced representation of examples from each procedure and action category. Specifically, we allocated 70% of the data for training (3,906 videos), 10% for validation (587 videos), and 20% for testing (1, 12, videos).
Class splits. In this study, also explore the long- tailed nature of the NurViD. We partitioned the videos into three subsets based on the task task-specific distribution: many, medium, and few. For procedure categories, the breakdown is as follows: many: top 26% frequent classes, medium: middle 41% % classes, and few: the remaining 33% classes. For action categories, many: top 5% frequent classes, medium: middle 37% and few: the remaining 58% classes. In addition to some unique actions, such as establish a sterile zone that only exists in the Modified Seldinger Technique with Ultrasound for PICC Placement procedure, hand washing, skin disinfection, and other steps common. Therefore, we established a joint classification task to explore the mutual influence between procedures and actions. For joint classification, many: top 5% frequent classes, mediam: middle 21 % flasses, the remaining 74% classes. We show the number of classes per each split in Table 3.
Baselines. We compare the performance of three models, SlowFast [14]. I3D [9], and C3D [38] on our tasks. These models are originally designed for human action recognition and lack the inherent ability predict both a procedure and action. To align with our joint prediction need, we introduce two task heads dedicated to procedure category recognition and action recognition, respectively. Consequently, compute the joint loss, Ljoint, handle these tasks. Hlyper-paramed using the validation data, and the detailed hyper- settings for each model can be found in the 1
tuned
supplementary material. The loss is defined as Ljoint M Ei vf" log(p?) ~ Σÿ= vf log(p)) where M is the number of procedure classes, N is the number of action classes, P. and vf denote the procedure prediction probability and ground truth label for the category i, pf and yf denote the action prediction probability and ground truth label for the category
Results. The results are summarized Table 3. All models perform well in the procedure classifi- cation task, with C3D[38] achieving a top-1 per-class accuracy of 71.2% across all splits. C3D[38] also demonstrates competitive performance in action and joint classification for all splits, with the best top-1 per-class accuracy of 22.8% and 13.1%, respectively. Transfer learning from a pre-trained model on Kinetics 400 [23] further improves the accuracy of procedure and action classification. For instance, using the C3D model, the per-class accuracy increases from 70.1% % to 73.2% for many classification, from 48.8% to 60.0% for medium classification, and from 33.0% to 39.6% for few classification.
Discussions. The experimental results indicate that the performance of procedure classification on trimmed videos is significantly better than on untrimmed videos, which may confirm that irrelevant motion information has been filtered out in the trimmed videos, and the model more likely to learn motion features. However, The imbalance in class frequencies poses difficulties in achieving
7

---

Acdon
Action
Actiun
Accion
Onset
Offict
Onset
Offset
Figure 4: Visualization of action detection results. From top to bottom: (1) input video frames; (2) action scores at each time step: (3) histogram of action onsets and offsets computed by weighting the regression outputs using action scores.
satisfactory performance for those few classes. To alleviate it, further model design could incorporate long-tail learning techniques such as re-weighting re-sampling techniques. These approaches can help mitigate the impact of class imbalance and improve the model's ability to generalize and classify the underrepresented classes more effectively.
4.3 Action Detection on Untrimmed Videos
The objective of this task is accurately identify actions in untrimmed videos by determining the temporal extent of the main activity. To establish a benchmark for this task, we adopt three baseline models [46, 28, 33] that have shown effectiveness in temporal action localization [13, 46]. The evaluation metric used is the mean Average Precision (mAP), calculated at various temporal Intersection Union (tloU) thresholds [0.5:0.1:0.9]. Additionally, we provide the average mAP across different tloUs.
Data settings. Following previous approaches for the standard procedure and action classification, adhere to split ratio of [train: 0.7, val: 0.1, test: 0.2] to divide the untrimmed videos at the procedure- action composition level. Consequently, have 1,077 videos for training, 153 for validation, and 308 for testing.
Baselines. To evaluate the performance of our datasets, we utilize baseline models, namely Action- Former [46], TAGS [28], and TriDet [33]. order to generate features for the NurViD videos, we fine-tune two-stream I3D model (I3D [9]) initially pre-trained on ImageNet [12]. Subsequently, we extract RGB and optical flow features for each video and concatenate them as the model input.
Results. We show the action detection results in Table 4. Among the baselines, ActionFormer [46] achieves the highest performance, with an average mAP of 23.9% and an mAP of 32.9% for the threshold of 0.5.
mAP (%)
Baselines
0.5
0.6
0.7
0.8
0.9
Avg.
TriDet [33]
30.3
26.7
24.3
20.1
10.7
20.8
TAGS [28]
31.4
26.5
22.6
19.2
11.5
22.4
ActionFormer [46]
32.9
29.6
25.8
20.8
12.7
23.9
Discussions. The outputs of the Action- Former [46] model visualized in Fig- ure. 4. These outputs consist of action scores and regression results, which are weighted by the action scores and pre- sented weighted histogram. Nursing actions, unlike actions in natural datasets, involve precise and meticulous movements rather than large-scale motion from frame frame. There- fore, more detailed action representation is necessary to accurately describe and learn nursing actions. For instance, in the hand wash procedure recommended by the WHO [S], it is crucial to consider small changes in hand position and environmental factors. Since these movements are extremely subtle, algorithms capable of handling finer granularity can potentially detect these subtle motion changes.
Table 4: The results of action detection. We report mAP the loU thresholds of [0.5:0.1:0.9]. The average mAP is calculated by averaging the mAP scores across various tloU thresholds.

---

5 Limitations
We believe that NurViD is beneficial for advancing the development of Al technology in the nursing field. However, we must also consider the potential risks and ímpacts that may arise from anticipated or foreseeable applications. Additionally, VuDV is downloaded from diverse YouTube channels, video quality, production style, and regional nursing practice differences can introduce biases in nursing procedure representation.
onteeele/Foreseeable Uses. After reviewing relevant literature and discussing with nursing pro- fessors and students, we have noticed two main challenges in nursing training and learning: (1) the issue of imprecise recommendations encountered by students when searching for learning videos. For example, when we want learn about Intravenous Injection procedure videos on YouTube, due to the limitations of the recommendation algorithm, the website may recommend Intravenous Blood Sampling procedure videos to some extent, which is quite common. However, the system trained on our dataset can provide more accurate classification recommendations for searches related to nursing operations. Additionally, since NurViD includes temporal localization annotations for actions, it means that we can dynamically adjust the playback range based on specific interests, for instance, if only need to watch the step of skin disinfection and not the other steps, I just need to enter the text "skin disinfection," and the system will automatically locate the specific segment for me. This undoubtedly saves some unnecessary time wastage, and (2) in China alone, there are over 200,000 undergraduate nursing students each year, and this number continues to grow. Each student is required to pass a professional nursing skills examination and undergo approximately 1000 to 2000 hours of practical training. However, currently, students still heavily rely on experienced teachers for real-time supervision and feedback during training, which requires a significant amount of human resources and time investment. (3) Real-time monitoring: NurViD is dataset that leans more towards general models and is currently primarily used for testing and improving deep learning models. It aims assist students' learning and training by recording, monitoring, and providing feedback during their practice sessions, reducing the need for teaching resources, and facilitating event documentation. In this mode, even rough feedback can save significant amount of costs. However, AI systems cannot guarantee absolute accuracy, which means that detection errors or omissions may occur. Therefore, is strongly recommended that any system built upon VirV or similar technologies clearly communicate their limitations and potential errors from the outset and appropriately incorporate human assistance during usage.
Potential Privacy. In human-action understanding of video datasets, it is often inevitable to encounter faces. Therefore, we provide script that uses OpenCV's Haar classifier to detect the facial regions in videos and blur them. We will implement more advanced methods to further alleviate the privacy problem in the future.
Employment Risks. The employment risks associated with nursing action recognition systems can involve the following aspects: (1) Reduced Workforce Demand: The integration of nursing action recognition systems may lead to a diminished need for human nursing professionals. The incorpo- ration of automation technology can handle routine nursing tasks, thereby potentially decreasing the reliance on human caregivers. Consequently, this could result in certain nursing professionals facing job scarcity or encountering uncertainty in their employment prospects. (2) Shift in Skill Requirements: Nonetheless, our perspective is that the future implementation of nursing action recognition systems will likely require nursing professionals to acquire new skills and knowledge to adeptly interact with the technology rather than being replaced by it. This could involve gaining proficiency in various areas, such effectively engaging with the system, accurately interpreting its outputs, and promptly addressing any discrepancies that arise. Nursing professionals encoun- tering challenges in adapting to these evolving technological demands might find it necessary to undergo retraining efforts. (3) Technical malfunctions and misidentification risks: Nursing action recognition systems come with inherent risks of technical glitches and misidentifications. Inaceu- racies or erroneous assessments made by the system could lead to misguided nursing judgments or actions. This potentially jeopardizes patient safety and well- well-being, obliging nursing professionals to dedicate extra time and effort toward rectifying system errors. To mitigate the employment risks linked to nursing action recognition systems, a comprehensive approach is crucial: (1) Workforce Enhancement Programs: Offering programs for upskilling and transitioning is vital to empower nursing professionals with the competencies needed to navigate evolving roles in tandem with the technology. (2) Ethical Guidelines and Standards: Establishing clear ethical guidelines ensures
9

---

responsible and morally sound utilization of nursing action recognition systems within healthcare settings. (3) Data Privacy and Security: Implementing robust measures to safeguard patient data and privacy is paramount to engender trust in the system's operation. (4) Human Oversight and Decision-making Maintaining human oversight ensures that critical nursing decisions are grounded in human judgment and understanding, acting as a safeguard against erroneous system outputs. (5) Continuous System Evaluation and Enhancement: Regularly evaluating and enhancing the system's performance is pivotal to addressing any technical shortcomings and refining its accuracy. (6) Stake- holder Engagement and Collaboration: Fostering engagement and collaboration among various stakeholders, including nursing professionals, technologists, and polieymakers, promotes holistic approach to system development and implementation.
Issues. It is important to acknowledge that while AI systems can assist in detecting standardized nursing procedures and actions, they are not infallible and cannot achieve perfect accuracy. Human supervision and oversight are indispensable in ensuring patient safety and quality care. Therefore, it highly recommended that any system developed based on NurViD or similar technologies explicitly state their limitations and potential errors upfront. This can be achieved by providing clear disclaimers, agreements, or warnings to users, emphasizing the need for human involvement, critical thinking, and professional judgment when interpreting and acting upon system outputs. By transparently communicating the system's 's limitations, healthcare professionals can make informed decisions and use the technology supportive tool rather than relying solely on its outputs.
Contestabblity/Exxllinaaaa
Potential Regional Biases. Standardization poses a significant challenge due to the diverse origins of the videos and the variation in nursing procedure guidelines across countries. VurVil) aims to cover almost all common action labels, providing flexibility for different regions to adopt their own standards based on it. However, despite our efforts, it is important to acknowledge that complete avoidance of bias is challenging. Additionally, the comprehensiveness of the dataset may be influenced by the sources and origins of the videos. The video collection process for ViD may inadvertently introduce biases towards certain regions or healthcare settings. This bias can limit the generalizability of the dataset to a broader context, as it may not fully capture the diverse range of nursing procedures and actions practiced worldwide. Addressing this limitation requires ongoing efforts collect data from diverse regions, collaborate with experts from different backgrounds, and ensure a balanced representation of nursing practices from various healthcare contexts. Furthermore, domain adaptation models may also be one of the potential solutions [41, 44, 43].
Comprehensiveness of Nursing Procedures and Actions. While efforts were made to include a wide range of common action labels, it is important to acknowledge that the dataset may not cover every possible nursing procedure or action. Variations in nursing practices and guidelines across different regions and healthcare systems can result in some actions being omitted or not adequately represented in the dataset. Furthermore, the dataset's composition may be influenced by the availability and accessibility of vidcos from different regions. Certain nursing procedures or actions that more prevalent or emphasized in specific regions may be overrepresented, while others may be underrepresented. Contimuously expanding the dataset's 's coverage through collaboration with experts and professionals from diverse backgrounds can help address this limitation and enhance its comprehensiveness.
6 Conclusion
We introduce NurViD, a comprehensive video dataset designed for nursing procedure activity under- standing. By collecting videos from YouTube and meticulously annotating action sequences at an expert-level, NirViD offers rich resource for studying nursing procedures. The dataset encompasses 1,538 untrimmed videos, each averaging 32 seconds in duration, covering 51 distinct procedures and 177 action steps. We present three tasks based on ViDV procedure classification on untrimmed videos, procedure and action classification on trimmed videos, and action detection on untrimmed videos. Our experiments demonstrate that accurate recognition of procedures and the action steps they contain is challenging even with current tfate-of-the-ar art- models, particularly when the dataset exhibits a long-tail distribution. To promote further development in the field of nursing procedure analysis, we will release all of our data and code to the public, enabling researchers to build upon our work and advance the understanding of nursing activities.
10

---

References
[1] Creative commons 4.0 on youtube. http://craaiccooooooooooogllelss
[2] National health commission the people's republic china. http://en.nhh..ov.cuu/ procedures/.
[3] Nurselabs.
[4] Sample: Hand wash dataset. https//www..aggg..oooommmeeeeeee wash--attsst..
World health organisaaionn (who). https://www.who.in/.
[6] Youtube. https//wwww.youthhe.oom
[7] Stefan Ameling, Johnson Li, Jiang Zhou, Anarta Ghosh, Gerard Lacey, Eilish Creamer, and Hilary Humphreys. iision-based system for hand washing quality assessment with real-time foedback. Pro- ceedings of the 8th IASTED International Conference on Biomedical Engineering, Biomed 2011, 04 2011.
[8] Peter Babington. Kexiong, Zhou and Canhua, Xiao and Rangke, Wu and Chenling, Luo and Lei, Shi and Canhua Xiao. China Science Publishing & Media (CSPM), 2006.
[9] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[10] Daniel Defays, An efficient algorithm for a complete link method. The Computer Journal, 20(4):364-366, 1977.
[11] Ann Delamont. How to avoid the top seven nursing emoos. Nurs. Made Incred. Easy!, 11(2)88-0,, March 2013.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[13] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[14] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202- 6211, 2019.
[15] David Fernández-Llorca. Fernando Vilariño, Z Zhou, and Gerard Lacey. A multi-class classifier ensemble for automatic hand washing quality assessment. 09 2007.
[16] Silvio Giancola, Mohieddine Amine, Tarek Dghaily, and Bernard Ghanem, Soccemnt:: A scalable dataset for action spotting soccer videos. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018
[17] D. Gupta, K. Attal, and D. Demner-Fushman. A dataset for medical instructional videa classification and question answering. Sci Data, 10. 158 (2023).
[18] Fabian Caba Heilbron and Juan Carlos Nicbles. Collecting and annotating human activitics in web videos. In Proceedings of International Conference Multimedia Retrieval, pages 377-384, 2014.
[19] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 7(1-3):185-205 1981.
[20] Annamma Jacob, Rekha R., and Jadhav Sonali Tarachand. Clinical Nursing Procedures: The Art of Nursing Practice. Jaypee Brothers Medical Pub, 2015.
[21] Maryam Janatolmakan and Alireza Khatony. Explaining the consequences of missed nursing care from the perspective of nurses: qualitative descriptive study in iran. BMC Nurs., 21(1):59, March 2022.
[22] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
[23] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action vidco dataset. arXiv preprint / 06950, 2017.
[24] Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, and Limin Wang. Multisports: A multi-
person video datasct of spatio-temporally localized sports actions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13536-13545, October 2021.
[25] Bruce D Lucas, Takeo Kanade, et al. An iterative image registration technique with application to stereo vision, volume 81. Vancouver, 1981.

---

[26] Martins Lulla, Aleksejs Rutkovskis, Andreta Slavinska, Aija Vilde, Anastasija Gromova, Maksims Ivanovs. Ansis Skadins, Roberts Kadikis, and Atis Elsts. Wasd-ng video dataset annotated according to the world health organization's 's hand-washing guidelines. Data, 6(4). 2021.
[27] Karen MacKinnon, Lenora Marcellus, Julie Rivers, Carol Gordon, Maureen Ryan, and Diane Butcher. Student and educator experiences of maternal-chile simulation based learning: systematic review of qualitative evidence protocol. JBI Database System. Rev. Implement. Rep., 13(1): 26, January 2015.
[28] Sauradip Nag. Xiatian Zhu, Yi-Zhe Song. and Tao Xiang. Temporal action detection with global segmenta- tion mask learning. European Conference on Computer Vision, 2022.
[29] S.M. Nettina. Lippincont Manual of Nursing Practice. Wolters Kluwer Health, 2013.
[30] Zi Qi Pamela Ng, Li Ying Janice Ling, Han Shi Jocelyn Chew, and Ying Lau. The role of artificial intelligence in enhancing clinical nursing care: A scoping review. Nurs. Manag. 30(8):36543377, November 2022.
[31] A.G. Perry, P.A. Potter, and W.R. Ostendorf. Clinical Nursing Skills and Techniques E-Book. Elsevier Health Sciences, 2013.
[32]
Naney S Redeker. Sensor technology for nursing research. Nurs. Outlook, 68(6)711-719, November 2020.
[33] Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet:
Temporal action
detection with relative boundary modeling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 18857-18866. 2023.
[34] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Holly- wood in homes: Crowdsourcing data collection for activity understanding. 2016.
[35] Silvana Cruz da Silva, Bruna Xavier Morais, Oclaris Lopes Munhoz, Juliana Dal Ongaro, Janete de Souza Urbanetto, and Tānia Solange Bosi de Souza Magnago. Patient safety culture, missed nursing care and its reasons in obstetrics. Rev. Lat. Am. Enfermagem, 29:e3461, June 2021.
[36] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
[37]
Finav Srulovici and Anat Drach-Zahavy. Nurses' personal and ward accountability and missed nursing care: A cross-sectional study. Int J. Nurs. Stud., 75:163-171, October 2017.
[38] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri, Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489-4497, 2015.
[39] Lieshet Van Bulck, Raphaël Couturier, and Philip Moons. Applications of artificial intelligence for nursing: has a new era arrived? European Journal of Cardiovascular Nursing. 22(33)e19--22 10 2022.
[40] Carl Vondrick, Donald Patterson, and Deva Ramanan. Efficiently scaling up crowdsourced video annotation. International journal of computer vision, 101(1):184-204, 2013.
[41] Mci Wang and Weihong Deng. Deep visual domain adaptation: A survey. Neurocomputting 312:135-153,
2018.
[42] Ting Wang, Jun Xia, Tianyi Wu, Huanqi Ni, Erping Long, Ji-Peng Olivia Li, Lanqin Zhao, Ruoxi Chen, Ruixin Wang, Yanwu Xu, Kai Huang, and Haotian Lin. Handwashing quality assessment via deep learning: a modelling study for monitoring compliance and standards in hospitals and communities. Intelligent Medicine, 2(3):152-166 160, 2022.
[43] Peng Xia, Di Xu, Lie Ju, Ming Hu, Jun Chen, and Zongyuan Ge. Lmpt: Prompt tuning with class-specific embedding loss for long-tailed multi-label visual recognition. arXiv preprint arXiv:2305. 334.:2300.00456,, 2023.
[44] Siyuan Yan, Zhen Yu, Xuelin Zhang, Dwarikanath Mahapatra, Shekhar S Chandra, Monika Janda, Peter Soyer, and Zongyuan Ge. Towards trustable skin cancer diagnosis via rewriting model's decision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11568- 11577, 2023.
[45] Liming You. A discussion on the development of advised nursing practice in china: Specialization in nursing. Chinese Nursing Management, 2017.
[46] Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers.
In European Conference on Computer Vision, 2022.
[47] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 2023.
[48] Chengzhang Zhong, Amy R. Reibman, Hansel A. Mina, and Amanda J. Deering. Multi-view hand-hygicne recognition for food safety. Journal of Imaging, 6, 2020.
[49] Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI Conference on Artificial Intelligence, pages 7590-7598, 2018.
12

---

A Nur ViD Statistics Supplement
The distribution of data for each procedure and action is depicted in Figure.5 and Figure.6, respectively. Our findings reveal long-tail distribution for the number of collected and trimmed vidcos per type.
60
6.
00-
R
B00
occ
0N
INCT.
OR
Procedure
Figure Number of untrimmed videos per each procedure category. We rank the categories according to their untrimmed video frequency.
300
9
2
R
10
Figure 6: Number of trimmed videos per each action category. We rank the categories according to their trimmed video frequency.
B Training Details
We conducted the classification experiments using four NVIDIA RTX3090TI GPUs and the detection experiments using four NVIDIA RTX A6000 GPUs. All of our code can be found attached in the supplementary material and also on the project homepage https://githul http://hhhh...iinnnnuuoeeoe //t///ooo/o/0/0///////// Nur DD-benchmark.
B.1 Training Details for Procedure and Action Classification
We employed the officially released codes to train all recognition models. The SlowFast [14] and 13D [9] models were trained for 196 epochs with a batch size of 64, using base learning rate of 0.1. We employed cosine decay learning rate scheduler with 34 warm-up epochs. We sampled 16 frames per clip with sampling rate of 24. For the C3D [38], we used epochs, with 34 warm-up epochs and batch size 32, sampled 16 frames per clip with sampling rate 24.
learning rate of 0.1, cosine decay learning rate scheduler, trained for
B.2 Training Details for Action Detection
Feature extraction. extract features from the videos, we first extracted RGB frames from each video rate of 25 frames per second. We also extracted optical flow using the TV-LI [19, 25] algorithm. We then fine-tuned an 13D [9] model that had been pre- pre-trained on the ImageNet [12] dataset, and used it generate features for each RGB and optical flow frame. Because each video has variable duration, performed uniform interpolation
13

---

generate 100 fixed-length features for each video. Finally, concatenated the RGB and optical flow features into 2048-dimensione embedding, which served the input for model.
Model training. We trained all detection models using their officially released code and default configurations. training the ActionFormer [46] model, we used base learning rate of 0.001, cosine decay learning rate scheduler, trained for epochs with 5 warmup epochs, and a batch size of 16. For the TAGS [28] model, we used a base learning rate of 0.0004, step decay learning rate scheduler, trained for 20 epochs, and batch size of 200. For the TriDet [33] model, we used base learning rate of 0.0001, trained for 50 epochs, and batch size of 256.
C
The Long-tail Distribution of Procedure, Action, and Their Composition
Our data has skewed distribution in terms of the procedure, action, and also their composition. To get better insights, we split the calegories into many. medium,, and few groups based their frequency. We show the procedure distribution Figure. and Figure, 8, the action distribution in Figure. and their compositional distribution Figure. 10.
Many
Medium
No. of Videos
8000
or E.B
Us-
Noo
sou
WE
GrEOa Pice
The
woo
Nursing Procedure
Figure 7: The number of untrinmed videos per each procedure. The procedures with the frequency 2 are grouped into many. The procedures with the frequency < 50 and 2 20 are grouped into medium. The procedures with the frequency 20 grouped into few. We rank the procedures based on their frequency.
Many
Medium
No. of Segnents
.4.
60
C2TH
IAC
PAT
LR
ON
With Gal
Nursing Procedure
Figure 8: The number of trimmed videos per each procedure. The procedures with the frequency > 150 are grouped into many. The procedures with the frequency < 150 and 2 45 are grouped into medium. The procedures with the frequency < 45 are grouped into few. We rank the procedures based their frequency.
Many
Medium
Few
No. of Segments
Action
Figure 9: The number of trimmed videos per each action. The actions with the frequency 100 grouped into many. The actions with the frequency < 100 and > 20 are grouped into medium. The actions with the frequency 20 are grouped into few. We rank the actions based on their frequency.
14

---

Many Medium
Few
No. of Segmentts
303 108
100
1st Procedure Action
290
Figure 10: The number of trimmed videos per each composition of procedure category and action. The compositions with the frequency 2 50 are grouped into many. The compositions with the frequency 50 and 2 20 are grouped into medium. The compositions with the frequency < 20 are grouped into few. We rank the compositions based on their frequency.
D Compositional Low-shot Procedure and Action Classification on Trimmed Videos
Obtaining an adequate number of labeled action samples all procedures in our collected laxonomy poses challenges. To address this, we plan to propose some compositional low-shot procedure and action classification tasks (0-shot, 1-shot, and 5-shot) to investigate these phenomena in the future.
Training Seen Procedure +Action
USC provide urine sample
IVInj + disinfect the skin
USC check
IVBS disinfect the skin
USC disinfect the skin
Testing Useen Procedure +Action
Figure 11: Demonstration of compositional zero-shot procedure and action recognition.
15

---

E Annotation Interface Demonstration
The annotation work in the early stages is mainly divided into two phases: (1) procedure verification; (2) action localization annotation. In the first phase, annotators need check the information prompted by the interface, judge whether the vidco belongs to the nursing procedure, and filter unqualified videos such animations, voice broadcasts, and slideshows through the skip button. In the second phase, the annotation mainly focuses on the action localization of the qualified videos screened in the first phase. The two frame windows correspond to the starting frame and ending frame of the annotated segment.
Play
Speed Ub X2
RST
10s
RW 10x
Video 10: (BoK/C. atvo)
Add Instaoo
Slip
choos
(30.17%)71.55]
(50.56
derument
(250 42:200
Figure 12: The procedure verification and action localization interface.
16

---

ID
Procedure
Abbreviation
0
Surgical Hand Scrub
SHS
Cardiopulmonary Resuscitation WIth Manual Resuscitation Bag
CPR-MRB
2
Perineal Care
PC
Donning and Doffing Isolation Gowns
DDIG
4
Closed Intravenous infusion
CIVI
Subcutaneous Injection
SCInj
6
Change Two-Piece Pouching System
C2-PPS
Blood Glucose Monitoring
BGM
8
Intravenous Injection
IVIn
9
Intravenous Blood Sampling
IVBS
10
Oral Care for Unconscious Patients
OC-UP
11
Logrolling with Draw Sheet
L-DS
12
Vital Sign Assessment
VSA
13
Use of Restraints
UR
14
Subcutaneous Injection Insulin
SCInj-Ins
15
Nasogastric Gavage
NGG
16
Transfer with Stretcher
T-Str
17
Administering Oral Medications
AOM
Change One-Piece Pouching System
C1-PPS
19
Intramuscular Injection
IMInj
Change Sheets of an Occupied Bed
CSOOB
21
Bed Rubbing
BR
Change Wound Dressings
CWD
23
Oxygen Nebulization
ON
Wheelchair Transfer Technique
WTT
25
Bed Shampoo
BSh
Urine Specimen Collection
USC
27
Penicillin Skin Testing
PST
28
Infusion by Pump
IFP
29
Volume Colonic Enemas
НVCE
30
Defibrillation
Df
31
Arterial Blood Sampling
ABS
32
Closed Bed Making
CBM
33
Female Retention Catheterization
FRC
34
Stool Specimen Collection
SSC
35
Oxygen Therapy with Central Oxygen Supply
OT-COS
36
Male Retention Catheterization
MRC
Modified Seldinger Technique with Ultrasound for PICC Placement
MSU-PICC
38
Throat Swab Collection
TSC
Oral and Nasal Suctioning with Electric Suction Device
ONS-ESD
40
Sputum Specimen Collection
SSpC
Retention Enema
RE
42
Peripheral Venous Indwelled Needle Infusion and Maintaince
PVIN-IM
43
Injection by Pump
IBP
44
Electrocardiogram
ECG
45
Oral and Nasal Suctioning with Central Negative Pressure Device
ONS-CNP
46
Aseptic Technique
AT
47
Nasogastric Tube
NGT
48
Murti--ar Monitoring
MP
49
Closed System Blood Transfusion
CSBT
50
Skin Preparation
SP
Table 5: The procedure names and their abbreviations this paper.
17

---

ID
Action
ID
Action
0
Clean and scrub the perineum
89
Measure respiration
1
Disinfect skin
90
Change upper clothing
2
Handwashing
Wear gloves
3
Position the patient
92
Collect pharyngeal swab specimen
4
Check
93
Two-person transfer
5
Inject medication
94
Adjust drip
Place underpad
Cleanse inner surfaces teeth
7
Venipuncture
96
Connect lead wires
8
Cleanse skin
97
Cleanse inner surfaces of teeth
9
Rinse with running water
98
Collect sputum specimen
10
Document
99
Prepare operating space
Secure ostomy bag
PICC insertion
12
Perform surgical hand scrub
101
Remove the base plate
13
Blood collection
102
Connect suction catheter
14
Prepare medication solution
103
Loosen isolation gown
15
Release trapped air
104
Perform pharpyagala suction
16
Select a vein
105
Remove urinary catheter
17
Immobilize the shoulder
106
Install oxygen inhalation device
18
Remove needle
107
Assist with bed rest
19
Perform subeutaneous puncture
108
WWithdraw the introducer sheath
Measure blood glucose level
Check thermometer
21
Remove ostomy bag
110
Perform surgical hand disinfection
22
Connect infusion device
111
Fasten buckle
23
Remove isolation gown
112
Shift to the right side
24
Apply leak prevention ointment
113
Organize the hed unit
25
Prepare glucometer
Perform three- pereepperson transfer
26
Perform chest compressions
115
Cover pillow with pillowcase
27
Assist with ventilation using simple respirator
116
Select the needle puncture site
28
Draw bed curtains
117
Cleanse perineum
Turn patient to left lateral position
Comb
30
Trim ostomy bag baseplate
119
Inspect urinary catheter
31
Establish a sterile zone
120
Perform four-pere pur-person transfer
32
Spread the proximal bedshect
121
Adjust negative pressure
33
Rinse shampoo
122
Secure urinary catheter
34
Insert urinary catheter
Observe skin around wound site
35
Measure blood pressure
124
Observe results of skin test
36
Apply skin protection film
125
Change pillowcase
37
Put isolation gown
126
Flush the sealed tube
Insert rectal tube
Withdraw nebulizer
39
Prepare medications
128
Secure the indwelling needle
40
Apply shampoo
129
Soak feet
41
Perform seven-step handwashing technique
130
Remove isolation gown
42
Aspirate medication
131
Dispose of arterial blood collection device
43
Assist patient taking medicine
Cleanse cheeks
44
Measure body temperature
133
Check the blood pressure meter
45
Set parameters
134
Replace clean bedsheet
46
Measure pulse
135
Move and transfer
Remove proximal bedsheet
Evaluate wound status
48
Transport in wheelchair
137
Rinse suction catheter
49
Fill in dressing
138
Expose the connection sit
50
Rub upper limbs
139
Evaluate resuscitation effect
51
Install nehulizer
140
Connect the monitor
52
Wash face
Measure the length PICC catheter
Perform nasopharyngeal and
Insert gastric
nasotracheal suction
54
Spread the opposite side bed sheet Identify cardiac arrest
143
Observe drainage situation
55
144
Save electrocardiogram (ECG) results
56
Remove rectal tube
145
Remove the lead wires
57
Perform intradermal puncture
146
Perform single-person transfer
58
Moisten hair
147
Connect suction tube
59
Prepare defibrillation device
148
Prepare cotton balls
60
Dry hair
Cleanse hard palate
61
Confirm the position of the gastric tube in the stomach
150
Cleanse tongue surface
18

---

ID
Action
ID
Action
62
Secure the base
151
Connect injection device
63
Cleanse lips
152
Withdraw contaminated bed shee
64
Defibrillate
153
Transfuse blood
65
Cleanse chest and abdomen
154
Withdraw oxygen inhalation device
66
Cleanse back
155
Remove gastric tube
67
Open airway
156
Organize the blood pressure mete
Cleanse outer surfaces of teeth
Disinfect instruments
69
Adjust oxygen flow rate
158
Change dressing
70
Tie waist knot
159
Observe defibrillation results
71
Dry hands
160
Take treatment towels
72
Withdraw the opposite side bed sheet
161
Apply conductive gel
73
Mcasure the length the gastric
Cover with bed sheet
74
Spray stoma care powder
163
Rinse mouth
75
Defibrillate
164
Change pants
76
Remove dressing
165
Secure drainage tube
77
Perform arterial puncture
166
Check medication
78
Collect urine specimen
167
Take treatment bowl
79
Spread the large shect
168
Pour sterile solution
80
Administer oxygen
169
Cleanse oral cavity bottom
81
Guide nebulization
170
Restrict knee
Check secure the tubing
Mark
83
Rub lower limbs
172
Monitor blood oxygen saturation
84
Mix blood sample
173
Check the pressure reducer
85
Prepare skin test solution
174
Inspect skin
86
Secure gastric tube
175
Perform endotracheal suctioning
Nasogastric feeding
Monitor electrocardiogram (ECG)
88
Collect stool specimen
Table 6: 177 actions in this paper.
19


Reviews:
[{"review_id": "esDSxv1ECh", "reviewer": "Reviewer_Y27H", "summary": "", "questions": "", "limitations": "The authors have mentioned and discussed the limitations of the presented dataset in the limitations section in an adequate manner. However, the authors are advised to include a discussion of limitations from a nursing procedure point of view such as missing procedures/actions (if any). ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) The dataset is the largest of its kind, and it offers a good variety of actions and procedures that are relevant and important for nursing procedure.\n2) The level of annotation includes high-level and fine-grained annotations which makes the dataset useful for various tasks\n3) The annotation was done by experts in the domain and students in the field of nursing who have passed a standardized test in their studies. \n4) The type of videos used makes it more realistic as they comes from actual nursing procedures from youtube. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Q9sZoF3BLa", "reviewer": "Reviewer_sFtw", "summary": "", "questions": "", "limitations": "For video action classification, one thing that happens for the general benchmark such as kinetics is that the model trained on those datasets sometimes relies more on static images rather than the activity motions to make prediction.  For example, the model can simply classify a person is playing football by simply inputting a single image about the soccer and the playfield. Some dataset such as somethingsomething try to address such bias by including the activities that put more emphasis on the motions. I am wondering what is the case for the proposed dataset. \n\nA brief discussion and some quantitative experiment would be helpful. For example, the author can investigate the influence on the number of frames as input to ablate the model so that it becomes more clear whether the model is making use of more temporal information or just the static image information.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "There is current lack of sufficiently labelled datasets for the development of automatic recognition systems within nursing procedure activity. Existing datasets face limitations including small-scale size, a focus on single procedures, and a lack of temporally localized annotations. \n\nTo address these issues, the authors propose NurViD, a large-scale video dataset with expert-level annotations for understanding nursing procedures. The NurViD dataset, which is about four times longer than the largest existing dataset, consists of over 1.5k videos totaling 144 hours. It covers 51 distinct nursing procedures and 177 action steps, offering a more comprehensive scope compared to previous datasets.\n\nThe authors also established three benchmarks on NurViD for evaluating the effectiveness of deep learning methods on nursing activity understanding. These benchmarks involve procedure recognition on untrimmed videos, procedure and action recognition on trimmed videos, and action detection.\n\nTo this end, they plan to make the benchmark and code publicly accessible.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xe8OpOiSyC", "reviewer": "Reviewer_ysqj", "summary": "", "questions": "", "limitations": "Limitations are very briefly, rather superficially, touched upon currently in the paper. This section maybe improved upon. ", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The dataset is second-largest in size and the most diverse nursing procedure video dataset. This would be helpful to the community.\n2. Paper well showcases the dataset and its properties/stats (unlike quite a few other dataset papers which ended up focusing framework development).\n3. Annotations are provided by field experts.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dmV1rIxq95", "reviewer": "Reviewer_CHPV", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1) The proposed nursing procedure video dataset has advantages over previous related datasets on the data scale, diversity and expert annotations. This can be useful for nursing procedure activity understanding.  \n2) Comprehensive experiments are conducted on the dataset to build benchmark results for three tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper receives four unanimous recommendations: four accepts. The reviewers all think this paper is valuable for video understanding. After a careful check, the AC agrees with the reviewers, and thus makes an accept recommendation to this paper.
Decision: Accept (Poster)