SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Class splits. In this study, we also explore the long-tailed nature of the NurViD. We partitioned the videos into three subsets based on the task-specific distribution: many, medium, and few. For procedure categories, the breakdown is as follows: many: top  $26\%$  frequent classes, medium: middle  $41\%$  classes, and few: the remaining  $33\%$  classes. For action categories, many: top  $5\%$  frequent classes, medium: middle  $37\%$ , and few: the remaining  $58\%$  classes. In addition to some unique actions, such as establish a sterile zone that only exists in the Modified Seldinger Technique with Ultrasound for PICC Placement procedure, hand washing, skin disinfection, and other steps are common. Therefore, we established a joint classification task to explore the mutual influence between procedures and actions. For joint classification, many: top  $5\%$  frequent classes, medium: middle  $21\%$  classes, few: the remaining  $74\%$  classes. We show the number of classes per each split in Table 3.
Baselines. We compare the performance of three models, SlowFast [14], I3D [9], and C3D [38] on our tasks. These models are originally designed for human action recognition and lack the inherent ability to predict both a procedure and an action. To align with our joint prediction need, we introduce two task heads dedicated to procedure category recognition and action recognition, respectively. Consequently, we compute the joint loss,  $\mathcal{L}_{joint}$ , to handle these tasks. Hyper-parameters are tuned using the validation data, and the detailed hyper-parameter settings for each model can be found in the supplementary material. The loss is defined as  $\mathcal{L}_{joint} = -\frac{1}{M}\sum_{i=1}^{M}y_i^p\log(p_i^p) - \frac{1}{N}\sum_{j=1}^{N}y_j^a\log(p_j^a)$  where  $M$  is the number of procedure classes,  $N$  is the number of action classes,  $p_i^p$  and  $y_i^p$  denote the procedure prediction probability and ground truth label for the category  $i$ ,  $p_j^a$  and  $y_j^a$  denote the action prediction probability and ground truth label for the category  $j$ .
Results. The results are summarized in Table 3. All models perform well in the procedure classification task, with C3D[38] achieving a top-1 per-class accuracy of  $71.2\%$  across all splits. C3D[38] also demonstrates competitive performance in action and joint classification for all splits, with the best top-1 per-class accuracy of  $22.8\%$  and  $13.1\%$ , respectively. Transfer learning from a pre-trained model on Kinetics 400 [23] further improves the accuracy of procedure and action classification. For instance, using the C3D model, the per-class accuracy increases from  $70.1\%$  to  $73.2\%$  for many classification, from  $48.8\%$  to  $60.0\%$  for medium classification, and from  $33.0\%$  to  $39.6\%$  for few classification.
**Discussions.** The experimental results indicate that the performance of procedure classification on trimmed videos is significantly better than on untrimmed videos, which may confirm that irrelevant motion information has been filtered out in the trimmed videos, and the model is more likely to learn motion features. However, The imbalance in class frequencies poses difficulties in achieving
Figure 4: Visualization of action detection results. From top to bottom: (1) input video frames; (2) action scores at each time step; (3) histogram of action onsets and offsets computed by weighting the regression outputs using action scores.
satisfactory performance for those few classes. To alleviate it, further model design could incorporate long-tail learning techniques such as re-weighting or re-sampling techniques. These approaches can help mitigate the impact of class imbalance and improve the model's ability to generalize and classify the underrepresented classes more effectively.
4.3 Action Detection on Untrimmed Videos
The objective of this task is to accurately identify actions in untrimmed videos by determining the temporal extent of the main activity. To establish a benchmark for this task, we adopt three baseline models [46, 28, 33] that have shown effectiveness in temporal action localization [13, 46]. The evaluation metric used is the mean Average Precision (mAP), calculated at various temporal Intersection over Union (tIoU) thresholds [0.5:0.1:0.9]. Additionally, we provide the average mAP across different tIoUs.
Data settings. Following previous approaches for the standard procedure and action classification, we adhere to a split ratio of [train: 0.7, val: 0.1, test: 0.2] to divide the untrimmed videos at the procedure-action composition level. Consequently, we have 1,077 videos for training, 153 for validation, and 308 for testing.
Baselines. To evaluate the performance of our datasets, we utilize baseline models, namely ActionFormer [46], TAGS [28], and TriDet [33]. In order to generate features for the NurViD videos, we fine-tune a two-stream I3D model (I3D [9]) initially pre-trained on ImageNet [12]. Subsequently, we extract RGB and optical flow features for each video and concatenate them as the model input.
Results. We show the action detection results in Table 4. Among the baselines, ActionFormer [46] achieves the highest performance, with an average mAP of  $23.9\%$  and an mAP of  $32.9\%$  for the threshold of 0.5.
Discussions. The outputs of the Action-Former [46] model are visualized in Figure. 4. These outputs consist of action scores and regression results, which are weighted by the action scores and presented as a weighted histogram. Nursing actions, unlike actions in natural datasets,
involve precise and meticulous movements rather than large-scale motion from frame to frame. Therefore, a more detailed action representation is necessary to accurately describe and learn nursing actions. For instance, in the hand wash procedure recommended by the WHO [5], it is crucial to consider small changes in hand position and environmental factors. Since these movements are extremely subtle, algorithms capable of handling finer granularity can potentially detect these subtle motion changes.
Baselines	mAP (%)
0.5	0.6	0.7	0.8	0.9	Avg.
TriDet [33]	30.3	26.7	24.3	20.1	10.7	20.8
TAGS [28]	31.4	26.5	22.6	19.2	11.5	22.4
ActionFormer [46]	32.9	29.6	25.8	20.8	12.7	23.9
Table 4: The results of action detection. We report mAP at the IoU thresholds of [0.5:0.1:0.9]. The average mAP is calculated by averaging the mAP scores across various tIoU thresholds.
We believe that NurViD is beneficial for advancing the development of AI technology in the nursing field. However, we must also consider the potential risks and impacts that may arise from anticipated or foreseeable applications. Additionally, NurViD is downloaded from diverse YouTube channels, video quality, production style, and regional nursing practice differences can introduce biases in nursing procedure representation.
Intended/Foreseeable Uses. After reviewing relevant literature and discussing with nursing professors and students, we have noticed two main challenges in nursing training and learning: (1) the issue of imprecise recommendations encountered by students when searching for learning videos. For example, when we want to learn about Intravenous Injection procedure videos on YouTube, due to the limitations of the recommendation algorithm, the website may recommend Intravenous Blood Sampling procedure videos to some extent, which is quite common. However, the system trained on our dataset can provide more accurate classification recommendations for searches related to nursing operations. Additionally, since NurViD includes temporal localization annotations for actions, it means that we can dynamically adjust the playback range based on our specific interests, for instance, if I only need to watch the step of skin disinfection and not the other steps, I just need to enter the text "skin disinfection," and the system will automatically locate the specific segment for me. This undoubtedly saves some unnecessary time wastage, and (2) in China alone, there are over 200,000 undergraduate nursing students each year, and this number continues to grow. Each student is required to pass a professional nursing skills examination and undergo approximately 1000 to 2000 hours of practical training. However, currently, students still heavily rely on experienced teachers for real-time supervision and feedback during training, which requires a significant amount of human resources and time investment. (3) Real-time monitoring: NurViD is a dataset that leans more towards general models and is currently primarily used for testing and improving deep learning models. It aims to assist students' learning and training by recording, monitoring, and providing feedback during their practice sessions, reducing the need for teaching resources, and facilitating event documentation. In this mode, even rough feedback can save a significant amount of costs. However, AI systems cannot guarantee absolute accuracy, which means that detection errors or omissions may occur. Therefore, it is strongly recommended that any system built upon NurViD or similar technologies clearly communicate their limitations and potential errors from the outset and appropriately incorporate human assistance during usage.
Potential Privacy. In human-action understanding of video datasets, it is often inevitable to encounter faces. Therefore, we provide a script that uses OpenCV's Haar classifier to detect the facial regions in videos and blur them. We will implement more advanced methods to further alleviate the privacy problem in the future.
Employment Risks. The employment risks associated with nursing action recognition systems can involve the following aspects: (1) Reduced Workforce Demand: The integration of nursing action recognition systems may lead to a diminished need for human nursing professionals. The incorporation of automation technology can handle routine nursing tasks, thereby potentially decreasing the reliance on human caregivers. Consequently, this could result in certain nursing professionals facing job scarcity or encountering uncertainty in their employment prospects. (2) Shift in Skill Requirements: Nonetheless, our perspective is that the future implementation of nursing action recognition systems will likely require nursing professionals to acquire new skills and knowledge to adeptly interact with the technology rather than being replaced by it. This could involve gaining proficiency in various areas, such as effectively engaging with the system, accurately interpreting its outputs, and promptly addressing any discrepancies that arise. Nursing professionals encountering challenges in adapting to these evolving technological demands might find it necessary to undergo retraining efforts. (3) Technical malfunctions and misidentification risks: Nursing action recognition systems come with inherent risks of technical glitches and misidentifications. Inaccuracies or erroneous assessments made by the system could lead to misguided nursing judgments or actions. This potentially jeopardizes patient safety and well-being, obliging nursing professionals to dedicate extra time and effort toward rectifying system errors. To mitigate the employment risks linked to nursing action recognition systems, a comprehensive approach is crucial: (1) Workforce Enhancement Programs: Offering programs for upskilling and transitioning is vital to empower nursing professionals with the competencies needed to navigate evolving roles in tandem with the technology. (2) Ethical Guidelines and Standards: Establishing clear ethical guidelines ensures
responsible and morally sound utilization of nursing action recognition systems within healthcare settings. (3) Data Privacy and Security: Implementing robust measures to safeguard patient data and privacy is paramount to engender trust in the system's operation. (4) Human Oversight and Decision-making: Maintaining human oversight ensures that critical nursing decisions are grounded in human judgment and understanding, acting as a safeguard against erroneous system outputs. (5) Continuous System Evaluation and Enhancement: Regularly evaluating and enhancing the system's performance is pivotal to addressing any technical shortcomings and refining its accuracy. (6) Stakeholder Engagement and Collaboration: Fostering engagement and collaboration among various stakeholders, including nursing professionals, technologists, and policymakers, promotes a holistic approach to system development and implementation.
Contestability/Explainability Issues. It is important to acknowledge that while AI systems can assist in detecting standardized nursing procedures and actions, they are not infallible and cannot achieve perfect accuracy. Human supervision and oversight are indispensable in ensuring patient safety and quality care. Therefore, it is highly recommended that any system developed based on NurViD or similar technologies explicitly state their limitations and potential errors upfront. This can be achieved by providing clear disclaimers, agreements, or warnings to users, emphasizing the need for human involvement, critical thinking, and professional judgment when interpreting and acting upon system outputs. By transparently communicating the system's limitations, healthcare professionals can make informed decisions and use the technology as a supportive tool rather than relying solely on its outputs.
Potential Regional Biases. Standardization poses a significant challenge due to the diverse origins of the videos and the variation in nursing procedure guidelines across countries. NurViD aims to cover almost all common action labels, providing flexibility for different regions to adopt their own standards based on it. However, despite our efforts, it is important to acknowledge that complete avoidance of bias is challenging. Additionally, the comprehensiveness of the dataset may be influenced by the sources and origins of the videos. The video collection process for NurViD may inadvertently introduce biases towards certain regions or healthcare settings. This bias can limit the generalizability of the dataset to a broader context, as it may not fully capture the diverse range of nursing procedures and actions practiced worldwide. Addressing this limitation requires ongoing efforts to collect data from diverse regions, collaborate with experts from different backgrounds, and ensure a balanced representation of nursing practices from various healthcare contexts. Furthermore, domain adaptation models may also be one of the potential solutions [41, 44, 43].
Comprehensiveness of Nursing Procedures and Actions. While efforts were made to include a wide range of common action labels, it is important to acknowledge that the dataset may not cover every possible nursing procedure or action. Variations in nursing practices and guidelines across different regions and healthcare systems can result in some actions being omitted or not adequately represented in the dataset. Furthermore, the dataset's composition may be influenced by the availability and accessibility of videos from different regions. Certain nursing procedures or actions that are more prevalent or emphasized in specific regions may be overrepresented, while others may be underrepresented. Continuously expanding the dataset's coverage through collaboration with experts and professionals from diverse backgrounds can help address this limitation and enhance its comprehensiveness.
We introduce NurViD, a comprehensive video dataset designed for nursing procedure activity understanding. By collecting videos from YouTube and meticulously annotating action sequences at an expert-level, NurViD offers a rich resource for studying nursing procedures. The dataset encompasses 1,538 untrimmed videos, each averaging 32 seconds in duration, covering 51 distinct procedures and 177 action steps. We present three tasks based on NurViD: procedure classification on untrimmed videos, procedure and action classification on trimmed videos, and action detection on untrimmed videos. Our experiments demonstrate that accurate recognition of procedures and the action steps they contain is challenging even with current state-of-the-art models, particularly when the dataset exhibits a long-tail distribution. To promote further development in the field of nursing procedure analysis, we will release all of our data and code to the public, enabling researchers to build upon our work and advance the understanding of nursing activities.
A NurViD Statistics Supplement
The distribution of data for each procedure and action is depicted in Figure.5 and Figure.6, respectively. Our findings reveal a long-tail distribution for the number of collected and trimmed videos per type.
Figure 5: Number of untrimmed videos per each procedure category. We rank the categories according to their untrimmed video frequency.
Figure 6: Number of trimmed videos per each action category. We rank the categories according to their trimmed video frequency.
We conducted the classification experiments using four NVIDIA RTX3090Ti GPUs and the detection experiments using four NVIDIA RTX A6000 GPUs. All of our code can be found attached in the supplementary material and also on the project homepage https://github.com/minghu0830/NurViD-benchmark.
B.1 Training Details for Procedure and Action Classification
We employed the officially released codes to train all recognition models. The SlowFast [14] and I3D [9] models were trained for 196 epochs with a batch size of 64, using a base learning rate of 0.1. We employed a cosine decay learning rate scheduler with 34 warm-up epochs. We sampled 16 frames per clip with a sampling rate of 24. For the C3D [38], we used a base learning rate of 0.1, a cosine decay learning rate scheduler, trained for 196 epochs, with 34 warm-up epochs and a batch size of 32. We sampled 16 frames per clip with a sampling rate of 24.
B.2 Training Details for Action Detection
Feature extraction. To extract features from the videos, we first extracted RGB frames from each video at a rate of 25 frames per second. We also extracted optical flow using the TV-L1 [19, 25] algorithm. We then fine-tuned an I3D [9] model that had been pre-trained on the ImageNet [12] dataset, and used it to generate features for each RGB and optical flow frame. Because each video has a variable duration, we performed uniform interpolation to
generate 100 fixed-length features for each video. Finally, we concatenated the RGB and optical flow features into a 2048-dimensional embedding, which served as the input for our model.
Model training. We trained all detection models using their officially released code and default configurations. For training the ActionFormer [46] model, we used a base learning rate of 0.001, a cosine decay learning rate scheduler, trained for 30 epochs with 5 warmup epochs, and a batch size of 16. For the TAGS [28] model, we used a base learning rate of 0.0004, a step decay learning rate scheduler, trained for 20 epochs, and a batch size of 200. For the TriDet [33] model, we used a base learning rate of 0.0001, trained for 50 epochs, and a batch size of 256.
C The Long-tail Distribution of Procedure, Action, and Their Composition
Our data has a skewed distribution in terms of the procedure, action, and also their composition. To get better insights, we split the categories into many, medium, and few groups based on their frequency. We show the procedure distribution in Figure. 7 and Figure. 8, the action distribution in Figure. 9 and their compositional distribution in Figure. 10.
Figure 7: The number of untrimmed videos per each procedure. The procedures with the frequency  $\geq 50$  are grouped into many. The procedures with the frequency  $< 50$  and  $\geq 20$  are grouped into medium. The procedures with the frequency  $< 20$  are grouped into few. We rank the procedures based on their frequency.
Figure 8: The number of trimmed videos per each procedure. The procedures with the frequency  $\geq$  150 are grouped into many. The procedures with the frequency  $< 150$  and  $\geq 45$  are grouped into medium. The procedures with the frequency  $< 45$  are grouped into few. We rank the procedures based on their frequency.
Figure 9: The number of trimmed videos per each action. The actions with the frequency  $\geq 100$  are grouped into many. The actions with the frequency  $< 100$  and  $\geq 20$  are grouped into medium. The actions with the frequency  $< 20$  are grouped into few. We rank the actions based on their frequency.
Figure 10: The number of trimmed videos per each composition of procedure category and action. The compositions with the frequency  $\geq 50$  are grouped into many. The compositions with the frequency  $< 50$  and  $\geq 20$  are grouped into medium. The compositions with the frequency  $< 20$  are grouped into few. We rank the compositions based on their frequency.
D Compositional Low-shot Procedure and Action Classification on Trimmed Videos
Obtaining an adequate number of labeled action samples for all procedures in our collected taxonomy poses challenges. To address this, we plan to propose some compositional low-shot procedure and action classification tasks (0-shot, 1-shot, and 5-shot) to investigate these phenomena in the future.
Training: Seen Procedure + Action
Figure 11: Demonstration of compositional zero-shot procedure and action recognition.
E Annotation Interface Demonstration
The annotation work in the early stages is mainly divided into two phases: (1) procedure verification; (2) action localization annotation. In the first phase, annotators need to check the information prompted by the interface, judge whether the video belongs to the nursing procedure, and filter out unqualified videos such as animations, voice broadcasts, and slideshows through the skip button. In the second phase, the annotation mainly focuses on the action localization of the qualified videos screened in the first phase. The two frame windows correspond to the starting frame and ending frame of the annotated segment.
Figure 12: The procedure verification and action localization interface.
ID	Procedure	Abbreviation
0	Surgical Hand Scrub	SHS
1	Cardiopulmonary Resuscitation WIth Manual Resuscitation Bag	CPR-MRB
2	Perineal Care	PC
3	Donning and Doffing Isolation Gowns	DDIG
4	Closed Intravenous infusion	CIVI
5	Subcutaneous Injection	SCInj
6	Change a Two-Piece Pouching System	C2-PPS
7	Blood Glucose Monitoring	BGM
8	Intravenous Injection	IVInj
9	Intravenous Blood Sampling	IVBS
10	Oral Care for Unconscious Patients	OC-UP
11	Logrolling with Draw Sheet	L-DS
12	Vital Sign Assessment	VSA
13	Use of Restraints	UR
14	Subcutaneous Injection Insulin	SCInj-Ins
15	Nasogastric Gavage	NGG
16	Transfer with Stretcher	T-Str
17	Administering Oral Medications	AOM
18	Change a One-Piece Pouching System	C1-PPS
19	Intramuscular Injection	IMInj
20	Change Sheets of an Occupied Bed	CSOOB
21	Bed Rubbing	BR
22	Change Wound Dressings	CWD
23	Oxygen Nebulization	ON
24	Wheelchair Transfer Technique	WTT
25	Bed Shampoo	BSh
26	Urine Specimen Collection	USC
27	Penicillin Skin Testing	PST
28	Infusion by Pump	IFP
29	High-Volume Colonic Enemas	HVCE
30	Defibrillation	Df
31	Arterial Blood Sampling	ABS
32	Closed Bed Making	CBM
33	Female Retention Catheterization	FRC
34	Stool Specimen Collection	SSC
35	Oxygen Therapy with Central Oxygen Supply	OT-COS
36	Male Retention Catheterization	MRC
37	Modified Seldinger Technique with Ultrasound for PICC Placement	MSU-PICC
38	Throat Swab Collection	TSC
39	Oral and Nasal Suctioning with Electric Suction Device	ONS-ESD
40	Sputum Specimen Collection	SSpC
41	Retention Enema	RE
42	Peripheral Venous Indwelled Needle Infusion and Maintainence	PVIN-IM
43	Injection by Pump	IBP
44	Electrocardiogram	ECG
45	Oral and Nasal Suctioning with Central Negative Pressure Device	ONS-CNP
46	Aseptic Technique	AT
47	Nasogastric Tube	NGT
48	Multi-Parameter Monitoring	MP
49	Closed System Blood Transfusion	CSBT
50	Skin Preparation	SP
Table 5: The procedure names and their abbreviations in this paper.
Table 6: 177 actions in this paper.