SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ming Hu $^{1,2,3*}$  Lin Wang $^{1,2,4*}$  Siyuan Yan $^{1,3*}$  Don Ma $^{1*}$
Qingli Ren $^{5}$  Peng Xia $^{1,2,3}$  Wei Feng $^{1,2,3}$
Peibo Duan3 Lie  $\mathbf{J}\mathbf{u}^{1,2,3}$  Zongyuan Ge $^{1,2,3}$
$^{1}$ AIM Lab, Faculty of IT, Monash University, Australia
$^{2}$ Airdoc-Monash Research, Airdoc, China
$^{3}$ Faculty of Engineering, Monash University, Australia
<sup>4</sup>College of Intelligent Systems Science and Engineering, Harbin Engineering University, China
$^{5}$ Nursing College, Shanxi Medical University, China
The application of deep learning to nursing procedure activity understanding has the potential to greatly enhance the quality and safety of nurse-patient interactions. By utilizing the technique, we can facilitate training and education, improve quality control, and enable operational compliance monitoring. However, the development of automatic recognition systems in this field is currently hindered by the scarcity of appropriately labeled datasets. The existing video datasets pose several limitations: 1) these datasets are small-scale in size to support comprehensive investigations of nursing activity; 2) they primarily focus on single procedures, lacking expert-level annotations for various nursing procedures and action steps; and 3) they lack temporally localized annotations, which prevents the effective localization of targeted actions within longer video sequences. To mitigate these limitations, we propose NurViD, a large video dataset with expert-level annotation for nursing procedure activity understanding. NurViD consists of over 1.5k videos totaling 144 hours, making it approximately four times longer than the existing largest nursing activity datasets. Notably, it encompasses 51 distinct nursing procedures and 177 action steps, providing a much more comprehensive coverage compared to existing datasets that primarily focus on limited procedures. To evaluate the efficacy of current deep learning methods on nursing activity understanding, we establish three benchmarks on NurViD: procedure recognition on untrimmed videos, procedure and action recognition on trimmed videos, and action detection. Our benchmark and code will be available at https://github.com/minghu0830/NurViD-benchmark.
The application of deep learning (DL) in understanding nursing procedure activities has the potential to greatly enhance the quality of nurse-patient interactions, while also playing a crucial role in preventing medical disputes, minimizing missed nursing procedures, and reducing nursing errors [32, 45, 35, 37, 21, 11]. DL-based automatic approaches offer several key benefits for nurses, including: (1) Reliable: it enables objective, precise, and consistent assessments of nursing skills, eliminating the subjectivity inherent in traditional evaluations by experts [27]. (2) Real-time guidance: it facilitates immediate feedback on nurses' performance, empowering them to identify areas for improvement in a timely manner [30]. (3) Cost-effective: it can alleviate the burden of manual observations, evaluations, and training by experts [39].
However, the development of automatic nursing recognition systems is currently hindered by the absence of suitably labeled datasets for nursing activities. Existing public video datasets for action recognition primarily focus on generic daily activities or specific sports, with minimal attention given to nursing or healthcare scenarios [23, 13, 36, 16, 22, 24, 17]. For example, the Kinetics 700 dataset [23] with 700 category labels includes only two labels related to nursing activities. Although some initiatives have attempted to create nursing activity understanding video datasets (e.g., handwashing-specific datasets [15, 7, 48, 42, 4, 26]), limitations arise due to the gap between them and real-world clinical settings, which is summarized as follows: (1) Limited procedure and action variety: Expensive annotation cost causes existing datasets only have a single nursing procedure, whereas real clinical environments involve a wide range of complex procedures. (2) Simple scenes only: The recorded videos are typically captured in controlled settings such as instructional or laboratory environments, which do not accurately reflect the complexity and variability of nursing procedures in actual clinical practice. (3) Un-professional labeling: They suffer from non-professional labeling or a lack of adherence to standard guidelines, leading to errors or inconsistencies. (4) Short video sequence: They primarily consist of short action clips, which do not facilitate understanding long-term activities and their context.
To mitigate these limitations, we proposed NurViD, a large-scale video benchmark for nursing procedure activity understanding. Compared to existing datasets, NurViD incorporates characteristics from the following aspects: (1) Diverse procedure and action: NurViD comprises 144 hours of annotated videos, which is approximately four times longer than the largest existing nursing activity datasets. It also contains 1,538 videos depicting 51 nursing procedure categories, covering the majority of common procedures, along with 177 action steps, providing much more comprehensive coverage, compared to previous datasets that primarily focus on single procedures with limited action steps. (2) Real-world clinic settings: Videos in NurViD were captured from over ten real clinical environments according to our statistics, including hospitals, clinics, and nursing homes. This diverse range of settings ensures that the models trained on NurViD are applicable in real-world clinical scenarios. (3) Expert-level annotations: NurViD was labeled by professionals with high expertise and knowledge in nursing. The procedure and action annotation process follows the guideline of Training Outline for Newly Employed Nurses issued by the National Health Commission of China [2], ensuring consistency and accuracy of the annotations. (4) Support multiple recognition and detection tasks: We have established two different classification tasks and an action temporal localization benchmark specifically targeting the long-tail distribution of the dataset.
We further compare our NurViD dataset with the other existing nursing activity video datasets and summarize the key difference in Table 1. The contributions of NurViD are summarized as follows:
- NurViD is the most diverse video benchmark to date for nursing procedure activity understanding tasks. It has been meticulously annotated by nursing professionals, and the annotation process follows standardized nursing procedure guidelines or protocols. NurViD exhibits a competitive video size and much more comprehensive coverage of procedures and action categories compared to existing datasets.
- In response to the practical needs of nursing and machine-learning communities, such as education and training, automatic action detection, and long-tail distribution, we establish three different recognition and localization benchmarks on NurViD.
- Long-term retention and availability of dataset. NurViD has been sourced from YouTube and follows the CC BY 4.0 license agreement [1].
Research aimed at enabling machines to understand human behavior and activities has led to advancements in various practical applications. However, building such systems comes with challenges that require appropriate datasets for training and evaluation. In recent years, numerous datasets have been created to support research in human behavior understanding [34, 13, 49, 36]. While these datasets have been valuable for general activity recognition, only a limited number cater to the specific needs of nursing professionals.
Sequential action prediction. Standardized datasets have been meticulously designed for the purpose of evaluating the performance of algorithms in understanding and accurately recognizing intricate
Datasets	Dataset Properties	Tasks
Publicly Available?	Expert-Level Annotations?	Real-World Settings?	No. of Videos	No. of Segments	No. of Procedures	No. of Actions	Total Duration	Procedure Recognition	Action Recognition	Action Detection
Llorca et al. [15]	X	✓	X	8	-	1	7	4.2min	X	✓	X
Ameling et al. [7]	X	-	X	24	-	1	6	-	X	✓	X
Zhong et al. [48]	X	-	-	200	1,400	1	7	-	X	✓	X
Wang et al. [42]	X	X	✓	280	2,760	1	8	-	X	✓	X
Kaggle [4]	✓	X	X	292	3,504	1	12	23.3h	X	✓	X
Lulla et al. [26]	✓	-	✓	3,185	6,689	1	7	38.9h	X	✓	X
NurViD (Ours)	✓	✓	✓	1,538	5,608	51	177	144.4h	✓	✓	✓
Table 1: The comparison among existing nursing procedure activity video datasets. Compared to other datasets, NurViD annotates the procedures and actions by following expert-level standards, focuses on more comprehensive coverage of various nursing procedure categories, collects a large number of videos, totaling 144 hours, and also enables action detection tasks.
activities within real-world scenarios [22, 16]. These datasets are used in various fields, including computer vision, robotics, natural language processing, and surveillance systems. Focusing on specific sets of actions carried out in a well-defined order, sequential action understanding datasets differ from those that encompass a broader range of contexts [49]. In many fields, it is essential to adhere to a strict sequence of steps to ensure optimal results. For example, in the medical field, following a specific order of steps is crucial in procedures such as administering medication, where any deviation from the established sequence could result in serious consequences for the patient. Standardized datasets can help ensure that the actions performed in real-world situations are accurately represented and provide a benchmark for evaluating the performance of algorithms.
Nursing procedure video dataset. Online learning has gained significant popularity, particularly through the utilization of instructional videos that provide step-by-step guidance, serving as valuable resources for teaching and learning specific tasks. Within the medical field, instructional videos have proven to be highly effective in conveying essential information using visual and verbal communication, thus benefiting learners [17]. On the other side, the absence of comprehensive and standardized nursing procedure video datasets presents a challenge in developing effective algorithms within the healthcare industry. The current datasets suffer from limited coverage, focusing only on nursing procedures relevant to specific healthcare settings or patient populations. Additionally, the quality of annotation plays a critical role in algorithm accuracy. The process of annotation involves identifying and labeling specific actions and events in the videos, which can be a time-consuming and challenging task. Annotation errors may arise due to factors such as human error, task ambiguity, or the absence of standardized protocols.
3 Building NurViD Dataset
In this section, we describe the process of building NurViD, from selecting the nursing procedures to acquiring, filtering, and annotating the video data. We leveraged the extensive collection of medical instructional videos available on YouTube [6] and carefully selected and filtered our video collection to ensure the quality and relevance of the dataset. We also developed a standardized labeling scheme for the actions performed in each video, providing a valuable resource for developing and evaluating algorithms that recognize and understand nursing procedures.
3.1 Procedure and Action Definition
The selection of nursing procedures and corresponding action steps is crucial for maintaining the relevance and usefulness of NurViD within the nursing profession and the broader healthcare community. The procedure selection adheres to a widely accepted nursing taxonomy, taking into account frequency of use and expert guidance. Then the actions involved in these procedures are gathered and standardized.
Figure 1: The examples for the annotated target action boundaries for Intravenous Blood Sampling and Modified Seldinger Technique with Ultrasound for PICC Placement procedures. The frames marked in colored boxes denote the annotated temporal boundaries for the target action steps.
Procedure selection. We compiled various nursing procedures from Nurselabs website [3] and nursing procedure books [8, 29, 31, 20]. With expert consultation, we identified 51 commonly employed nursing procedures that align with the requirements of most nursing scenarios. To validate their relevance and prevalence, we searched for corresponding videos on YouTube. Table 5 provides the full names and summarized abbreviations of these 51 procedures.
Action definition. We developed action steps for various nursing procedures based on college tutorials, and a nursing lecturer summarized the appropriate action labels by analyzing the action descriptions and video content. This was necessary for three main reasons: (1) There are variations in actions performed during specific nursing procedures, even within patients or instances of the same procedure, that require accurate representation of nuances. (2) The existence of diverse nursing procedure standards across countries and regions highlights the importance of establishing a unified standard. (3) Dealing with fine-grained video data from real-world nursing procedures requires rearranging action tags for the precise depiction of procedure nuances.
3.2 Online Video Crawling
Our objective in this stage was to gather sufficient videos demonstrating the pre-selected nursing procedures. By utilizing the extensive collection of medical instructional videos available on YouTube [6, 17], we acquire a wide range of videos without the need for third-party video production. To accomplish this, we queried YouTube using text-based searches for each procedure and obtained videos whose titles included the desired procedure keywords. To expand the video collection, we enhanced the search queries by including synonyms of each procedure. For example, Subcutaneous Injection Insulin can also be called Subcutaneous Insulin Administration, Subcutaneous Insulin Therapy, or abbreviated as SCII. Each video was downloaded at the highest resolution available. During video retrieval, we prioritized videos shorter than 20 minutes to limit the total storage.
3.3 Localization Annotation and Quality Control
In the NurViD dataset, each video is divided into multiple temporal segments, each of which contains only one action. Each action is annotated with its starting and ending timestamps as well as its frame position in the video. The annotation process is performed by undergraduates with medical and nursing backgrounds to ensure the accuracy and consistency of the annotations.
Employing nursing professionals. Data curation is an expensive process that typically involves extensive manual annotation. In some cases, datasets have employed a semi-automatic crowdsourcing approach for collection and annotation [12, 18, 40, 13]. For tasks that require greater reliability,
Figure 2: The average, maximum, and minimum number of action segments for each procedure.
(a) Video Duration Distribution
Figure 3: NurViD dataset duration statistics.
(b) Segment Duration Distribution
certain datasets rely on domain experts for annotation, albeit at a higher cost. In our study, we formed a medical team of 26 individuals, consisting of a nursing lecturer and 25 nursing majors from a medical college, to perform labeling. Over half of the students have at least three years of undergraduate education, possess extensive practical experience in nursing procedures, and have successfully completed the university's standardized nursing procedure assessment.
Invalid video filtering. Additionally, certain videos may contain irrelevant content due to inaccuracies in text-based retrieval. Thus, in the first stage of labeling, we excluded videos that fall into the following categories: 1) showcasing unrealistic environments (e.g., movies, animation), 2) providing only verbal descriptions instead of visual demonstrations, 3) featuring static images instead of continuous videos, and 4) lacking the specified procedure.
Action boundaries annotation. To ensure localization annotation quality, we followed a three-round annotation process: (1) Each annotator was assigned 2-3 nursing procedures based on video count and tasked with filtering out inappropriate videos, (2) After filtering, the action segments of each procedure video were annotated by three members, (3) Finally, cross-checking of annotation results between every two groups was conducted to identify and rectify errors and omissions. This process resulted in a minimum of three annotated action boundaries for each video. To ensure reliable annotations, we employed the complete linkage algorithm [10] to cluster and merge various temporal boundaries into stable boundaries that received multiple agreements. It is important to mention that a single video may feature multiple separate instances of the target action, leading to multiple boundary definitions. Several examples of annotated target action boundaries are shown in Figure.1.
3.4 NurViD Statistics
Our NurViD dataset is a comprehensive collection of nursing procedure videos that includes 1,538 videos (144 hours) demonstrating 51 different nursing procedures and 177 actions for recognition and detection tasks, which are summarized in Table. 5 and Table. 6 in the supplementary material. To facilitate the development and evaluation of algorithms, we trimmed the videos based on annotated action boundaries, resulting in 5,608 trimmed video instances that totaled 50 hours. The trimmed videos have an average duration of 32 seconds, while the untrimmed videos have an average duration of 337 seconds. Over  $74\%$  videos have HD resolutions of  $1280 \times 720$  pixels or higher. We observed a long-tailed distribution in the number of collected videos for both procedure and action.
In our study, we focus on three tasks using the NurViD dataset: (1) procedure classification in untrimmed videos, (2) procedure and action classification on trimmed videos, and (3) action detection on untrimmed videos. To establish reliable baselines for these classification and detection tasks, we employ state-of-the-art models that have demonstrated effectiveness in human action recognition and detection. We provide a comprehensive analysis of the baseline models, taking into consideration the specific challenges posed by the NurViD dataset, such as long-tailed class distributions. For each task, we formulate the problem in detail and evaluate the performance of the baseline models. The results of our analysis can guide the development of more accurate and robust models for fine-grained action recognition and detection in healthcare applications.
4.1 Procedure Classification on Untrimmed Videos
This task involves identifying nursing procedures from untrimmed videos, which typically consist of multiple standardized action steps that must be executed in a specific order and other unrelated parts. The goal is to explore the effectiveness of DL technology in retrieving specific nursing procedures from a large video library. By accurately classifying nursing procedures, we can provide healthcare professionals with a powerful tool for quickly accessing relevant videos.
Data settings. The examples from each procedure category are randomly divided into three sets:  $70\%$  for training,  $10\%$  for validation, and  $20\%$  for testing, resulting in 1,054 training, 173 validation, and 311 testing videos, respectively.
Class splits. To account for the long-tailed nature [47, 43] of the NurViD dataset, we divided the procedure classes into three splits: many, medium, and few, based on the number of videos for each procedure. The accuracy of each split is the average accuracy of the included procedures within that split. Specifically, the many category includes the top  $19.6\%$  most frequent classes, the medium category includes the middle  $43.1\%$  classes,
Baselines	Procedure Classification
Many 10	Medium 22	Few 18	All 50
SlowFast [14]	9.9	7.5	0.1	7.4
C3D [38]	10.7	5.1	1.8	7.7
I3D [9]	9.9	9.0	2.8	8.7
SlowFast*	19.9	10.2	5.0	13.5
C3D*	21.5	11.3	5.8	14.8
I3D*	19.8	12.5	5.6	13.1
Table 2: Per-class Top-1 accuracy for procedure prediction on untrimmed videos. The best performance for each split has been highlighted in **bold**.
and the few category includes the remaining  $37.3\%$  classes. The number of classes per split is presented in Table 2.
Baselines. We compare the performance of SlowFast [14], I3D [9], and C3D [38] models on this task. These models are evaluated in two versions: 1) Random initialization training and 2) Pre-training with weights from Kinetics 400 [23], which is a human action recognition dataset.
Results. The results for per-class accuracy are summarized in Table 2. We find that C3D [38] is able to achieve competitive results for all the splits. However, the best top-1 per-class accuracy is  $14.8\%$ , indicating that there is significant room for improvement in this challenging task. We also observe that transfer learning from the model that is pre-trained on Kinetics 400 [23] improves the classification accuracy for all splits. With the C3D model, this corresponds to a per-class accuracy gain from  $10.7\%$  to  $21.5\%$  for the many category and from  $5.1\%$  to  $11.3\%$  for the medium category. Despite this improvement, accurately predicting procedure categories remains a significant challenge.
**Discussions.** Based on the results of the classification benchmarks established on NurViD, we found that even models pre-trained on Kinetics 400 [23] cannot achieve satisfactory classification performance. We speculate that this may be due to several reasons: (1) videos are not always exclusively focused on nursing procedure activities and may contain other unrelated content, such as verbal instructions and brief introductions to devices; (2) some actions, such as handwashing, disinfection, and document, are commonly used in various procedures, which may cause the model to obtain similar features in different procedural videos, making it difficult to classify them. Overall, focusing on the main procedure actions in the video remains a challenging task.
Baselines	Procedure Classification	Action Classification	Joint Classification
Many 13	Medium 21	Few 17	All 51	Many 9	Medium 66	Few 87	All 162	Many 17	Medium 78	Few 224	All 302
SlowFast [14]	68.9	50.0	33.0	63.0	25.7	10.2	3.2	17.1	12.5	7.2	3.3	7.5
C3D [38]	70.1	48.8	33.0	63.9	22.9	9.3	2.9	15.9	13.8	7.3	3.5	7.7
I3D [9]	67.6	49.9	32.9	62.9	26.3	9.8	4.1	17.9	12.7	7.9	4.0	7.9
SlowFast*	71.2	61.8	39.0	68.9	29.8	15.5	7.9	21.1	21.2	9.4	5.6	12.8
C3D*	73.2	60.0	39.6	71.2	28.1	14.6	7.3	22.8	21.8	10.8	5.6	13.1
I3D*	70.7	60.4	40.9	70.0	31.3	14.8	8.2	21.5	19.5	9.9	4.7	12.5
Table 3: Per-class Top-1 accuracy (%) for the procedure, action, and their joint prediction on trimmed videos. * denotes the initialization from the model pre-trained on Kinetics 400 [23]. The best performance for each split has been highlighted in **bold**.
4.2 Procedure and Action Classification on Trimmed Videos
This task focuses on classifying both the primary action that occurs in a trimmed video and their associated procedures. By accurately classifying procedures and actions, building automated systems can automate the monitoring of each step in the nursing process, thereby helping to identify potentially missed diagnoses, nursing errors, and other issues. These systems can also help doctors and nurses quickly find useful nursing procedure videos, thereby improving their learning and work efficiency.
Data settings. The dataset was randomly partitioned, ensuring a balanced representation of examples from each procedure and action category. Specifically, we allocated  $70\%$  of the data for training (3,906 videos),  $10\%$  for validation (587 videos), and  $20\%$  for testing (1,122 videos).