SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andy Zhou $^{1*}$  Samuel Li $^{1*}$  Pranav Sriram $^{2*}$  Xiang Li $^{1*}$  Jiahua Dong $^{1*}$  Ansh Sharma $^{1}$  Yuanyi Zhong $^{1}$  Shirui Luo $^{3}$  Maria Jaromin $^{3}$  Volodymyr Kindratenko $^{1,2,3}$  George Heintz $^{4}$  Christopher Zallek $^{5}$  Yu-Xiong Wang $^{1,2,3}$
$^{1}$ Department of Computer Science, University of Illinois Urbana-Champaign  
 $^{2}$ Department of Electrical and Computer Engineering, University of Illinois Urbana-Champaign  
 $^{3}$ National Center for Supercomputing Applications, University of Illinois Urbana-Champaign  
 $^{4}$ Health Care Engineering Systems Center, University of Illinois Urbana-Champaign  
 $^{5}$ OSF Healthcare Illinois Neurological Institute - Neurology
https://uiuc-yuxiong-lab.github.io/YouTubePD
The healthcare and AI communities have witnessed a growing interest in the development of AI-assisted systems for automated diagnosis of Parkinson's Disease (PD), one of the most prevalent neurodegenerative disorders. However, the progress in this area has been significantly impeded by the absence of a unified, publicly available benchmark, which prevents comprehensive evaluation of existing PD analysis methods and the development of advanced models. This work overcomes these challenges by introducing YouTubePD – the first publicly available multimodal benchmark designed for PD analysis. We crowd-source existing videos featured with PD from YouTube, exploit multimodal information including in-the-wild videos, audios, and facial landmarks across  $200+$  subject videos, and provide dense and diverse annotations from a clinical expert. Based on our benchmark, we propose three challenging and complementary tasks encompassing both discriminative and generative tasks, along with a comprehensive set of corresponding baselines. Experimental evaluation showcases the potential of modern deep learning and computer vision techniques, in particular the generalizability of the models developed on our YouTubePD to real-world clinical settings, while revealing their limitations. We hope that our work paves the way for future research in this direction.
As one of the most prevalent neurodegenerative disorders, Parkinson's Disease (PD) affects over 10 million people worldwide, and the number of PD subjects is projected to double within the next 20 years [2]. Notably, PD is a progressive disease, with symptoms becoming increasingly pronounced and severe over time. Meanwhile, the diagnostic and treatment costs associated with PD are substantial, amounting up to $14 billion in the United States alone [2]. Therefore, there is an urgent need for developing AI-assisted systems for automated PD assessment in the healthcare and AI communities. Such systems can facilitate the recognition of undiagnosed people with PD, aid clinicians in their evaluation, and play a crucial role in the continuing care of people with PD by monitoring disease progression and tracking responses to therapies.
However, the progress in this area has been significantly impeded by the absence of a unified, publicly available benchmark. For AI-driven applications, the first and foremost endeavor lies in establishing
Figure 1: We propose YouTubePD, an open-source, multimodal benchmark on Parkinson's Disease (PD) analysis. Our dataset contains in-the-wild videos, audios, and facial landmarks. On the right, we show the three tasks on our benchmark: (a) facial-expression-based PD classification, (b) multimodal PD classification, and (c) PD progression synthesis.
a suitable benchmark. Ideally, the benchmark is openly accessible to facilitate methodology development, exemplified by the notable ImageNet benchmark that has substantially catalyzed object recognition performance [12]. In contrast, existing PD datasets are typically constructed in clinical settings and are thus kept private to protect patient privacy [3, 5, 17, 19, 25, 33, 41, 48]. In addition, the creation of PD datasets is a costly, time-consuming process, which requires a significant amount of effort for collecting patient data and curating annotations from qualified clinical experts.
In this paper, we overcome the aforementioned challenges by introducing an open benchmark that not only enables comparisons of existing PD analysis techniques, but also facilitates the development of advanced models that leverage state-of-the-art computer vision and deep learning techniques. Our key insight is that, instead of curating data from clinical settings as is normally the case, we exploit publicly available online resources and crowd-source existing videos featured with PD from YouTube.
These YouTube videos contain rich examination modalities, making our benchmark inherently multimodal. This property allows for the investigation of some particularly indicative symptoms associated with PD. More concretely, PD is often characterized by the manifestation of a motor-based symptom known as facial bradykinesia – the lessened movement of orofacial muscles and a reduction in spontaneous emotional expressions [8]. This results in a constant “mask-like” expression in PD subjects referred to as hypomimia [8], a sensitive biomarker for PD [1]. Following prior work [3, 5, 19, 25, 48], our benchmark considers hypomimia as a primary indicator for PD diagnosis. Moreover, our benchmark includes other symptoms experienced by PD patients, including tremors and speech changes. These additional symptoms are considered alongside hypomimia, providing a more comprehensive evaluation of the disease and capitalizing towards a multimodal solution.
Our contribution is described as follows:
1. To tackle the lack of a unified, open, and multimodal benchmark in PD analysis, we introduce YouTubePD. By leveraging YouTube videos of public figures experiencing PD, we obtain high-quality, in-the-wild data of Parkinson's symptoms. As shown in Figure 1, our YouTubePD benchmark consists of multiple modalities, including facial expression videos, facial landmarks, and audios, across  $200+$  subject videos containing dense and diverse annotations from a clinical expert. Our annotations encompass both video-level and frame-specific information with informative region and textual descriptions. In addition, our dataset spans a number of years for each subject, providing natural PD progression information.
2. We define three challenging tasks for our YouTubePD benchmark: Facial-expression-based PD classification, multimodal PD classification, and PD progression synthesis.
3. We present a comprehensive set of baselines for the three proposed tasks. In contrast to prior deep learning methodologies presented for PD analysis, our baselines leverage more recent, state-of-the-art techniques for enhanced video and multimodal understanding. We demonstrate that these models developed on our benchmark transfer well to real-world clinical datasets and exhibit strong performance on the PD classification task.
In this section, we discuss prior work and studies on facial expressivity and hypomimia in PD, as well as previous vision-based frameworks used to address PD classification.
Dataset	# Videos	# Images	# PD Subjects	# Healthy Controls	Open Access	Modality
Abrami et al. [3]	107	-	68	-	X	Video
Bandini et al. [5]	306	-	17	17	X	Video
Grammatikopoulou et al. [19]	-	6236	221	1071	X	Image
Jin et al. [25]	176	-	33	31	X	Video
Novotny et al. [33]	166	-	91	75	X	Video
Su et al. [41]	172	-	47	39	X	Video
FacePark-GITA [18]	270	-	30	24	X	Video
YouTubePD (Ours)	283	-	16	89	✓	Video, Audio, Landmark
Table 1: Comparison of statistics between datasets used in prior work and our benchmark. Our YouTubePD is the first open-access and multimodal benchmark for PD analysis.
Facial expressivity in PD. The connection between PD and facial expressivity has been thoroughly studied in prior work across a variety of experimental scenarios and controlled variables [9, 37, 40, 44, 47]. Such work explores the role of facial expressivity (both posed and voluntary expressions) and emotion recognition in patients in relation to their diagnosis of PD, primarily based on the UPDRS-III scale [16]. Follow-up work [33, 38, 44, 45, 47, 48] expands upon these seminal studies on facial expressivity in PD, further exploring its relationships with the subjective emotional experience, the correlation with PD severity, and de-novo conditions.
Video-based PD assessment. Video-based PD assessment techniques are typically categorized into geometric approaches [5, 19, 25, 33, 41] and appearance-based approaches [3, 17]. The former type employs low-dimensional, geometric features extracted from facial landmarks and action units (AUs) [5, 17, 19, 33, 41]. The latter type uses the raw visual information contained within the videos. Some of these approaches apply convolutional neural networks [3, 17], support vector machines [5], and long short-term memory (LSTM) models [25].
PD benchmarks. Previous studies commonly use their own (private or semi-private) benchmarks to evaluate the performance of their methods, lacking a shared benchmark for comparison. Table 1 provides a summary of these datasets, which often exhibit variations in size and distribution (e.g., mean age, gender, and disease progression). Consequently, comparing the performance of different methods becomes challenging, further hindering advancements in PD classification and analysis. This is a key motivation for our work in establishing a public and common benchmark.
Our objective is to identify not only the presence but also the severity of Parkinson's Disease (PD). With this in mind, we introduce YouTubePD, which stands as the first publicly available benchmark designed for PD analysis that (i) utilizes multimodal information including in-the-wild videos, audios, and facial landmarks, and (ii) enables the exploration of various task types including unimodal and multimodal PD classification, as well as PD progression synthesis. The comparison between our benchmark and datasets in various previous studies is summarized in Table 1. To further support comprehensive classification approaches, we offer video-level and region-level clinician annotations pertaining to PD diagnosis. In the interest of open access and wider research contributions, the basis of this benchmark is sourced from YouTube interview videos featuring public figures who are openly sharing their experiences with PD. Our work is among the first endeavors advocating for PD analysis using multimodal information. This approach aligns with the diagnostic techniques employed by human clinicians, who utilize a wide range of cues in order to diagnose PD. This convergence of multimodal information in our model mimics the human approach, thereby enriching the accuracy and depth of PD analysis. Our dataset collection and annotation pipeline is outlined in Figure 2.
Videos. We manually curate a list of 16 public figures with a confirmed and open PD diagnosis and collect multiple videos (spanning multiple years) of the individual before and after their diagnosis. We collect a total of 283 videos from YouTube. Each video is roughly 10 seconds long of an individual public figure speaking in an interview setting. Our corpus can be divided into three subgroups: 65 videos of public figures after their diagnosis of PD was made public, 68 videos of the same public figures several years before their diagnosis, and 150 videos of a broader healthy control group of public figures without PD. We collect and store both video and audio data. The videos are preprocessed by cropping and resizing with a facial keypoint detection model [29], so that clips are centered on the subject's face.
Figure 2: Our dataset collection and annotation pipeline. First, we compile a list of public figures who have publicly confirmed their PD diagnosis. We then source their videos from YouTube. From these videos, we handpick clips that are informative for PD detection. A clinical expert then reviews these clips, providing both video-level and region-level annotations, detailing the severity of their PD, and highlighting specific symptoms of the condition.
(a) An example annotation of a video from our dataset.
Figure 3: Representative examples for our video annotations and important facial regions.
(b) Examples of annotated frames with corresponding bounding boxes in increasing severity.
Annotations. Our annotations (Figure 3) are generated by a clinician. Each video has both (i) an overall severity label with 6 levels, with 0 denoting the absence of PD and 5 indicating a severe form of PD; and (ii) a confidence level label ranging from 1 to 10. A label of 1 indicates that the clinician was not confident in their assessment, while 10 indicates absolute confidence. For selected frames in each video, facial regions particularly informative for PD are also annotated. We define a set of 14 important facial regions for PD analysis, derived as a combination of facial creases [20] and other areas of facial movement observed in PD hypomimia. Each region is mapped to a distinct region index that describes the facial region used for the diagnosis and 10 distinct symptom indices (e.g., moving, apart, increasing) that describe the anomaly. In these frames, (i) a polygon bounding box is drawn around the region, (ii) a text caption is used to summarize the region and anomaly, and similar to the video-level annotations, (iii) a severity label and (iv) a confidence label are provided for each region. These annotations and indices are summarized in Section B of the Appendix.
Audios. We provide audios as an additional modality. Each video clip has its associated audio sharing the same label. The audio clips are preprocessed to account for cases where multiple speakers are present or significant background noise or music overpowers the intended speaker. We utilize source separation [42] to clean any segments that have either of these two violations, isolating the desired subject's audios from other speakers and background noise. Though the audio modality has a distinct set of UPDRS standards [16], we do not re-announce the ground truth label for the audios but use the holistic severity and confidence annotation based on the videos instead. This aims to ensure the label consistency between the audios and other modalities to facilitate multimodal PD classification.
Landmarks. Our dataset incorporates facial landmarks extracted from video frames to leverage facial motion. Using the landmark detection model from Face++ [29], we extract 106 2D point landmarks from each frame. The landmarks can be grouped into 7 parts: contour, left/right eyebrow, left/right eye, nose, and mouth.
4 Tasks
We propose three tasks on YouTubePD: facial-expression-based PD classification, multimodal PD classification with facial expressions, audios, and facial landmarks, and PD progression synthesis. We describe our tasks, as well as the dataset splits and evaluation metrics for each task.
4.1 Facial-Expression-Based PD Classification
Task description. Following prior work [17, 18] and clinical findings [25], we focus on facial expression as the primary modality for diagnosing PD. In this task, we aim to analyze the videos of facial expressions in our dataset and obtain a classification result for either a binary classification of PD or healthy, or a multiclass classification of PD severity on healthy or 5 severity levels.
Dataset split. Due to the imbalanced distribution between PD-positive and PD-negative videos, we use a relatively small training set and a relatively large evaluation set. For the training set, we randomly sample 36 PD-positive and 36 PD-negative videos. We use the remaining 211 videos for the evaluation set. Due to the small number of annotated PD-positive samples, we use K-fold cross-validation on the training set for hyperparameter tuning.
Evaluation metrics. The PD classification task on our benchmark is particularly challenging, due to its strong imbalance and limited number of positive samples. This mirrors the medical application setting of PD and presents a challenge: a model whose errors consist of mostly false negatives might still achieve a high accuracy due to the small number of positive samples, but may misdiagnose a patient with PD. This is a more harmful error than a false positive. Therefore, the conventional classification accuracy may not be the best choice for our setting. We mainly focus on additional metrics, in particular, F1 and AUROC. F1-score calculates the harmonic mean of precision and recall. For multiclass classification, we report the weighted F1-score computed with a one-vs-rest (ovr) approach. AUROC computes the area under the receiver operating characteristic curve for a set of decision thresholds and reports a summarized metric through some averaging policy (e.g., macro).
Meanwhile, we classify the severity of PD, which is an ordinal attribute. For example, classifying severity 4 as 3 is more accurate than classifying it as 2. Therefore we also provide a Mean Squared Error (MSE) to account for this issue. To summarize, we recommend prioritizing F1-score and AUROC, while also paying attention to the MSE error.
4.2 Multimodal PD Classification
In this task, we consider the combination of multimodal information for PD classification. The multimodal inputs in our study include facial expression videos, facial landmarks, and audios. The task aims to obtain a binary or multiclass severity prediction from the collaboration of all available modalities as input. We follow the identical data split and evaluation metric as the facial-expression-based PD classification task. We also directly use the holistic label from the facial expression video as the ground truth label for each modality derived from that video.
4.3 PD Progression Synthesis
Task description. We further propose a task of PD progression synthesis, which aims to simulate the symptoms of PD in facial expressions given images of healthy individuals. Previous work on image translation primarily focuses on texture alterations derived from a reference image or content modifications from a pretrained conditional model [11, 15, 30, 35, 49, 50]. Our PD progression synthesis task, however, introduces a more challenging aspect by incorporating more nuanced PD-specific information for the transfer. Our task naturally makes use of the PD progression information provided in our dataset, which contains sets of images depicting individuals in both healthy and PD states across an extended time frame. Note that these image pairs from healthy to PD states are not strictly aligned, which further increases the task difficulty. Complementary to the discriminative tasks in Sections 4.1 and 4.2, our synthesis task provides a comprehensive overview of the progression and manifestation of PD.
Dataset split. We divide the data by individual subjects, allocating 11 for training and 5 for evaluation. Our experiment validates that this split is sufficient for training and produces reliable evaluation results.
Evaluation metrics. To establish baselines for our synthesis or "style" transfer task, we employ a variety of metrics that assess the quality of synthesized facial images. Our evaluation begins with the Fréchet Inception Distance (FID) score computed between the source and generated images, serving as an indicator of visual fidelity and consistency. Subsequently, we devise a facial content change evaluation rooted in the paired healthy and PD states of the same individual, called direction score. Specifically, we use a pretrained VGGFace [36] model  $\phi$  to extract the instance-level feature from each image. For each healthy frame  $N_{i}$ , we first identify its corresponding frame  $P_{i}$  from the same
Figure 4: Our baseline methods for the facial-expression-based PD classification (left) and multimodal PD classification (right) tasks.
individual's PD frame with the highest feature similarity. Then, given the generated image  $N_{i}^{\prime}$ , the direction score is calculated as
$$
D = \frac {1}{M} \sum_ {i = 1} ^ {M} \operatorname {c o s i n e} \left(\phi \left(N _ {i} ^ {\prime}\right) - \phi \left(N _ {i}\right), \phi \left(P _ {i}\right) - \phi \left(N _ {i}\right)\right), \tag {1}
$$
where cosine denotes the cosine similarity and  $M$  is the number of frames. Finally, we assess a CLS score based on classification accuracy using a pretrained PD classification model in Section 4.1.
5 Methods
In this section, we propose baseline methods for each of our three tasks. For facial-expression-based classification, we propose an attention-based architecture that leverages both region and video annotations for stronger and more interpretable performance. For multimodal PD classification, we establish a simple fusion pipeline to exploit different modalities and improve performance. Finally, for PD progression synthesis, we make comparisons of various image translation models. The experimental results reveal limitations of different methods and the challenging nature of the tasks. We include illustrations of our baseline methods for the first two tasks in Figure 4.
5.1 Facial-Expression-Based PD Classification
We develop a simple but effective baseline method for this task. Apart from leveraging the most suitable pretrained representation, we integrate the region-level information explicitly alongside the holistic video information. Moreover, we apply an attention-based classifier and use a novel multiclass hierarchy-guided loss to train our baseline model. Our pipeline is in Figure 4.
PD representation learning. Due to the lack of annotated PD data, we find that it is beneficial to transfer representations learned from related domains to PD classification. To this end, we use a ResNet50 [22] pretrained on VGGFace [36], a face dataset with a wide range of subjects, as our frozen feature backbone. Pretraining on the base domain helps the model learn a wide range of identities across variations in pose, environment, and demographics. We find that for our task, pretraining on general facial recognition outperforms backbones trained on video action recognition as well as facial expression recognition, contrary to [18].
PD informative regions. Our PD informative regions are based on areas of the face which experts typically use to diagnose patients. Our quantitative results show that the use of region features instead of entire frames not only improves performance, but also reduces the tendency to learn spurious features. Correspondingly, we use a pretrained facial landmark classification module and RoIAlign [21] to extract a feature map for each of the 14 PD informative regions from the feature map produced by the backbone from each video.
Attention with video-level conditioning. Following feature extraction, we process the feature maps with a learnable positional embedding and a linear projection of the global feature map to the same dimension as an individual region feature map. Next, we pass our region features through two multi-headed attention blocks. Following TimeSFformer [7], we use divided spatial-temporal attention, which we apply on the region feature maps instead of raw pixels. This consists of an attention block that applies self-attention over the individual regions in each frame (spatial), followed by an attention block that applies self-attention over successive frames (temporal). We include ablation studies on the use of region annotations, region information, and attention to verify each component of our method.
Multitask hierarchy-guided loss. In addition to region features, we leverage region annotations to further improve performance. After attention, we extract the class embedding corresponding to each
Model	Top-1 Acc ↑	F1 ↑	AUROC ↑	Recall (Binary) ↑	MSE (Multiclass) ↓
VGGFace [22]	83.56(±0.84)/78.20(±3.13)	0.56(±0.01)/0.23(±0.02)	0.86(±0.01)/0.68(±0.01)	0.79(±0.01)	2.29(±0.77)
Ours	88.00(±1.88)/77.03(±3.36)	0.59(±0.02)/0.25(±0.01)	0.92(±0.01)/0.74(±0.02)	0.85(±0.05)	2.25(±0.45)