SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Binary/multiclass classification results on YouTubePD with facial expression. We observe superior performance on F1, AUROC, and MSE with our method.
Modality	Top-1 Acc ↑	F1 ↑	AUROC ↑	Recall (Binary) ↑	MSE (Multiclass) ↓
VGGFace [22]	83.56(±0.84)/78.20(±3.13)	0.56(±0.01)/0.23(±0.02)	0.86(±0.01)/0.68(±0.01)	0.79(±0.01)	2.29(±0.77)
Landmark (Ours)	56.37(±1.72)/51.69(±1.94)	0.32(±0.02)/0.26(±0.02)	0.72(±0.02)/0.68(±0.01)	0.81(±0.05)	4.87(±0.28)
Audio (Ours)	54.84(±1.63)/47.79(±1.66)	0.27(±0.02)/0.16(±0.01)	0.66(±0.01)/0.50(±0.02)	0.70(±0.04)	6.26(±0.39)
Multimodal (Ours)	70.61(±2.13)/82.75(±2.85)	0.61(±0.02)/0.28(±0.02)	0.87(±0.02)/0.80(±0.03)	0.83(±0.04)	1.40(±0.25)
Table 3: Binary/multiclass classification results on YouTubePD in the multimodal setting. Multimodal fusion further improves the performance over unimodal baselines, even when additional modalities have lower performance than the primary modality of facial expression.
region. We add an additional multi-layer perception (MLP) head for each of the 14 regions and add the corresponding cross-entropy loss to the overall loss objective. We convert the region annotations to binary annotations, where 1 indicates that the region is indicative of PD and 0 indicates that the region is normal. We also extract a class embedding corresponding to the entire video, which is passed through a final MLP head to obtain the video-level class distribution. This is trained with a hierarchy-guided loss function demonstrated by the weighted cross-entropy loss, which we add to the region-level loss,
$$
\operatorname {L o s s} = \lambda l _ {\text {b i n}} (\theta (x), y _ {\text {v i d}}) + (1 - \lambda) l _ {\text {m u l t i}} (\theta (x), y _ {\text {v i d}}) + \frac {1}{| R |} \sum_ {r \in R} l _ {\text {r e g}} (\theta (x _ {r}), y _ {\text {r e g}}), \tag {2}
$$
where  $x$  is an input image,  $\theta$  is the model,  $y_{\mathrm{vid}}$  is the video-level annotation,  $y_{\mathrm{reg}}$  is the region-level annotation,  $l_{\mathrm{bin}}$  and  $l_{\mathrm{multi}}$  are the cross-entropy losses on binary and multiclass labels respectively,  $l_{\mathrm{reg}}$  is the cross-entropy loss on region labels,  $R$  is the set of regions, and  $\lambda$  is a trade-off hyperparameters. We use both the original multiclass annotations and binary annotations for the video-level loss. We freeze the first three stages of the pretrained ResNet50 backbone and train the additional layers, as well as our attention blocks and MLP heads, end-to-end with this loss.
5.2 Multimodal PD Classification
We provide a baseline multimodal fusion method for PD classification. As shown in Figure 4, we first obtain predictions from each modality, and then use a logit fusion step to fuse them together considering their respective confidence.
Single modal prediction. For the facial expression video modality, we utilize the same encoder as in the facial-expression-based PD classification (Section 5.1), which yields a 2,048-dimensional feature representation. We obtain the final prediction with a linear classifier. For the landmark modality, we select 10 frames from the video, extract their facial landmarks, and classify them using an MLP. The features from all selected frames are concatenated for classification. For the audio modality, we extract the features using a pretrained masked auto-encoder applied on the spectrogram representations of the audios [32]. After averaging across all frames to get a 768-dimensional video-level feature representation, an MLP is trained to obtain the video-level logits.
Fusion strategy. We employ a simple but effective fusion strategy. For the video and landmark modalities, rather than using concatenation for the classification, we predict each frame independently. Then we sum the predictions of 10 frames to obtain two logits representing the video modality and landmark modality. After that, we apply  $\mathcal{L}_1$  normalization to the logits of each modality. Thus, we treat the logits for different modalities equally. Then we directly sum them up and determine the final prediction based on the max fused logits. During training, the same multitask hierarchy-guided loss is adopted. For the facial expression and audio modules, all the pretrained weights are frozen.
5.3 PD Progression Synthesis
We investigate a wide spectrum of image translation methods, including generation-based generative adversarial networks (GANs), translation-based GANs, and recently emerging diffusion models. Specifically, for generation-based GANs, we adopt [11, 26, 34]. We use their pretrained weight on FFHQ [27] and finetune it on our positive training data. Then, we project the PD-negative face to the
Metric	Generation-Based GAN	Translation-Based GAN	Diffusion Model
ADA [26]	Few-Shot [34]	JoJoGAN [11]	HRFAE [49]	CycleGAN [50]	CUT [35]	SD [39]
FID ↓	87.08	94.60	129.61	37.65	73.82	46.63	72.12
CLS ↑	34.29	26.67	37.74	11.32	26.42	11.32	66.98
Direction ↑	0.2874	0.2912	0.3219	0.2021	0.2832	0.3237	0.0491
Table 4: Quantitative comparisons on PD progression synthesis. Different generative methods exhibit distinct behaviors under three metrics.
Model	Top-1 Acc ↑	F1 ↑	AUROC ↑
VGGFace [22]	79.4	0.24	0.66
Ours	76.4	0.25	0.71
Table 5: We observe similar multiclass classification performance, when conducting a leave-one-subject-out analysis over three patients.
learned image space using the GAN model. For GANs designed for image translation, we either use their official pretrained model [49] or directly train from scratch [35, 50]. For the diffusion models, we finetune Stable Diffusion [39] with LoRA [24] and use SDEdit [30] for image translation.
In this section, we present the results of our baselines and proposed methods. We find that modern computer vision methods can be applied on YouTubePD and transfer well to real clinical applications.
Main results: Facial-expression-based PD classification. We report the average results and standard deviations over 5 different runs. The results are summarized in Table 2. For the baseline, we use a ResNet50 pretrained with VGGFace and finetuned on our dataset with linear probing. This obtains relatively strong performance, suggesting the efficacy of facial representations for PD diagnosis. In both the binary and multiclass settings, our proposed attention-based model outperforms the pretrained VGGFace baseline on most metrics.
Main results: Multimodal PD classification. We additionally provide baseline results for the other modalities as described in Section 5. From Table 3, we note that either modality alone ends up being considerably weaker than our primary modality of facial expression, and the fusion result achieves better performance in both the multiclass and binary settings. Our results primarily serve to demonstrate the potential for achieving better performance from the additional modalities in tandem with facial expressions, despite their weaker performance when used in isolation.
Main results: PD progression synthesis. The synthesis results of various baselines are shown in Table 4 and Figure 5. FID represents the consistency of the image translation. The CLS score and direction score focus on the semantic change in different aspects. Generally speaking, generation-based GANs often obtain balanced results on CLS and direction metrics. However, they fail to achieve low FID, since the image needs alignments to pretrained data. Translation-based methods can often get high consistency, but are easy to focus on other modifications rather than PD progression. The diffusion models achieve a significantly higher CLS score; however, they tend to overfit the training data and create faces that are similar to training images but do not retain the facial features of the test image. Thus, they will largely change the faces in test images, resulting in a low direction score. Our extensive comparisons reveal the substantial limitations of state-of-the-art generative models in extracting very fine-grained and subtle information for image translation, further highlighting the importance of our benchmark.
Analysis: Improvements are robust to different dataset splits. In Table 5, we present the results when we train the facial expression model on a modified training and test split, where we ensure there is a PD-positive patient only represented in the test split. We report the average multiclass results of three such leave-one-subject-out splits. In these new splits, we still observe that our method outperforms the baseline under the F1 and AUROC metrics. In the leave-one-subject-out setting, the AUROC of our method improves by 0.05 and F1 improves by 0.01. This improvement is similar to our original setting (AUROC by 0.06 and F1 by 0.02). Please note that similar to the main results in Table 2, the Top-1 accuracy is not indicative of model performance, because of the presence of a large amount of negative samples.
ADA
Few-Shot
DF
JoJoGAN
$\therefore m = \frac{3}{11}$
HRFAE
Figure 5: Qualitative comparisons on PD progression synthesis. The diffusion model yields superior results; however, it may inadvertently introduce inaccurate alterations to the face.
CycleGAN
CUT
Stable Di
PD Positive
Model	Top-1 Acc ↑	F1 ↑	AUROC ↑
VGGFace [22]	69.31/71.29	0.42/0.45	0.79/0.81
Ours	68.31/77.22	0.52/0.61	0.92/0.96
Table 6: Binary classification results on real-world patients in the speech test set and the facial activation set [23]. Our method consistently outperforms baselines on real-world PD data.
Model	Top-1 Acc ↑	F1 ↑	AUROC ↑
AffectNet-8 [31]	76.00	0.45	0.86
VGGFace	83.56	0.56	0.86
Table 8: Ablation study on facial expression classification pretraining, in terms of binary classification with the AffectNet-8 backbone. We observe a decrease in performance compared with when using our facial recognition pretraining.
Model	Binary Acc ↑	Multiclass Acc ↑
I3D [10]	70.60	45.64
C2D [46]	69.05	40.96
X3D-XS [13]	74.17	42.77
SlowFast-8x8 [14]	63.81	44.52
VGGFace	83.56	78.20
Table 7: Ablation study on action recognition pretraining. We observe significantly lower performance than using our facial recognition pretraining.
Model	Top-1 Acc ↑	F1 ↑	AUROC ↑
Video / Temporal Attention	77.14	0.49	0.91
Region / Spatial Attention	82.86	0.56	0.90
Ours	88.00	0.59	0.92
Table 9: Ablation study on binary classification results using various attention schemes. Our spatial-temporal attention performs the best.
Generalizability: From YouTube to real patients. From a healthcare perspective, a crucial and natural question arises: Do the models developed in our YouTubePD benchmark generalize to real-world PD diagnosis? To answer this question, we train our model on YouTubePD and test it on a private dataset of clinical PD patients [23]. This dataset consists of two sets of videos, a speech test of 18 PD-positive and 83 PD-negative patients, and a facial activation test of 19 PD-positive and 85 PD-negative patients. From Table 6, our proposed method has higher F1 and AUROC scores than the baseline on this dataset. In addition, the t-SNE [43] visualizations in Figure 11 of the Appendix show that the PD-positive and PD-negative samples are better separated for our approach. These experimental results validate the efficacy of our benchmark for real-world applications.
Ablations. We conduct ablation studies to investigate the design choices and components of our proposed facial-expression-based PD classification method. These studies specifically focus on understanding the benefits of different types of pretraining and attention for PD classification.
Action recognition pretraining. We investigate the transferability of generic video representations to PD classification. We finetune various action recognition architectures pretrained with Kinetics400 [10] on YouTubePD with linear probing. From Table 7, we observe much lower performance compared with generic facial representations from VGGFace [36], despite using a simpler architecture.
Facial expression classification pretraining. We also investigate the use of a backbone pretrained for facial expression classification, a more fine-grained task than facial recognition. We use a ResNet50 pretrained on AffectNet-8 [31] as our backbone, and finetune it on YouTubePD with linear probing. From Table 8, we observe lower performance with the AffectNet-8 backbone, suggesting that generic facial representations transfer more easily to PD classification.
Attention types. We investigate the use of attention over regions and attention over frames. This is combined in our proposed facial-expression-based method. From Table 9, we observe similar overall performance for either temporal or spatial attention and improved performance when combined.
Clinical impact. The long-term goal of our work is to develop a Parkinson's Disease (PD) early screening tool to aid primary clinicians in the earlier recognition of persons exhibiting physical signs
that may indicate an evolving Parkinson's syndrome. Those persons may then undergo additional neurological evaluation as clinically indicated. This will help patients to be diagnosed and treated sooner. Many patients with Parkinson's, in retrospect, exhibited subtle signs of Parkinson's for months or even years. The signs were not recognized by the patient, their family, or primary clinicians and the signs instead were attributed to aging, and diagnosis was delayed. We hope that our effort could eventually inspire such a PD early screening tool to detect PD symptoms at an early stage, and the patients could get timely treatments.
Annotation reliability. In the construction of our dataset, we made use of annotation information beyond what a single annotator provided. Since we did not construct the dataset completely from scratch but repurposed existing YouTube videos, we explicitly utilized the topic of these YouTube videos and their meta-information. Therefore, we precisely know which videos are PD-positive and which ones are negative, and there will very unlikely be mistakes in the annotation of video-level PD labels. In addition, the longitudinal meta-information associated with the YouTube videos provides useful prior knowledge about the PD severity. Empirically, models trained on our annotated dataset generalize to real-world clinical PD data as shown in Table 6. Our method achieves an F1 score that is  $10\%$  and  $16\%$  higher than the baseline on two different sets of real patient data, respectively. These results reflect that our annotation should be reliable.
Limitations and future work. The dataset presented in our benchmark possesses its own caveats, particularly in relation to its small size and limited diversity due to the constraint of publicly available videos. While our benchmark serves as a strong first step, there is still a need for further comprehensive datasets and benchmarks to establish thorough and holistic evaluation of models developed for PD analysis. In addition, an important question remains regarding the generalizability of models developed on our benchmark to critical, real-world medical applications across various settings. Further exploration and analysis of this aspect are necessary to advance towards the actual deployment of these models in clinical scenarios.
Beyond providing the dataset, our contribution includes the establishment of a crucial protocol for collecting a PD dataset. This ensures that our dataset can be readily expanded in the future with more public figures or individuals from additional geographical regions and across different social media platforms other than YouTube. We hope that our work will inspire efforts to create larger PD benchmarks with more annotators. We discuss our limitations and future work in more detail in Section E of the Appendix.
Ethics. Our research involves the analysis of videos featuring public figures with PD. The videos, derived from YouTube, were publicly shared by figures who had willingly discussed their PD condition. To ensure ethical considerations, we sought explicit consent from these figures. In addressing concerns of data privacy, the research protocol was reviewed and approved by the Institutional Review Board (IRB) at University of Illinois Urbana-Champaign (IRB approval number: 24426). While we acknowledge the potential biases and limitations in solely relying on facial or audio information for PD diagnosis, our main objective is to inspire tools for early detection using easily accessible media like webcams. This initiative does not negate the need for comprehensive medical diagnostics. We stress that our efforts aim to further the understanding of PD and its facial expression impacts. The collected data are shared with researchers who possess ethical training and commit to adhering to high standards. All data are hosted on a dedicated and secure platform, and the code is made available on GitHub. No conflicts of interest exist among the study's contributors. More discussion on the ethical aspect of YouTubePD is included in Section F of the Appendix.
We introduce YouTubePD, the first publicly available multimodal benchmark for Parkinson's Disease (PD) analysis. Within this framework, we propose several important discriminative and generative tasks along with corresponding baselines, showcasing how a range of modern machine learning and computer vision techniques is leveraged and extended to advance AI-assisted systems for automated PD diagnosis. Furthermore, we demonstrate that models trained on YouTubePD perform well on clinical data, indicating potential for real-world medical applications. Notably, the methodology and protocol developed in constructing YouTubePD can be adapted to address other healthcare problems, where a public, standardized benchmark is missing. We encourage further exploration in this direction and hope that our benchmark will facilitate a more seamless transfer of advancements in machine learning and computer vision to impactful medical research and clinical applications.
This work was supported in part by the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign through the NCSA Fellows program, NSF Grant 2106825, and NIFA Award 2020-67021-32799. We thank Minh Do and the team for providing real-world Parkinson's patient data.
In this appendix, we include (1) discussion regarding our YouTubePD benchmark detail, release, and licensing, (2) additional analysis of the dataset, (3) additional illustration and details about our methods, (4) additional experimental results and analysis, (5) more discussion on limitations and future work, and (6) potential negative societal impact.
A Benchmark Detail, Release, and Licensing
We provide our code and benchmark at https://uiuc-yuxiong-lab.github.io/YouTubePD. We release our benchmark under the CC0 license. Here, we describe how we publicly release our benchmark:
1. We include all the code used in the process of converting publicly available YouTube videos into our benchmark.
2. We include all our annotations and extracted landmarks. Note that offensive content is not included in our dataset, as all the sources are publicly available interviews of public figures speaking openly about their experiences with Parkinson's Disease.
Despite that all the videos used in our benchmark are publicly available YouTube videos, we are also actively taking steps to approach the public figures involved to respect their autonomy and privacy. This ensures that we uphold the highest standards of ethical data usage. We strive to balance open access and reproducibility with respect for privacy, all while providing a resource that could significantly advance the analysis and understanding of Parkinson's Disease (PD).
B Dataset: Additional Analysis
Region/Severity	0	1	2	3	4	5
Overall Video	187	15	8	10	17	7
Forehead	187	12	3	4	16	18
Left nasolabial fold	187	9	6	10	20	10
Right nasolabial fold	187	11	7	10	21	8
Right lip crease	187	1	0	3	7	3
Left lip crease	187	1	0	2	10	3
Left outer eye	187	4	2	1	6	17
Right outer eye	187	2	2	2	5	16
Between eyebrows	187	5	1	3	5	3
Right above between eyebrows	187	3	3	4	12	5
Right eye	187	15	4	3	8	6
Left eye	187	16	4	3	8	6
Mouth	187	1	0	1	5	0
Right eyebrow	187	6	5	8	11	12
Left eyebrow	187	6	4	7	10	12
Total Annotations	2808	107	49	71	161	126
Table 10: Distribution of severity labels in YouTubePD for the overall video-level analysis and for all 14 facial regions, with 0 denoting the absence of PD and 5 indicating a severe form of PD.
Country	Count	Race	Count	Gender	Count
United States	12	White/Caucasian	13	Male	15
United Kingdom	3	Black/African descent	3	Female	1
Canada	1
Table 11: Country/race/gender statistics of PD-positive public figures in YouTubePD.
Country	Count	Race	Count	Gender	Count
South Africa	1	South African+Swiss-German	1	Male	68
United States	52	Black/African Descent+Filipino	1	Female	23
United Kingdom	12	Black/African Descent	11
Israel	3	White/Caucasian	63
Russia/Canada	1	Indian	1
Sweden	2	Latinx	4
Kenya/Mexico	2	Asian	9
Brazil	2	Black/African Descent+Samoan	1
Serbia	1
South Korea	2
Puerto Rico	1
Mexico	1
Japan	1
Canada	4
Denmark	1
Russia	2
Germany	1
France	2
Table 12: Country/race/gender statistics of healthy control or PD-negative public figures in YouTubePD.
In Table 10, we summarize the severity label distribution in YouTubePD. This includes severity labels for the overall subject in each video and severity labels for each of the 14 important facial regions informative for PD analysis. The number of annotations varies depending on severity levels and regions.
We also summarize the demographic distribution in YouTubePD, split between PD-positive and healthy control (HC), or PD-negative, subjects. Table 11 provides the country, race, and gender statistics of PD-positive subjects in YouTubePD. Similarly, Table 12 provides the country, race, and gender statistics of HC subjects in YouTubePD.
We would like to provide additional details in our annotation process, particularly regarding how we denote the severity of PD. Our annotation strategy utilizes a detailed scale, ranging from 0 to 5, where 0 signifies a healthy individual, and 5 corresponds to severe PD. We do not apply the Unified Parkinson's Disease Rating Scale (UPDRS) [16] for facial expression. This decision is based on the clinician's suggestion, since an accurate UPDRS facial expression rating would require more information (e.g., observing the subject's facial expression pattern at rest or when not talking) than facial expression videos contain. This strategy also allows for a finer classification. In addition, we do not apply UPDRS because facial expression and audio have distinct UPDRS standards. We instead use the holistic severity and confidence annotation based on the video. Doing so ensures the label consistency between the audio data and other modalities, thereby facilitating multimodal PD classification.
In addition, we provide (i) illustrations of our facial landmarks and regions in detail in Figures 6, 7, and 8; (ii) the longitudinal statistics of our PD videos showing their distribution in time in Figure 9.
B.2 Are Annotated Regions Correlated to PD Severity?
To understand if the annotated regions are informative for PD severity, we investigate the correlation between the 14 annotated facial regions and a patient's annotated PD severity level. We accomplish this by training a linear classifier that takes as input the annotated informative region index and predicts the severity level. More specifically, for each video, the input is a binary vector which maps the clinician-annotated facial region indices, while the output is matched to the annotated severity level of the video. As a control experiment, we instead train on random region-level annotations as input. We find that the linear model trained on the actual region annotation achieves  $70\%$  test accuracy, while the model trained on random annotations achieves  $15\%$  test accuracy. This validates that the severity level is predictable from the annotated informative regions, but not from random region annotations. Therefore, the annotated regions and PD severity are correlated. This is also
Figure 6: Original landmark extraction.
Figure 7: Interpolated landmarks.
Figure 8: Visualized regions from interpolated landmarks.