SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Howard Zhong $^{1,2}$ , Samarth Mishra $^{2,3}$ , Donghyun Kim $^{2,6}$ , SouYoung Jin $^{4}$ , Rameswar Panda $^{2}$ , Hilde Kuehne $^{2,5}$ , Leonid Karlinsky $^{2}$ , Venkatesh Saligrama $^{3}$ , Aude Oliva $^{1,2}$ , Rogerio Feris $^{2}$
$^{1}$ MIT,  $^{2}$ MIT-IBM Watson AI Lab,  $^{3}$ Boston University,  $^{4}$ Dartmouth College,  $^{5}$ Goethe University,  $^{6}$ Korea University
Pre-training on massive video datasets has become essential to achieve high action recognition performance on smaller downstream datasets. However, most large-scale video datasets contain images of people and hence are accompanied with issues related to privacy, ethics, and data protection, often preventing them from being publicly shared for reproducible research. Existing work has attempted to alleviate these problems by blurring faces, downsampling videos, or training on synthetic data. On the other hand, analysis on the transferability of privacy-preserving pre-trained models to downstream tasks has been limited. In this work, we study this problem by first asking the question: can we pre-train models for human action recognition with data that does not include real humans? To this end, we present, for the first time, a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Furthermore, we propose a novel pre-training strategy, called Privacy-Preserving MAE-Align, to effectively combine synthetic data and human-removed real data. Our approach outperforms previous baselines by up to  $5\%$  and closes the performance gap between human and no-human action recognition representations on downstream tasks, for both linear probing and fine-tuning. Our benchmark, code, and models are available at https://github.com/howardzh01/PPMA.
Action recognition is the task of classifying human actions from video sequences [9, 8, 42, 54, 70]. Pre-training action recognition models on large-scale datasets significantly improve the accuracy of these models, especially when generalizing to downstream tasks with limited data [1, 29, 15]. Furthermore, with the advent of transformers in vision [25, 69], pre-training on large-scale datasets has become increasingly important for high performance [36].
While the use of large-scale datasets can yield significant accuracy improvements, it poses several problems. First, it is costly and time-consuming to curate and annotate these large-scale real-world datasets. Unsupervised [50] and semi-supervised pre-training [2] can leverage large datasets without the cost of labeling, but it does not address the privacy and ethical concerns related to pre-training with large datasets. Visual characteristics such as skin tone and gender can lead to cognitive biases in models trained from large-scale datasets. It is difficult to control for these biases [10, 74]. Furthermore, many large-scale datasets contain videos including humans, who may be identifiable based on their face, clothing, or other sensitive attributes. In particular, most existing video datasets contain images of people that were collected without consent. This is a legal issue, as personal data is protected by legislation such as GDPR [3]. Moreover, it is difficult to control undesirable biases related to gender
Pre-training
Real Data with Humans
Preprocessing:
Remove Humans
Real Data without Humans
Playing chess
Nausea vomiting
Fall to the floor
Pointing with finger
Telephoning
Waving
Walk hold hands
Evaluation: Downstream Tasks
UCF101
Playing golf
HMDB51
Archery
Figure 1: Our privacy-preserving action recognition benchmark. Real-world videos are initially pre-processed to remove humans and then combined with synthetic data containing virtual humans for pre-training. The transferability of these pre-trained models are then evaluated on a diverse set of downstream human action recognition tasks.
Mini-SSV2
Spilling something next to something
Diving48
Reverse dive no-twist tuck
IkeaFA
Flip table
UAV-Human
Put something into a bag
and race in existing large-scale datasets. As an example, state-of-the-art models may fail to predict actions such as "woman snowboarding" [39], given that the training set contains more videos of men performing this action. Finally, membership reconstruction attacks retrieve training data from a trained model and can expose identifying information [35, 62].
Existing works have proposed various ways to alleviate these problems. Methods include downsampling [21], obfuscation [11, 82], and adversarial training to modify the image [22, 58, 77]. However, these works do not analyze the transferability of the representations in pre-trained models to downstream tasks. As membership reconstruction attacks that recover training data from a pre-trained model are becoming increasingly sophisticated [35, 62], there is a growing need to determine how to leverage privacy-preserving data to pre-train models. The PASS dataset [5] was recently created to study representations learned from images without personally identifiable information for image classification.
The goal of this work is to pre-train representations for human action recognition without real humans in the data. SynAPT [75] is one of the few works that address this privacy-preserving transferability problem but only explores the use of synthetic data. It proposed a benchmark to evaluate the transferability of action recognition features pre-trained on synthetic data on different downstream tasks. SynAPT showed that pre-training on synthetic data was useful for downstream tasks with low scene-object bias but not as effective for tasks with high scene-object bias (i.e., when action categories are likely to be recognized solely by the scene or objects in the background, instead of the temporal dynamics of the action itself). High scene-object bias tasks require better real-world embeddings of the context surrounding the action, but this may not necessarily require the human to
be present in the video. In this work, we demonstrate that we can solve this problem while preserving privacy by obfuscating humans in real videos.
We pre-train models with a combination of synthetic videos and human-removed real videos so that the synthetic data will teach the model temporal dynamics of the action while the human-removed real data will help it learn contextual features in the scene. To the best of our knowledge, no previous work has analyzed the transferability of action recognition features trained on a mix of synthetic data and real data without humans on downstream tasks.
As shown in Fig. 1, we extend the SynAPT benchmark to address this important problem. We use a pre-training dataset consisting of the synthetic videos from SynAPT and human-removed real videos from No-Human Kinetics. This consists of videos in the Kinetics dataset [42] where the humans are removed from each frame by applying the HAT framework [19] to first detect the human segmentation mask and then inpaint within this mask. We evaluate the model on the same six diverse, downstream tasks from the SynAPT benchmark.
We then analyze how to best leverage synthetic data and human-removed real data to pre-train the models for different downstream tasks. Vision transformers [4, 8] yield state-of-the-art performance on human action recognition benchmarks but require extensive pre-training on large datasets such as ImageNet-21k [61]. As datasets of such scale usually contain humans, we require a more data-efficient method to pre-train models on the smaller, privacy-preserving datasets. Based on the Masked Autoencoders (MAE) method [37, 67, 30], we propose a two-stage pre-training strategy we denote as Privacy-Preserving MAE-Align (PPMA). We first perform the data-efficient self-supervised MAE pre-training on No-Human Kinetics to learn contextual features. We then conduct supervised pretraining on both No-Human Kinetics and synthetic data through action classification to align the representations learned from MAE to action labels. We found both stages of pre-training were crucial to achieve high performance and able to significantly outperform the methods in SynAPT [75] across the six downstream tasks.
Furthermore, when applying our PPMA pre-training strategy on a combination of no-human real data and synthetic data, we were able to achieve similar performance to models trained on real-human data. Compared to SynAPT, we made significant progress bridging the performance gap between representations trained on data with humans and without humans.
1. We are the first to study the transferability of privacy-preserving representations learned from real-world videos without humans (by curating the No-Humans Kinetics dataset) to a variety of human action recognition datasets.
2. We propose a novel extension of the SynAPT benchmark [75], where the model is pretrained on both real videos without humans and synthetic videos containing virtual humans, and then evaluated on a diverse set of downstream tasks.
3. To combine human-removed real data and synthetic videos, we propose a new pre-training strategy called Privacy-Preserving MAE-Align (PPMA). Compared to baselines, our approach improves downstream task performance by up to  $5\%$  on average.
Video Datasets for Pre-training. Large-scale video datasets, such as YouTube 8M [1], Kinetics [42], and Moments-in-Time [54], are crucial for training large models. Pre-training on these datasets is especially important when the target dataset is limited in size. While traditional datasets focus on human-annotated action labels, recent efforts have explored alternative annotations like social-media videos [29], instructional narrations [53], and spoken captions [55]. However, using real video data with humans could raise privacy and ethical concerns. In this paper, we propose a novel approach to address these issues by pre-training on human-free real and synthetic video data.
Privacy-Preserving Action Recognition. Prior methods for privacy-preserving action recognition involve downsampling [21], blurring [11], using off-the-shelf detectors to blur specific regions like the face [57] [82], replacing faces with synthetic faces [60], or even using adversarial image modification to maximize privacy while preserving performance [58, 77, 22]. AViD [57] is a popular dataset that blurs faces, but complete privacy isn't guaranteed. Specifically, other features like height, clothing, or
body shape can still reveal individual identity [78, 56]. In our work, we eliminate humans entirely via mask prediction and inpainting. In addition, unlike these prior works, we study the transferability of privacy-preserving representations to a diverse set of downstream human action recognition datasets.
Background Bias. State-of-the-art action recognition models often overrely on background features such as objects and scene cues instead of foreground action content, which leads to sub-optimal performance when generalizing to other tasks [80, 18]. One solution to this problem is to add data augmentation that swaps the action and background of two videos [80, 84, 32]. Additionally, one can add an adversarial loss to reduce background bias [18]. Our proposed PPMA method is different in that we learn background features from real data and temporal information from synthetic data. Thus, when we generalize to new downstream tasks, the model has priors about background features but can also focus on temporal action features.
Learning with Synthetic Data. Synthetic data has been extensively researched as a substitute for real-world training data in computer vision [24, 59, 63]. It can serve as a cost-effective solution for scaling datasets and can also be utilized to preserve privacy by replacing humans with virtual counterparts in images or videos [6]. Previous works in action recognition have employed graphic engines to generate videos simulating real human actions [40, 68]. However, there exists a performance gap when using synthetic data to learn transferable representations compared to real videos [27]. SynAPT [75] discovered that synthetic action videos performed well on downstream tasks with low scene-object bias but performed worse on tasks with high scene-object bias. In contrast to previous approaches, our method combines real-world data without humans with synthetic data (virtual humans) for pre-training, resulting in significant improvements in representation transferability, particularly for tasks with high scene-object bias.
Pre-training Transformers for Action Recognition. The model architecture plays a crucial role in training high-performing action recognition models, whether using real or synthetic data. Vision transformers [25, 4] have emerged as the state-of-the-art, surpassing CNNs like I3D [15] and TSN [72] [31, 67, 30, 73]. Pre-training on large datasets is essential for the good predictive performance of these models [15, 28, 49, 26, 43]. While supervised pre-training can serve this purpose, it is limited by the cost of labeling large-scale data. Self-supervised pre-training serves as an effective replacement. Methods include contrastive learning [16, 38, 66], clustering [12, 13], self-distillation [14, 34], or masked auto-encoders (MAE) [37, 67]. In our work, we propose the Privacy-Preserving MAE-Align (PPMA) pre-training scheme. We first perform self-supervised pre-training with MAE and then proceed to supervised training for action recognition. This scheme provides us with more discriminative features than any of its individual components do. This observation is similar to findings of concurrent work [64].
3 Proposed Benchmark and Privacy Preserving MAE-Align (PPMA)
Our goal is to study the transferability of pre-trained representations on large scale human-free data that preserves privacy. Towards this end, we build on the SynAPT benchmark [75] by adding privacy-preserving real data (i.e., No-Human Kinetics) (Sec. 3.1) for pre-training. We then pre-train state-of-the-art action recognition models using Privacy Preserving MAE-Align (PPMA) (Sec. 3.5) and evaluate them on 6 diverse downstream action recognition tasks (Sec. 3.3).
3.1 Privacy-Preserving Real Data
We aim to improve the transferability of pre-trained models by leveraging real data and remove privacy concerns. In this work, we choose Kinetics [42] as our real data. Kinetics is a large-scale video dataset consisting of 400 classes of human action clips curated from YouTube. To remove humans from these clips, in order to eliminate privacy and ethical issues with the data, we employ the HAT [19] pipeline. This consists of first using a SeMask segmentation model [41] to obtain per-frame segmentation masks around humans. The clips along with the segmentation masks are input to  $\mathrm{E}^2\mathrm{FGVI}$  [48], an optical flow-based inpainting model, to remove humans from the video. We run this pipeline on a subset of Kinetics training videos with 150 action classes and 1000 video clips per class following SynAPT. The outputs of these steps are videos without humans comprising the No-Human Kinetics (NH Kinetics) dataset. Fig. 2 shows some examples of these. In most cases, the HAT framework is effective at identifying and removing humans from all the frames in the video
Figure 2: Examples of privacy-preserving real video data without humans. With the HAT [19] framework, we identify the human segmentation mask and remove the human from the video.
while keeping the context in place. As segmentation and inpainting are off-the-shelf models, we conduct a manual review of No-Human Kinetics and discuss limitations in Appendix B.
3.2 Synthetic Data
The pre-training data in our benchmark is also comprised of synthetic videos from SynAPT [75]. This includes videos of virtual humans performing various actions curated from ElderSim [40], SURREACT [68], and Procedural Human Action Videos (PHAV) [23]. Same as No-Human Kinetics, it has 150 action categories with 1000 examples each. One of the primary aspects of the synthetic data is that it de-correlates actions and backgrounds, helping the model focus on temporal features. This is a property often lacking in real videos where the actions may be correlated to the contexts they take place in. More information about assumptions and bias of this data is available in SynAPT [75] and its sources.
3.3 Downstream Evaluation
The pre-trained models' transferability is assessed using six diverse tasks from SynAPT [75] listed in decreasing order of scene-object bias: UCF101 [65] features 13,320 YouTube videos across 101 action classes, offering significant variations in actions and camera motion. HMDB51 [45] provides 6,849 clips, mostly from movies, divided into 51 action classes. Mini Something-Something V2 (Mini-SSV2) is a smaller 87-category version of the Something-Something V2 [33] dataset, emphasizing fine-grained human hand gestures with everyday objects. Diving48 [47], a specialized dataset for competitive diving, tests a model's robustness due to its similar background and object features. It has 18,000 clips from 48 categories. Shea Furniture Assembly [7] provides 111 videos of 14 actors assembling and disassembling furniture under consistent camera and scene settings. The dataset has 12 action categories. UAV-Human [46] contains 22,476 videos captured by unmanned aerial vehicles (UAVs), presenting 155 action classes and 119 subjects.
3.4 MAE-Align
While SynAPT [75] relied on models with a CNN architecture, as mentioned in the introduction, state-of-the-art action recognition models use vision transformers (ViT). However, these ViT models require pre-training on additional data such as Imagenet-21K, which contain additional pictures of humans, giving up privacy even when no humans appear in videos. Thus, we choose to leverage MAE [67] because it outperforms other state-of-the-art self-supervised method for action recognition
Stage 1: MAE Pre-training on No-Human Kinetics
Stage 2: Supervised Pre-training for Label-Alignment
Figure 3: Privacy-Preserving MAE-Align (PPMA). Stage 1: An encoder-decoder transformer is pre-trained for the MAE reconstruction task with privacy-preserving (no human) real video data in order to learn the context features of actions. Stage 2: the encoder is further pre-trained on synthetic human action recognition datasets for alignment with the action recognition task.
[67, 71] and it is data-efficient, meaning we do not need to pre-train on additional image data such as Imagenet-21K. Specifically, we propose a two-stage pre-training approach:
Stage 1: MAE Training. We use the standard MAE training process for videos [67, 30]. An encoder-decoder architecture is trained to predict masked pixel values in videos, following which the decoder is discarded.
Stage 2: Supervised Alignment. To the trained MAE encoder, we add a linear classifier layer and train this whole network for classification with supervision from action labels.
The two stages are crucial for learning discriminative features for downstream tasks, as we shall show in Sec. 4.3.
3.5 Privacy-Preserving MAE-Align (PPMA)
Fig. 3 shows our approach. Using the no-human data from our benchmark, we present Privacy Preserving MAE-Align (PPMA) as a method of learning strong transferable video representations and maintaining privacy, while simultaneously closing the performance gap to pre-training with video data containing real humans (Kinetics) (see Tab. 1). Stage 1 involves MAE training with No-Human Kinetics. The reconstruction task allows features to learn the real-world context of actions. Stage 2 of PPMA involves supervised alignment using videos from No-Human Kinetics and SynAPT Synthetic data. As the size of the No-Human Kinetics and Synthetic dataset are roughly equal and we use the same loss function for both sources of data, both datasets have roughly equal impact on training the model. Additional training details can be found in Appendix A.
For all experiments, we use the ViT-B [4] model. We train all models from scratch without ImageNet pre-trained weights. First, we pre-train the model with MAE for 200 epochs using the random masking [67] strategy. After this, we discard the MAE decoder and only keep the encoder. We perform supervised pre-training for 50 epochs during the label-alignment stage. For comparisons using Kinetics data, we use the same 150 class subset as used in SynAPT [75]. For all downstream tasks, we finetune (FT) the entire network or train a linear probe (LP) for 30 epochs. More training details can be found in Appendix A.
4.2 PPMA Performance
Tab. 1 reports the individual task and average downstream performance of different pre-trained representations including our proposed Privacy Preserving MAE-Align (PPMA). The table is split into two halves with the top half containing representations that do not preserve privacy and the
Table 1: PPMA performance. Reported are top-1 downstream task accuracy for finetuning (FT) and linear probing (LP). Average represents the average FT and LP accuracy across all downstream tasks. Unless specified, a ViT-B backbone is used. We find that Privacy Preserving MAE-Align (PPMA) is  $2.5\%$  better with FT and  $5\%$  better with LP than the next best baseline (MAE-Align w/ Synthetic). It learns features comparable to pre-training with data with real humans.  $\mathrm{NH} = \mathrm{No}$ -Human.\*uses an Imagenet-21K pre-trained backbone before training with videos. $\dagger$ results reported in [75].
Pre-trained Model	Privacy Preserving	Stage 1: MAE	Stage 2: Alignment	Downstream Task Accuracy	Average
UCF101	HMDB51	Mini-SSV2	Diving48	IkeaFA	UAV-Human
FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP
MAE-Align w/ real humans	X	Kinetics	Kinetics	93.3	91.4	73.4	69.5	68.8	37.4	66.3	19.9	72.0	58.3	34.9	13.9	68.1	48.4
TimeSformer Kinetics*†	X	x	Kinetics	92.1	89.4	59.5	55.4	48.9	21.5	46.4	17.0	61.9	47.7	23.3	8.4	55.3	39.9
TimeSformer Synthetic*†	X	x	Synthetic	89.0	82.1	54.4	49.2	51.1	21.2	44.9	19.2	63.6	45.5	25.0	13.8	54.7	38.5
Scratch	✓	x	x	30.1	-	14.8	-	16.0	-	9.3	-	19.5	-	0.7	-	15.1	-
TSN (RN50 backbone)†	✓	x	Synthetic	83.4	28.0	54.4	20.9	49.7	12.8	63.5	10.9	42.7	36.0	35.6	5.7	54.9	19.1
I3D (RN50 backbone)†	✓	x	Synthetic	82.1	27.6	55.7	22.6	50.7	12.3	55.3	10.1	42.7	33.2	35.1	5.8	53.6	18.6
R(2+1)D (RN50 backbone)†	✓	x	Synthetic	80.0	26.4	53.3	22.2	52.0	13.3	57.3	10.0	41.5	35.7	31.8	5.5	52.6	18.9
MAE-Align w/ Synthetic	✓	Synthetic	Synthetic	88.7	76.6	69.7	59.7	64.3	26.0	61.1	16.7	67.3	57.7	36.1	20.6	64.5	42.9
Ours : Privacy-Preserving MAE-Align (PPMA)	✓	NH Kinetics	NH Kinetics + Synthetic	92.5	88.4	71.2	64.9	67.8	34.9	64.0	21.9	67.9	57.7	38.5	19.3	67.0	47.9
bottom, representations that do. These models pre-trained on conventional large-scale real video data with humans have a performance edge over models trained with synthetic data due to a realism gap. The best performing model with real human videos is "MAE-Align w/ real humans" in Tab. 1.
While other baselines (described below) which preserve privacy present a significant performance gap, with PPMA and the use of No-Human Kinetics, the average downstream performance gap from the human-baseline is down to  $1.1\%$  (68.1% vs 67%) with FT and  $0.5\%$ . One reason for this gap is that PPMA does worse than the human-baseline on high scene-object bias tasks such as UCF101, HMDB51, and Mini-SSV2. This could be because including humans in the videos of Kinetics helps the model better learn both scene-object cues and action features. Nevertheless, compared to training solely with synthetic data in the MAE-Align w/ Synthetic model, PPMA has significantly reduced the gap with the human-baseline. Thus, while preserving privacy, we can achieve comparably good pre-trained representations for action recognition.
"Scratch" refers to randomly initializing a ViT-B backbone and training it with label supervision on downstream tasks. Comparing its performance with the other methods offers an understanding of the downstream performance benefit of pre-training representations.