SlowGuess/NeurIPS_2023_OCR huggingface dataset:
"TSN (RN50 backbone)" is the best model trained entirely using synthetic data from SynAPT [75]. This model lags "MAE-Align w/ real humans" significantly in downstream performance. A contributing factor to this is the limitation in the backbone architecture. Using the same data to pre-train a high capacity ViT-B backbone requires a more data efficient pre-training process which we achieve with MAE-Align (see further discussion of data efficiency in 4.3). This pre-trained model, "MAE-Align w/ Synthetic" indeed improves average downstream performance significantly (from  $54.9\%$  to  $64.5\%$  FT and from  $19.1\%$  to  $42.9\%$  LP). Also noteworthy is the fact that using MAE-Align and synthetic data, this model outperforms both ViT-B architecture based TimeSformer models from [75], indicating the strength of MAE training compared to large scale supervised pre-training onImagenet-21K. Finally when we add No-Human Kinetics data to the mix, PPMA further improves over "MAE-Align w/ Synthetic" by  $2.5\%$  FT and  $5\%$  LP.
4.3 The effectiveness of MAE-Align
To understand the effectiveness of our pre-training strategy, using each of the three datasets: Kinetics, Synthetic, and NH Kinetics, we ablate one of the pre-training stages. The average fine-tune downstream accuracy over the 6 tasks, is reported in Tab. 2. Fig. 4 shows results for these pre-trained models with FT for different downstream tasks. The same analysis with LP is in Appendix C.
Comparing only MAE pre-training to only supervised alignment, we find features are more transferable (with higher average downstream task performance) in the former case than the latter. This
Figure 4: Ablating stages 1 and 2 of MAE-Align for different pre-training datasets. The title of each chart indicates the pre-training dataset. Each chart plots the downstream accuracies of the models finetuned for the 6 downstream tasks. We show three different model performances for each downstream task with three different pre-training strategies: Align only, MAE only, and MAE+Align. Our results show that both MAE and Align stages are crucial for good downstream performance.
is because high-capacity models like ViTs with a classification objective have a loss landscape with several sharp local minima, making it hard to optimize using fewer data without large-scale pre-training (e.g. with ImageNet-21K) [17]. The MAE objective is more data-efficient in learning generalizable representations for these high-capacity networks [67].
While MAE pre-training helps with features when finetuned on downstream tasks, the improvement compared to supervised training only is smaller when only a linear probe is learned. We hypothesize that the MAE pre-trained backbone is not well aligned with the downstream action recognition tasks. Hence, when the features are frozen (LP), the performance of the downstream classifier is not significantly higher, like in the case of FT. On training with supervised labels on the pre-training set after MAE, the backbone is more aligned to action recognition and improves performance significantly even with LP.
4.4 Combining Contextual and Temporal Features
Pre-training on synthetic data is helpful for learning temporal action features but not contextual features [75]. This is compensated for by using no-human real data to learn contextual features.
In Tab. 2, we find that MAE pretraining on No-human Kinetics learns more transferable features than MAE on Synthetic data. The reconstruction objective of MAE helps models learn the contexts of actions making it more suitable for videos with real backgrounds.
After MAE pre-training on NoHuman Kinetics, we explore the effect of supervised alignment on different datasets: Synthetic, No-Human Kinetics or a combination of the two. In Tab. 3 we find that on average over the 6 downstream tasks, combining label alignment using both these datasets as we propose in PPMA, leads to the most transferable features.
Table 2: Ablating stages 1 and 2 of MAE-Align for different pre-training datasets. Average downstream accuracy over 6 tasks is reported. Each row also mentions the performance difference to performing only supervised alignment under  $\Delta$ FT and  $\Delta$ LP. We find that MAE is better for pretraining features than supervised alignment but a subsequent alignment stage improves representations even further.
Stage 1 : MAE	Stage 2 : Alignment	Average Downstream
FT	ΔFT	LP	ΔLP
x	Kinetics	32.3	-	25.1	-
Kinetics	x	59.9	+27.6	28.7	+3.6
Kinetics	Kinetics	68.1	+35.8	48.4	+23.3
x	NH Kinetics	31.1	-	25.3	-
NH Kinetics	x	58.0	+26.9	28.4	+3.2
NH Kinetics	NH Kinetics	65.6	+34.5	42.4	+17.2
x	Synthetic	31.1	-	25.4	-
Synthetic	x	56.4	+25.3	27.8	+2.4
Synthetic	Synthetic	64.5	+33.4	42.9	+17.5
Averaging Model Weights. Among alternate ways of combining temporal (from Synthetic data) and contextual (from No-Human Kinetics) features, we additionally explored averaging pre-trained model weights as done in [76]. Tab. 4 contains the results of averaging the weights of the three pre-trained models from Tab. 3 in the proportions  $\alpha_{1}$ ,  $\alpha_{2}$  and  $\alpha_{3}$  respectively, where  $\alpha_{1} + \alpha_{2} + \alpha_{3} = 1$ . In the first row of Tab. 4 we combine two models which are aligned using NH-Kinetics and Synthetic respectively, in equal proportion. In contrast to PPMA, which label aligns one model on NH Kinetics + Synthetic, this separately label-aligns two models on the two datasets and later combines the trained model weights. In performance, we find that this lags behind PPMA, thus indicating label-alignment on combined data is the better strategy for learning more transferable features.
Next, we evaluate averaging the weights of three models adding PPMA to the mix above. Row 2 of Tab. 4 shows the results of mixing the three models in equal proportion and row 3, the results of mixing them in proportion of the amount of data used for label-alignment. We find that these mixtures outperform the mixture of two models, with the last mixture performing significantly better. These results indicate averaging models pre-trained on different subsets of our human-free data could be effective representations for downstream tasks. We leave further exploration of this to future work.
Table 3: Supervised alignment with temporal and contextual features. Comparing the downstream performance of models label-aligned on No-Human (NH) Kinetics (contextual), Synthetic (temporal), or both datasets (contextual and temporal). All models were first pre-trained with MAE on No-Human Kinetics. Note that the first row is our proposed PPMA, which performs better downstream compared to label-alignment using either of the individual datasets, Synthetic or NH Kinetics.
Stage 1: MAE	Stage 2: Alignment	Downstream Task Accuracy	Average
UCF101	HMDB51	Mini-SSV2	Diving48	IkeaFA	UAV-Human
FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	LP	FT	LP
NH Kinetics	NH Kinetics	90.3	84.5	67.0	58.1	67.3	32.5	63.1	17.8	74.4	51.2	31.6	10.6	65.6	42.4
Synthetic	91.4	81.9	71.5	62.0	66.4	31.8	65.3	21.8	67.3	57.7	38.3	20.8	66.7	46.0
NH Kinetics + Synthetic	92.5	88.4	71.2	64.9	67.8	34.9	64.0	21.9	67.9	57.7	38.5	19.3	67.0	47.9
Table 4: Averaging model weights to combine temporal and contextual features. The three models label-aligned on No-Human Kinetics, Synthetic, and both datasets are combined in proportions of  $\alpha_{1}$ ,  $\alpha_{2}$  and  $\alpha_{3}$  respectively. We find that averaging the first two models in equal proportion performs slightly poorer than PPMA. Adding PPMA into the mix with a non-zero  $\alpha_{3}$  improves downstream performance significantly.
Model Proportions	Downstream Task Accuracy	Average
NH Kinetics α1	Synthetic α2	Both α3	UCF101	HMDB51	Mini-SSV2	Diving48	IkeaFA	UAV-Human
FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP
0.50	0.50	0.00	91.6	85.0	70.8	63.4	67.2	36.8	67.5	21.9	65.2	63.4	36.9	15.8	66.5	47.7
0.33	0.33	0.33	92.2	85.5	72.7	64.1	67.8	37.3	68.0	22.1	70.1	59.8	38.0	17.1	68.1	47.6
0.25	0.25	0.50	93.0	86.6	72.1	65.9	67.8	37.7	69.3	23.2	73.2	61.6	38.6	18.6	69.0	48.9
4.5 Averaging Model Weights with Learned Proportions
In Tab. 5, on changing the proportions  $\alpha_{1}$  and  $\alpha_{2}$  of combining model weights, we find different ratios of alignment on No-Human Kinetics and Synthetic can improve the representations for downstream tasks. This motivates exploring a method where we can learn, based on the downstream task, a proportion of mixing the two models for optimal performance. The last row of Tab. 5 reports performance of this model, where  $\alpha_{1} = 1 - \beta$  and  $\alpha_{2} = \beta$ .  $\beta$  is a parameter initialized to  $1/2$  and learned via gradient descent during LP (while the pre-trained representations are frozen and only the linear probe is updated). In practice the constraint  $\beta \in [0,1]$  is enforced by making it the output of a softmax function. Note that we only learn this mixing proportion of pre-trained weights for LP, since this mode of evaluation keeps those weights frozen. This would not be possible if the backbone weights are updated while learning the parameter as is the case in FT.
From the results reported in the last row of Tab. 5, we observe that learning this mixing proportion per dataset benefits the performance over PPMA or the pre-defined proportions we evaluated on Ika-FA and UAV-Human datasets. On the rest, it is on par with (UCF101, Diving48) or worse than (HMDB51, Mini-SSV2) the best mixing proportion. This is possibly due to  $\beta$  fitting aggressively to the training set of downstream tasks. Tab. 6 presents the learned  $\beta$  at the end of training for
Table 5: Adaptive averaging model weights to combine temporal and contextual features. Models are label-aligned on No-Human Kinetics and Synthetic and averaged in proportions of  $\alpha_{1}$  and  $\alpha_{2}$  respectively. For the bottom row, we learn adaptive mixing ratio  $\beta$  for each downstream task with LP.
Model Proportions	Downstream Task Accuracy	Average
α1	α2	UCF101	HMDB51	Mini-SSV2	Diving48	IkeaFA	UAV-Human
FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP	FT	LP
PPMA (Mixing data)	92.5	88.4	71.2	64.9	67.8	34.9	64.0	21.9	67.9	57.7	38.5	19.3	67.0	47.9
0.25	0.75	91.5	82.9	72.1	64.8	66.6	34.8	66.4	21.8	67.9	64.0	37.8	20.2	67.0	48.1
0.5	0.5	91.6	85.0	70.8	63.4	67.2	36.8	67.5	21.9	65.2	63.4	36.9	15.8	66.5	47.7
0.75	0.25	91.8	86.0	69.3	59.7	67.7	36.2	66.1	21.3	73.8	60.4	34.5	12.8	67.2	46.1
1 - β	β	-	85.0	-	60.3	-	34.7	-	21.6	-	64.6	-	20.4	-	47.8
Table 6: Learned mixing ratio for different downstream tasks. We adaptively learn the mixing ratio of Synthetic and No-Human Kinetics models for each downstream task.
UCF101	HMDB51	Mini-SSV2	Diving48	IkeaFA	UAV-Human
Percentage of Synthetic: β	14.3%	21.5%	66.0%	75.8%	71.9%	83.0%
Percentage of Real: 1-β	85.7%	78.5%	34.0%	24.2%	28.1%	17.0%
each of the downstream tasks and we see a preference for a higher ratio for the Synthetic backbone in tasks with low scene-object bias (Mini-SSV2, Diving48, IkeaFA, UAV-Human). This might be expected since real data helps the model learn contextual cues whereas synthetic data helps the model learn temporal action features. We leave further investigation of this model and other techniques of combining pre-trained representations to future work.
While real world videos make for excellent pre-training data for action recognition models, they are riddled with privacy issues. Prior use of synthetic data to circumvent these issues was limited in performance because of no exposure to real-world context of actions. In this work, we propose a benchmark to overcome these challenges by using real videos with obfuscated humans, along with synthetic videos for pre-training. Using this data we proposed Privacy-Preserving MAE-Align (PPMA), a method for pre-training representations that while preserving privacy, learns features that are at par with pre-trained representations on real human data in terms of transferability to downstream action recognition tasks.
Limitations. While PPMA is effective, additional representation learning methods could improve upon it. One potential direction of exploration is using averages of multiple pre-trained models, that we leave for future work. Also beyond the scope of our current work is a study of how privacy-preserving pre-training methods scale with more videos, presenting an interesting direction for future research.
Potential Social Impacts. Our work aims to mitigate the negative impacts of using data with real humans by obfuscating them and preserving their privacy. Additionally, this can make models pre-trained on this data less biased towards aspects such as races or genders of humans appearing in the data. Potential negative impacts of this work include simply applications where automated action recognition can be used to make rash decisions without human intervention. One example of this could be automated identification of people and incrimination of activities deemed illicit in camera feeds.
Acknowledgement. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750-19-C-1001. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA). This work was partially supported by Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University).
[70] Heng Wang and Cordelia Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international conference on computer vision, pages 3551-3558, 2013.
[71] Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking, 2023.
[72] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos, 2017.
[73] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning, 2023.
[74] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5310-5319, 2019.
[75] Yo when Kim, Samarth Mishra, SouYoung Jin, Rameswar Panda, Hilde Kuehne, Leonid Karlinsky, Venkatesh Saligrama, Kate Saenko, Aude Oliva, and Rogerio Feris. How transferable are video representations based on synthetic data? In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[76] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022.
[77] Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin, and Zhangyang Wang. Privacy-preserving deep action recognition: An adversarial learning framework and a new dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[78] Kaiyu Yang, Jacqueline H Yau, Li Fei-Fei, Jia Deng, and Olga Russakovsky. A study of face obfuscation in imagenet. In International Conference on Machine Learning, pages 25313–25330. PMLR, 2022.
[79] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
[80] Sangwoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, and Jinhyung Kim. Videomix: Rethinking data augmentation for video classification, 2020.
[81] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018.
[82] Zhixiang Zhang, Thomas Cilloni, Charles Walter, and Charles Fleming. Multi-scale, class-generic, privacy-preserving video. *Electronics*, 10(10):1172, 2021.
[83] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation, 2017.
[84] Yuliang Zou, Jinwoo Choi, Qitong Wang, and Jia-Bin Huang. Learning representational invariances for data-efficient action recognition. Computer Vision and Image Understanding, 227:103597, 2023.
A Training Details
Our implementation is based upon OmniMAE [30] repository. All models were trained distributed over 4-8 NVIDIA V100 and A100 GPUs.
A.1 Pre-training
A.1.1 Stage 1: MAE
Architecture: All experiments use the ViT-B backbone architecture. The encoder consists of 12 transformer blocks with embedding dimension of 768, and the decoder consists of 4 transformer blocks with an embedding dimension of 384.
Input: We represent the video as a 4D tensor of shape  $T \times H \times W \times 3$  where  $T$  represents the temporal dimension,  $H$  and  $W$  are spatial dimensions, and 3 is the number of color channels. We then convert this 4D tensor into  $N$  spatio-temporal patches, each of size  $t \times h \times w \times 3$ . We set  $t = 2$  and  $h = w = 16$ .
Loss Function: To minimize the reconstruction error between the image pixel values and the decoder's prediction, we measure reconstruction error with  $\ell_2$  loss after normalizing the pixels to zero mean and unit variance.
Masking Ratio: For each frame, we mask out  $90\%$  of the patches in the image at random. The high masking ratio is chosen to make the reconstruction task more difficult and encourage extracting more effective video representations. While VideoMAE [67] advocates to use tube masking for small improvements in accuracy, we decide to follow the training details in OmniMAE and use the random masking scheme [30].
Other Training Details: We train our model with a mini-batch size of 128. Each batch consists of 32 distinct videos, each replicated 4 times but with different masks as in [30]. We resize the videos spatially to  $224 \times 224$  pixels and sample a clip of  $T = 16$  frames. The model takes in patch sizes of  $2 \times 16 \times 16$  and uses a non-learnable sinusoidal positional encoding. We choose to use global pooling instead of a class token. We use the AdamW optimizer [52, 44] with a cosine learning rate scheduler [51] that uses a maximum learning rate of 8e-4. We train for 200 epochs, with a 10 epoch learning rate warmup that linearly grows from 5e-7 to 8e-4.
A.1.2 Step 2: Supervised Alignment
After MAE pre-training, we discard the decoder and add a linear classification layer to the encoder to perform action recognition on No-Human Kinetics and synthetic data. This is a crucial step to align the model's features learned from MAE to downstream action recognition tasks.
Architecture: We use the same ViT-B encoder and add a final linear classification layer for supervised training. When co-training on two datasets, the model shares an encoder but has a different linear classification layer for each dataset.
Input: This input is the same as in MAE as described in Appendix A.1.1.
Loss Function: We minimize cross entropy loss between the final classification layer output and the true action category.
Details for training on two datasets: When training on both No-Human Kinetics and Synthetic, each batch consists of samples from one dataset: either only real or only synthetic data. Both sources of data are roughly equal size, and each epoch involves training on every example of No-Human Kinetics and Synthetic data once. Loss functions for real and synthetic data are the same and are weighted equally.
Other Training Details: Most training parameters, such as the mini-batch size of 128, are the same as in Appendix A.1.1. However, we train for only 50 epochs and choose different hyperparameters for our optimizer. The cosine rate scheduler we use has a maximum learning rate of 2e-3, with a 6 epoch learning rate warmup where learning rate linearly grows from 5e-7 to 2e-3. We add data
augmentation techniques such as RandAugment [20], Random Erasing [83], Mixup [81], and CutMix [79].
A.2 Downstream Evaluation Description
We evaluate the pretrained action recognition representations on six diverse, downstream tasks described in Sec. 3.3 of the main paper. Starting from the pre-trained ViT-B encoder, for the specific downstream task, we add a linear classification layer. We train for 30 epochs with a mini-batch size 32. In the finetuning (FT) setting, we employ the same number of warmup steps, same data augmentations, and same maximum learning rate of 2e-3 for the cosine rate scheduler as in supervised alignment (Appendix A.1.2). For the linear probing stage, we only tune the final linear classification layer and use the same hyperparameter settings as FT except the cosine rate scheduler has a maximum learning rate of 1.6e-2.
B Quality Assessment of No-Human Kinetics
In this section, we present the findings from a manual quality assessment of our inpainting pipeline to remove humans. 4 random samples from each of 150 categories were reviewed by 3 different people. We found that in  $96.3\%$  of the videos humans were completely removed. Additionally, on a subsequent review of the remaining  $3.7\%$  of the videos, we found in all of them, either the humans were too small or mixed in a crowd and hence unidentifiable on visually inspecting the inpainted video. We display more qualitative results/visuals for both failure and correct cases.
Figure 5: Limitations of the HAT inpainting pipeline. Displayed are examples of failure cases. In the top example, the human is connected to hat and trump and then inpainting becomes blurry. The 2nd and 3rd example showcase the error where not all humans are removed, which is more likely to occur when there are crowds of people or the human is small. In these cases, the segmentation model did not detect all humans in the video.
Fig. 5 shows examples of failure cases. The HAT inpainting pipeline can fail to accurately remove all humans from videos (i) with crowds (row 2) and (ii) when the human is too small (rows 2 and
3). Often, the errors result from the segmentation model not accurately detecting the human. In addition, the inpainted region may sometimes blurry (row 1), which may occur if the human is touching different objects or if there is poor lighting. While the human is still unidentifiable, this could be source of noise for representation learning.
Nevertheless, as seen from the manual review, the majority of inpaintings are able to remove humans and are effective for representation learning. See Fig. 6 for additional examples of where the segmentation network identifies the human and thus is able remove them.
Figure 6: Additional examples of No-Human Kinetics. For the most part, HAT successfully removes the human from the video.
While Fig. 1 and Fig. 2 show examples where HAT successfully removed the human, we display failure cases where the segmentation model or inpainting model fails to identify all humans in Fig. 5. For example, at the top scene of Fig. 5, even after applying HAT to remove the humans, the video still shows remnants of spectators in the crowd and basketball players who are moving. We found that these errors of not removing all humans occur only in a few edge cases (3.7% total, as mentioned above): (i) videos with crowds (top 2 rows); (ii) when the human is moving too fast and is blurry and non-identifiable (top row); (iii) if the human is too small and non-identifiable (3rd row); (iv) or if the face was blurry and non-recognizable to begin with (4th row). We believe that the issue mainly arises when the off-the-shelf segmentation network fails to detect the human, and further improvements in these aspects are possible with additional training of this network.
Additionally, another failure case of human removal is that inpainting network sometimes does not fully remove the outline of the human. Thus, while it is impossible to identify the person, one may be able to infer information such as body shape or age. As segmentation and inpainting networks improve, we expect these issues to occur even less.