SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C MAE Align for Linear Probing
Analogous to Fig. 4 from the main paper, where downstream performance is evaluated with downstream finetuning(FT), we report linear probe(LP) downstream performance in Fig. 7. With LP, same
as with FT, we find MAE+Align for pre-training performs better than either MAE or Align. In Sec 4.3 of the main paper, we discuss MAE pre-training is not well aligned to downstream action recognition using average downstream performance (reported in Tab. 2). This results in poor downstream performance when the backbone features are frozen during LP. The same effect can be observed for each individual downstream task in Fig. 7.
KINETICS
NH KINETICS
SYNTHETIC
Figure 7: Ablating stages 1 and 2 of MAE-Align for different pre-training datasets (LP). The title of each chart indicates the pre-training dataset. Each chart plots the downstream accuracies of the models linear probed for the 6 downstream tasks. We show three different model performances for each downstream task with three different pre-training strategies: Align only, MAE only, and MAE+Align. Our results show that both MAE and Align stages are crucial for good downstream performance.