SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we evaluate the performance of the position reconstruction task using pre-trained models under different settings. Specifically, we vary  $\gamma \in \{0, 0.25, 0.5, 0.75\}$  and  $\gamma_{\mathrm{pos}} \in \{0.25, 0.5, 0.75, 0.95\}$  when measuring the position prediction accuracy. We report performance under different evaluation settings as well as the averaged accuracy among 16 different cases. From Tabs. S3 to S5, we find evidence to support the three difficulties for designing an appropriate position-related pretext task introduced in Sec. 1: (i) discrepancies between pre-training and fine-tuning, (ii) failing to learn highly semantic representations by solving this simple position reconstruction task, and (iii) difficult to decide which patch positions to reconstruct precisely.
We study the effectiveness of different values of  $\gamma$  during pre-training in Tab. S3. Interestingly, we find evidence for failing to learn highly semantic representations by solving this simple position reconstruction task. As illustrated by Tab. S3a, the pre-trained model performs extremely well when we set  $\gamma = 0$  for evaluation but fails to keep this trend when we enlarge  $\gamma$ . This indicates that given
Table S3: Top-1 accuracy of position reconstruction using ViT-B/16 [21] pre-trained with different mask ratio  $\gamma$ . We underline the special parameter different from our default settings. Default settings are highlighted in color. "Avg. acc" is the averaged top-1 accuracy over 16 different cases. We evaluate the performance using the same pre-trained model under different  $\gamma$  and  $\gamma_{\mathrm{pos}}$ . Larger  $\gamma$  and  $\gamma_{\mathrm{pos}}$  indicates a more challenging task.
(a) Pre-training with  $\gamma = 0$  (Avg. acc: 59.79).
γpos	0.25	0.50	0.75	0.95	avg.
0.00	99.37	99.26	99.15	98.34	99.03
0.25	87.97	87.45	86.20	70.79	83.10
0.50	55.37	55.11	49.84	22.96	45.82
0.75	12.99	14.85	12.85	4.21	11.23
(b) Pre-training with  $\gamma = 0.25$  (Avg.acc:79.62).
γpos	0.25	0.50	0.75	0.95	avg.
0.00	99.24	99.26	99.18	98.92	99.15
0.25	98.67	98.81	98.63	97.21	98.33
0.50	93.96	93.62	90.01	62.11	84.93
0.75	50.02	47.79	35.03	11.46	36.08
(c) Pre-training with  $\gamma = 0.5$  (Avg. acc: 87.27).
γpos	0.25	0.50	0.75	0.95	avg.
0.00	99.33	99.23	99.13	98.85	99.14
0.25	99.05	98.95	98.78	98.20	98.75
0.50	94.60	95.94	94.28	83.31	92.03
0.75	78.77	72.75	59.54	25.59	59.16
(d) Pre-training with  $\gamma = 0.75$  (Avg. acc: 87.83).
γpos	0.25	0.50	0.75	0.95	avg.
0.00	97.19	98.69	98.20	92.30	96.60
0.25	96.78	98.26	97.66	91.05	95.93
0.50	97.26	96.82	95.66	89.12	94.72
0.75	79.94	78.10	68.73	40.24	66.75
Table S4: Top-1 accuracy of position reconstruction using ViT-B/16 [21] pre-trained with different positional mask ratio  $\gamma_{\mathrm{pos}}$ . We underline the special parameter different from our default settings.
(a) Pre-training with  $\gamma_{\mathrm{pos}} = 0.25$  (Avg.acc:65.19).
γpos	0.25	0.50	0.75	0.95
0.00	96.52	89.38	59.29	14.58
0.25	96.62	91.27	66.00	19.17
0.50	95.58	91.10	72.07	21.69
0.75	80.56	73.72	57.89	17.66
avg.	92.32	86.37	63.81	18.28
(b) Pre-training with  $\gamma_{\mathrm{pos}} = 0.5$  (Avg. acc: 86.70).
γpos	0.25	0.50	0.75	0.95
0.00	98.98	98.81	98.24	92.80
0.25	98.59	98.34	97.51	89.73
0.50	96.63	95.85	93.40	74.38
0.75	81.94	76.70	64.87	30.44
avg.	94.04	92.43	88.51	71.84
(c) Pre-training with  $\gamma_{\mathrm{pos}} = 0.75$  (Avg.acc:87.83).
γpos	0.25	0.50	0.75	0.95
0.00	97.19	98.69	98.20	92.30
0.25	96.78	98.26	97.66	91.05
0.50	97.26	96.82	95.66	89.12
0.75	79.94	78.10	68.73	40.24
avg.	92.79	92.97	90.06	78.18
(d) Pre-training with  $\gamma_{\mathrm{pos}} = 1$  (Avg. acc: 19.44).
γpos	0.25	0.50	0.75	0.95
0.00	15.57	23.24	20.41	23.59
0.25	12.51	21.10	24.30	29.41
0.50	7.76	14.18	25.56	45.01
0.75	3.34	6.00	12.53	26.45
avg.	9.80	16.13	20.70	31.12
the strength of ViTs in modeling long-range dependencies, they have easily solved this task in a superficial way, and thus pre-training with  $\gamma = 0$  becomes trivial for ViTs. To this end, an appropriate  $\gamma$  is necessary to increase the difficulty of the pretext task and avoid trivial solutions.
We study the effectiveness of different values of  $\gamma_{\mathrm{pos}}$  during pre-training in Tab. S4, and we find evidence for discrepancies between pre-training and fine-tuning. As shown by Tab. S4d, the model fails to reconstruct accurate positions given some visible anchors. This is because the model has never been exposed to any positional embeddings (PEs) during pre-training. Therefore, providing some anchors is necessary to address discrepancies. Also, it may help the model focus on modeling relative relationships instead of simply reconstructing absolute positions.
Table S5: Top-1 accuracy of position reconstruction using ViT-B/16 [21] pre-trained with different (i)  $\sigma$  and (ii)  $\tau$ . We underline the special parameter different from our default settings.
(a) Pre-training with  $\sigma = 0$  (Avg. acc: 88.81).
γpos	0.25	0.50	0.75	0.95
0.00	98.48	98.74	98.29	94.37
0.25	98.06	98.31	97.76	93.48
0.50	96.06	96.08	94.48	85.49
0.75	82.19	78.64	69.39	41.23
(c) Pre-training with  $\sigma = 2$  (Avg. acc: 69.48).
γpos	0.25	0.50	0.75	0.95
0.00	78.45	78.81	78.19	72.33
0.25	78.03	78.41	77.68	70.49
0.50	76.14	76.27	74.53	63.52
0.75	63.75	61.21	53.30	30.47
(e) Pre-training with  $\sigma = 2\rightarrow 0$  (Avg. acc: 87.65).
γpos	0.25	0.50	0.75	0.95
0.00	97.63	98.61	97.93	91.30
0.25	97.22	98.19	97.46	89.96
0.50	94.99	95.85	94.06	82.50
0.75	80.33	77.92	68.42	40.11
(g) Pre-training with  $\tau = 0.1$  (Avg.acc:87.83).
γpos	0.25	0.50	0.75	0.95
0.00	97.19	98.69	98.20	92.30
0.25	96.78	98.26	97.66	91.05
0.50	97.26	96.82	95.66	89.12
0.75	79.94	78.10	68.73	40.24
(b) Pre-training with  $\sigma = 1$  (Avg. acc: 87.13).
γpos	0.25	0.50	0.75	0.95
0.00	97.03	98.50	97.93	91.40
0.25	96.63	98.04	97.33	89.69
0.50	94.30	95.58	93.68	81.47
0.75	79.14	77.06	67.43	38.89
(d) Pre-training with  $\sigma = 1\rightarrow 0$  (Avg.acc:87.83).
γpos	0.25	0.50	0.75	0.95
0.00	97.19	98.69	98.20	92.30
0.25	96.78	98.26	97.66	91.05
0.50	97.26	96.82	95.66	89.12
0.75	79.94	78.10	68.73	40.24
(f) Pre-training with  $\underline{\tau} = \infty$  (Avg. acc: 88.66).
γpos	0.25	0.50	0.75	0.95
0.00	98.66	98.31	97.72	91.22
0.25	98.54	98.17	97.46	91.00
0.50	96.85	96.20	94.52	84.64
0.75	83.57	79.30	70.04	42.29
(h) Pre-training with  $\underline{\tau} = 0.5$  (Avg. acc: 87.78).
γpos	0.25	0.50	0.75	0.95
0.00	97.68	97.94	97.82	91.78
0.25	97.83	97.91	97.53	91.01
0.50	96.44	96.06	94.52	84.49
0.75	80.16	77.16	66.01	40.18
We study the effectiveness of different values of  $\gamma_{\mathrm{pos}}$  during pre-training in Tab. S4, and we find evidence for hard to decide which patch positions to reconstruct precisely. As shown by Tabs. S5a and S5f, the model achieves higher position prediction accuracy but performs worse on downstream tasks (please refer to Tabs. 3 and 4 for downstream performances). Therefore, to prevent being overwhelmed by this particular position reconstruction task, techniques for relaxing the patch-wise classification problem become necessary, i.e., position smoothing, and attentive reconstruction.