SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By the convexity of  $F(\theta)$ , we have  $F(\bar{\theta}_K) \leq \frac{1}{K} \sum_{k=0}^{K-1} F(\theta_k)$ . Combined with (16), we attain the desired result.
B.2 Analysis for the Minimax Pre-training Method
B.2.1 Proof of Proposition 5.1
By the assumption on the downstream task losses  $\ell_{\lambda}$  and the task space  $\Lambda$ , the equation
$$
\max  _ {\lambda \in \Lambda} \mathbb {E} _ {z \sim P} \left[ \ell_ {\lambda} (\theta , z) \right] = \max  _ {t \in [ T ]} \mathbb {E} _ {z \sim P} \left[ \ell_ {t} (\theta , z) \right]
$$
holds for all  $\theta \in \Theta$
By the definition of  $\theta_{\mathrm{max}}^*$ , we have
$$
\begin{array}{l} \max  _ {\lambda \in \Lambda} \mathbb {E} _ {z \sim P} \left[ \ell_ {\lambda} \left(\theta_ {\max } ^ {*}, z\right) \right] = \max  _ {t \in [ T ]} \mathbb {E} _ {z \sim P} \left[ \ell_ {t} \left(\theta_ {\max } ^ {*}, z\right) \right] \\ \leq \max  _ {t \in [ T ]} \mathbb {E} _ {z \sim P} \left[ \ell_ {t} \left(\theta_ {\text {a v e r a g e}} ^ {*}, z\right) \right] \\ = \max  _ {\lambda \in \Lambda} \mathbb {E} _ {z \sim P} \left[ \ell_ {\lambda} \left(\theta_ {\text {a v e r a g e}} ^ {*}, z\right) \right], \\ \end{array}
$$
which is the result to prove.
B.2.2 Proof of Proposition 5.3
Proposition 5.3 is a standard result of gradient descent for strongly-convex and smooth functions. We include the proof here for completeness. By the convexity of  $f(x)$  and the choice of the step size  $\eta$  we have
$$
\begin{array}{l} f (x _ {k + 1}) \leq f (x _ {k}) + \left\langle \nabla f (x _ {k}), x _ {k + 1} - x _ {k} \right\rangle + \frac {L _ {f}}{2} \left\| x _ {k + 1} - x _ {k} \right\| ^ {2} \\ = f (x _ {k}) - \left(\frac {1}{\eta} - \frac {L}{2}\right) \| x _ {k + 1} - x _ {k} \| ^ {2} \\ \leq f \left(x _ {k}\right), \\ \end{array}
$$
which means the objective values are nonincreasing and implies  $f(x_{k}) \leq f(x_{0})$  for all  $k \in [K]$ .
By the  $\mu_f$ -strongly convexity at the point  $x^*$ , it holds for all  $x \in \mathbb{R}^d$  that
$$
\left\| x - x ^ {*} \right\| ^ {2} \leq \frac {2}{\mu_ {f}} \left(f (x) - f \left(x ^ {*}\right)\right). \tag {17}
$$
Combining (17) and the sequence  $\{f(x_k)\}_{k=0}^K$  decreasing, we obtain
$$
\left\| x _ {k} - x ^ {*} \right\| ^ {2} \leq \frac {2}{\mu_ {f}} \left(f \left(x _ {0}\right) - f \left(x ^ {*}\right)\right), \text {f o r a l l} k = 0, 1, \dots , K - 1,
$$
B.2.3 Proof of Theorem 5.4
The following proposition characterizes the sample complexity to find an  $\epsilon$ -approximately optimal parameter by ERM for a downstream task  $\lambda$  within the parameter space  $\Theta_{\lambda}(\theta_0)$ . Theorem 5.4 follows directly from Proposition B.1 by considering the worst-case downstream task  $\lambda \in \Lambda$ . The remaining is to prove Proposition B.1.
Proposition B.1. For a given task  $\lambda \in \Lambda$  and a parameter space  $\Theta_{\lambda}(\theta_0)$ , let the parameter  $\hat{\theta}_{\lambda}^{*} \in \Theta_{\lambda}(\theta_0)$  be the minimizer in the empirical risk for  $N_{\lambda}$  i.i.d. samples  $\{z_i\}_{i=1}^N$  from a distribution  $P$ , i.e.,  $\hat{\theta}_{\lambda}^{*} = \arg \min_{\theta \in \Theta_{\lambda}(\theta_0)} = \frac{1}{N_{\lambda}} \sum_{i=1}^{N_{\lambda}} \ell_{\lambda}(\theta, z_i)$ . The parameter  $\hat{\theta}_{\lambda}^{*}$  is  $\epsilon$ -approximately optimal with probability at least  $1 - \delta$  if
$$
N _ {\lambda} \geq \frac {8 d B ^ {2}}{\epsilon^ {2}} \log \left(1 + \frac {1 6 L ^ {\prime}}{\epsilon} \sqrt {\frac {2}{\mu} \mathbb {E} ^ {*}}\right) + \frac {8 B ^ {2}}{\epsilon^ {2}} \log \frac {2}{\delta}, \tag {18}
$$
where  $\mathbb{E}^* = \mathbb{E}_{z\sim P}\left[\ell_\lambda (\theta_0,z)\right]$
Denote  $\mathbb{E}_{z\sim P}\left[\ell_{\lambda}(\theta ,z)\right]$  as  $f_{\lambda}(\theta)$  and  $\frac{1}{N}\sum_{i = 1}^{N}\ell_{\lambda}(\theta ,z_i)$  as  $\hat{f}_{\lambda}(\theta)$ . First, we note
$$
\begin{array}{l} \Pr \left(f _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - f _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \geq \epsilon\right) \\ = \Pr \left(\left[ f _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - \hat {f} _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) \right] + \left[ \hat {f} _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - \hat {f} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \right] + \left[ \hat {f} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) - f _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \right] \geq \epsilon\right) \\ \stackrel {(A)} {\leq} \Pr \left(\left[ f _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - \hat {f} _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) \right] + \left[ \hat {f} _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) - f _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \right] \geq \epsilon\right) \tag {19} \\ \leq \Pr \left(2 \sup  _ {\theta \in \Theta_ {\lambda} \left(\theta_ {0}\right)} \left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \geq \epsilon\right) \\ = \Pr \left(\sup  _ {\theta \in \Theta_ {\lambda} \left(\theta_ {0}\right)} \left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \geq \frac {\epsilon}{2}\right). \\ \end{array}
$$
The inequality A is because  $\hat{f}_{\lambda}\left(\hat{\theta}_{\lambda}^{*}\right) - \hat{f}_{\lambda}\left(\theta_{\lambda}^{*}\right)\leq 0$  by the definition of  $\hat{\theta}_{\lambda}^{*}$ .
We derive the upper bound by covering numbers. We only consider Euclidean space for simplicity.
Definition B.2 (Covering numbers [63, Chapter 4]). Consider a subset  $S \subset \mathbb{R}^d$  and let  $\epsilon > 0$ . A subset  $\mathcal{N} \subset S$  is called an  $\epsilon$ -net of  $S$  if every point in  $S$  is within distance  $\epsilon$  of some points of  $\mathcal{N}$ , i.e., for all  $x \in S$ , there exists  $x_0 \in \mathcal{N}$  such that  $\| x - x_0 \| \leq \epsilon$ . The smallest possible cardinality of an  $\epsilon$ -net of  $S$  is called the covering number of  $S$  and is denoted  $C(S, \epsilon)$ , i.e.,
$$
C (S, \epsilon) := \min  \left\{\left| \mathcal {N} \right| \mid \mathcal {N} \text {i s a n} \epsilon \text {- n e t} \text {o f} S \right\}.
$$
Consider an  $\epsilon'$ -net  $\mathcal{N}(\Theta_{\lambda}(\theta_0), \epsilon')$  of  $\Theta_{\lambda}(\theta_0)$  where  $\epsilon' = \frac{\epsilon}{8L'}$ . By the definition of  $\epsilon'$ -net, we have
$$
\sup  _ {\theta \in \Theta_ {\lambda} \left(\theta_ {0}\right)} \left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \leq \sup  _ {\theta \in \mathcal {N} \left(\Theta_ {\lambda} \left(\theta_ {0}\right), \epsilon^ {\prime}\right)} \left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| + \frac {\epsilon}{4}. \tag {20}
$$
Combining (19) and (20), we obtain
$$
\begin{array}{l} \Pr \left(f _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - f _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \geq \epsilon\right) \\ \leq \Pr \left(\sup  _ {\theta \in \mathcal {N} \left(\Theta_ {\lambda} \left(\theta_ {0}\right), \epsilon^ {\prime}\right)} \left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \geq \frac {\epsilon}{4}\right) \tag {21} \\ \leq C \left(\Theta_ {\lambda} \left(\theta_ {0}\right), \epsilon^ {\prime}\right) \sup  _ {\theta \in \mathcal {N} \left(\Theta_ {\lambda} \left(\theta_ {0}\right), \epsilon^ {\prime}\right)} \Pr \left(\left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \geq \frac {\epsilon}{4}\right) \\ \end{array}
$$
We leverage the upper bounds of covering numbers of balls [63, Chapter 4].
Lemma B.3. The covering number of a ball of radius  $R$ , denoted as  $B_R$ , in  $\mathbb{R}^d$  satisfies
$$
C \left(B _ {R}, \epsilon\right) \leq \left(\frac {2 R}{\epsilon} + 1\right) ^ {d}.
$$
By Lemma B.3, we have
$$
C \left(\Theta_ {\lambda} \left(\theta_ {0}\right), \epsilon^ {\prime}\right) \leq \left(\frac {2 R _ {\lambda}}{\epsilon^ {\prime}} + 1\right) ^ {d}, \tag {22}
$$
where  $R_{\lambda} = \sqrt{\frac{2}{\mu}\mathbb{E}_{z\sim P}[\ell_{\lambda}(\theta_0,z)]}$
By Hoeffding's inequality, for each  $\theta \in \mathcal{N}(\Theta_{\lambda}(\theta_0),\epsilon ')$  , we have
$$
\Pr \left(\left| f _ {\lambda} (\theta) - \hat {f} _ {\lambda} (\theta) \right| \geq \frac {\epsilon}{4}\right) \leq 2 \exp \left(- \frac {n \epsilon^ {2}}{8 B ^ {2}}\right) \tag {23}
$$
Plugging (22) and (23) into (21), we have
$$
\Pr \left(f _ {\lambda} \left(\hat {\theta} _ {\lambda} ^ {*}\right) - f _ {\lambda} \left(\theta_ {\lambda} ^ {*}\right) \geq \epsilon\right) \leq 2 \left(\frac {2 R _ {\lambda}}{\epsilon^ {\prime}} + 1\right) ^ {d} \exp \left(- \frac {N _ {\lambda} \epsilon^ {2}}{8 B ^ {2}}\right). \tag {24}
$$
By (24), when the number of samples  $N_{\lambda}$  satisfies
$$
N _ {\lambda} \geq \frac {8 d B ^ {2}}{\epsilon^ {2}} \log \left(1 + \frac {1 6 L ^ {\prime}}{\epsilon} \sqrt {\frac {2}{\mu} \mathbb {E} _ {z \sim P} [ \ell_ {\lambda} (\theta_ {0} , z) ]}\right) + \frac {8 B ^ {2}}{\epsilon^ {2}} \log \frac {2}{\delta},
$$
we have  $\operatorname*{Pr}\left(f_{\lambda}\left(\hat{\theta}_{\lambda}^{*}\right) - f_{\lambda}\left(\theta_{\lambda}^{*}\right)\geq \epsilon\right)\leq \delta.$
C Example for Remark 5.2
Consider an extreme example where  $\Theta = \mathbb{R}$ ,  $\mathbb{E}_{z\sim P}[\ell_1(\theta ,z)] = A(\theta -1)^2$  where  $1 < A < T$  and  $\mathbb{E}_{z\sim P}[\ell_t(\theta ,z)] = \theta^2$  for  $t = 2,\ldots ,T$ . Then we have  $\theta_{\mathrm{max}}^{*} = \frac{\sqrt{A}}{\sqrt{A} - 1}$  and  $\theta_{\mathrm{average}}^{*} = \frac{A}{A + T - 1}$ . It further implies that
$$
\ell_ {\max } := \max  _ {\lambda \in \Lambda} \mathbb {E} _ {z \sim P} \left[ \ell_ {\lambda} \left(\theta_ {\max } ^ {*}, z\right) \right] = \frac {A}{\left(\sqrt {A} - 1\right) ^ {2}},
$$
$$
\ell_ {\text {a v e r a g e}} := \max  _ {\lambda \in \Lambda} \mathbb {E} _ {z \sim P} \left[ \ell_ {\lambda} \left(\theta_ {\text {a v e r a g e}} ^ {*}, z\right) \right] = \frac {A (T - 1) ^ {2}}{(A + T - 1) ^ {2}}.
$$
If  $A = T - 1$ , we have  $\frac{\ell_{\text{average}}}{\ell_{\text{max}}} \geq \frac{1}{4} (\sqrt{T - 1} - 1)^2$ , which is as large as  $O(T)$ .
D Training details
D.1 Part-of-Speech Mask BERT Training Setting
In this work, we denote the number of layers (i.e., Transformer blocks) as  $L$ , the hidden size as  $H$ , and the number of self-attention heads as  $A$ . For comparison purposes, we primarily report results on two models with the same size: PoS-BERT<sub>BASE</sub> and BERT<sub>BASE</sub>(L=12, H=768, A=12, Total Parameters=110M). The model is trained with AdamW [42] by setting  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ ,  $\epsilon = 1\mathrm{e}-6$ , and  $L_{2}$  weight decay of 0.01. The learning rate is warmed up over the first 10K steps to a peak value of 1e-4, then linearly decayed. We duplicate training data ten times to avoid using the same mask for each training instance in every epoch so that each sequence is masked in 10 different ways over the 40 training epochs. Thus, each training sequence was seen with the same mask four times. The hyperparameters for experiments are shown as Table 4 and Table 5.
Hyperparam	Part-of-Speech Mask BERT
Number of Layers	12
Hidden size	768
Attention heads	12
Attention heads size	64
Dropout	0.1
Warmup steps	10K
Peak Learning Rate	2e-4
Batch Size	256
Weight Decay	0.01
Max Steps	1000K
Learning Rate Decay	Linear
Adam ε	1e-6
Adam β1	0.9
Adam β2	0.999
Gradient Clipping	0.0
Hyperparam	GLUE
Learning Rate	2e-5
Batch Size	32
Weight Decay	0.1
Learning Rate Decay	Linear
Warmup Ratio	0.06
D.2 Multi-Modal Mask MAE Training Setting
We use Vit-B [21] with a patch size of  $16 \times 16$  pixels as the backbone for our MAE experiments, and estimate the model's performance under different pre-training epochs, i.e., 400 and 1,600 epochs on ImageNet1K and 800 epochs on ImageNetS50. We choose AdamW as the optimizer with a base learning rate of 1e-4 and weight decay of 0.05. We first warm up the learning rate with 40 epochs and then decay it with cosine decay [41]. We set the batch size to 2048 and trained the models using  $8 \times \mathrm{A}100$  GPUs with automatic mixed precision enabled. Our data augmentations are straightforward. We randomly crop the images, setting the random scale between 0.2 and 1.0 and the random aspect ratio between 0.75 and 1.33. Afterward, we resize the crops to  $224 \times 224$  pixels and apply a random horizontal flip with a probability of 0.5. The hyperparameters for experiments are shown as Table 6 and Table 7.
Table 5: Hyperparameters for finetuning Part-of-Speech Mask BERT on GLUE.
Table 4: Hyperparameters for pre-training Part-of-Speech Mask BERT.
Hyperparam	{None, GradNorm, DWA}	Uncertainty	Minimax
Batch Size	2048	2048	2048
Learning Rate	8e-4	8e-4	8e-4
Min Learning Rate	1e-6	1e-6	1e-6
Weight Decay	0.05	0.05	0.05
Adamw ε	1e-8	1e-8	1e-8
Adamw β1	0.9	0.9	0.9
Adamw β2	0.95	0.95	0.95
Epoch	{800}	{400, 800, 1600}	{400, 800, 1600}
Warm up Epoch	40	40	40
Learning Rate Schedule	cosine decay	cosine decay	cosine decay
Non-masked tokens	98	98	98
Input resolution	224×224	224×224	224×224
Augmentation	RandomResizeCrop	RandomResizeCrop	RandomResizeCrop
Dropout	0.0	0.0	0.0
Patch Size	16	16	16
Table 6: Hyperparameters for pre-training Multi-Modal Mask MAE. We only pre-train 800 epochs on ImageNetS50, and pre-train both 400 and 1600 epochs on ImageNet1K.
Hyperparam	Classification	Semantic Segmentation	Depth
ImageNet1K	ImageNetS50	ImageNetS50	NYUv2	NYUv2
Epoch	100	100	100	100	2000
Warm up Epoch	5	5	20	20	100
Batch Size	1024	1024	1024	1024	2048
Learning Rate	4e-3	4e-3	1e-4	1e-4	1e-4
Min Learning Rate	1e-6	1e-6	1e-6	1e-6	0
Weight Decay	0.05	0.05	0.05	0.05	1e-4
Adamw β1	0.9	0.9	0.9	0.9	0.9
Adamw β2	0.999	0.999	0.999	0.999	0.999
Layer Decay	0.65	0.65	0.75	0.75	0.75
Patch Size	16	16	16	16	16
Drop path	0.1	0.1	0.1	0.1	/
LR Schedule	cosine decay	cosine decay	cosine decay	cosine decay	cosine decay
Input resolution	224×224	224×224	224×224	224×224	256×256
Augmentation	Rand(9, 0.5)	Rand(9, 0.5)	LSJ	LSJ	LSJ
Table 7: Hyperparameters for fine-tuning Multi-Modal Mask MAE on various downtasks. The augmentation strategy LSJ is large scale jittering [26]. And we use drop path [31] in classification and semantic segmentation tasks.
Contemporary pre-training models are consistently enlarging in size. However, due to limitations associated with computational power and the non-disclosure tendency of large-scale models, we were unable to conduct our experimentation directly on ultra-large models such as LLaMA [58], GPT3 [9], and V-MoE [52]. Notwithstanding, we have validated our hypothesis on two commonly encountered domains and model frameworks, thus illustrating the extensive applicability of our proposed methodology.