SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition H.3 (Tracking Probe/Linear Probe). When optimizing model parameters  $\omega_{t}$  of an SSL method, simultaneously optimize the parameters  $\xi$  of a probe model  $r(\cdot ;\xi)$  under a downstream objective  $\mathcal{L}^{(d)}$ . For example, in classification, with data  $x$  and samples  $y$
$$
\mathcal {L} ^ {(d)} (x, y, \theta_ {t}, \xi_ {t}) = - \log P (y | r (\text {S t o p G r a d} (h (x; \omega_ {t})); \xi)) \tag {76}
$$
$$
\mathcal {L} ^ {(t o t a l)} (x, y; \boldsymbol {\theta} _ {t}, \boldsymbol {\omega} _ {t}, \zeta_ {t}, \xi_ {t}) = \mathcal {L} (x; \boldsymbol {\theta} _ {t}, \boldsymbol {\omega} _ {t}, \zeta_ {t}) + \mathcal {L} ^ {(d)} (x, y, \boldsymbol {\omega} _ {t}, \xi_ {t}), \tag {77}
$$
The is a probe for the teacher, which is typically the better choice due to Polyak-Ruppert averaging effects (see Section 3.2). When the  $r$  is a linear model, the tracking probe is called a linear probe.
Table 9: BYOL ResNet-18 hyperparameters for CIFAR10
ResNet-18
Weight initialization	kaiming.uniform (He et al., 2015)
Backbone normalization	BatchNorm
Head normalization	BatchNorm
Synchronized BatchNorm over replicas	Yes
Learning rate schedule	Single Cycle Cosine
Learning rate warmup (epochs)	20
Learning rate minimum value	0
Training duration (epochs)	100
Optimizer	SGD
Optimizer scaling rule	SGD
Optimizer momentum	0.9
Gradient clipping	0.1
Base learning rate	0.02
Base batch size	1024
Base teacher momentum	0.992
Weight decay	\( 1 \times 10^{-6} \)
Weight decay scaling rule	None
Weight decay skip bias	Yes
Numerical precision	tf32
Augmentation stack	BYOL CIFAR10
It is also typical to use a Batch Normalization layer without trainable affine terms before this linear layer as in He et al. (2022) to stabilize probe training. In this case, the running statistics can be absorbed into a definition of the linear layer weights and biases, and so this is still a linear probe, although we will call this a pre-bn linear probe to remove ambiguity.
H.2 A ResNet-18 recipe for BYOL
Hyperparameters We present the base hyperparameters for training BYOL with a ResNet-18 backbone using SGD in Table 9. This recipe was developed by starting from a well-known BYOL ResNet-50 recipe (Grill et al., 2020), adapting the input augmentations for CIFAR10, and performing a search over learning rate choices for an SGD optimizer.
H.3 A Vision Transformer recipe for BYOL
Hyperparameters We present the base hyperparameters for training BYOL with a ViT-B/16 backbone in Table 10. This recipe was developed by starting from a well-known supervised ViT-B/16 recipe (He et al., 2022) and performing a search over weight decay and learning rate hyperparameter choices. We find that BYOL performs well with heavy weight decay ( $\lambda = 0.3$ ) and a low learning rate ( $\eta = 10^{-3}$ ) at a base batch size  $B = 4096$ . The AdamW optimizer is used, and so for scaling to other batch sizes  $\hat{B} = \kappa B$  we use the Adam Scaling Rule (Definition C.3) $^{15}$ . We use a pre-bn linear probe as discussed in Appendix H.1. Finally, the performance of BYOL can be further improved by employing multicrop (Caron et al., 2020) by  $\approx +2\%$  in absolute test top-1 performance on ImageNet1k compared to without multicrop, however, as this is not our focus, we omit this from the presented recipe.
Additional background Achieving large scale SSL training with ViTs to large scale SSL training has been a long standing goal in the community. MoCo-v3 (Chen et al., 2021) enables the use of ViTs with contrastive learning, but achieves this through modifications of the ViT training procedures, including gradient freezing on the image patching layer, and re-introducing Batch Normalization to post-attention MLP layers. Despite these modifications, MoCo-v3 was only trained up to a batch size of 6144, where model performance begins to suffer (Chen et al., 2021). In Figure 6 we demonstrate that combining dynamic batch scaling (Appendix C.4) with the EMA Scaling Rule
<sup>15</sup>We note that Adam (Kingma & Ba, 2015) and AdamW (Loshchilov & Hutter, 2019) are equivalent in the limit of zero weight decay, and that the Adam Scaling Rule (Definition C.3) was derived with zero weight decay (Malladi et al., 2022).
Table 10: BYOL ViT-B/16 hyperparameters.
BYOL ViT-B/16
ImageNet1k Linear Probe Test Top-1	74.47% (Figure 19)
Weight initialization	trunc_normal (.02)
Backbone normalization	LayerNorm
Head normalization	BatchNorm
Synchronized BatchNorm over replicas	No
Learning rate schedule	Single Cycle Cosine
Learning rate warmup (epochs)	40
Learning rate minimum value	1 × 10-6
Training duration (epochs)	480
Optimizer	AdamW
Optimizer scaling rule	Adam
Base (β1, β2)	(0.9, 0.95)
Base learning rate	1 × 10-3
Base batch size	4096
Base teacher momentum	0.99
Weight decay	0.3
Weight decay scaling rule	None
Weight decay skip bias	Yes
Numerical precision	bf16
Augmentation stack	BYOL (Grill et al., 2020)
Stochastic depth	0.1
(Definition 1.2) enables BYOL to be trained using ViTs to batch sizes of 24,576 without any drop in performance compared to the reference batch size of 4096. We emphasize that the piecewise transitions in the schedules are important for preserving training dynamics.
H.4 The role of Batch Normalization and Layer Normalization in BYOL with ViTs
Figure 19: BYOL ViT-B/16 on ImageNet1k for different scalings  $\kappa$ . We present runs comparing LayerNorm (blue) to BatchNorm (red) in the projection and prediction heads of BYOL ViT models for batch size 3072 (dashed) and 24,576 (solid) without the EMA Scaling Rule.  $\kappa = 1$  corresponds to  $B = 4096$ . In all scenarios the transformer backbone only uses LayerNorm. We truncate the training of the large batch size LayerNorm variant to preserve compute (indicated by  $\times$ ).
Here we compare the roles of Batch Normalization (BatchNorm, Ioffe & Szegedy (2015)) and Layer Normalization (LayerNorm, Ba et al. (2016)) in the projection and prediction heads of BYOL (Grill et al., 2020) using ViTs.
It has been observed that BatchNorm plays a critical role in BYOL predictor and projector dynamics (Fetterman & Albrecht, 2020), and using either LayerNorm or no normalization significantly decreases model performance. Subsequently, it was demonstrated (Richemond et al., 2020) that competitive BYOL performance could be achieved through a combination of Group Normaliza
tion (GroupNorm, Wu & He (2018)) and Weight Standardization (Qiao et al., 2019). Additionally, Richemond et al. (2020) showed that if BatchNorm is used in the backbone, one can use LayerNorm or no normalization in the predictor and projector without any performance drop.
In this work, we show it is possible to train BYOL ViT using only LayerNorm across the backbone, projector and predictor (see Figure 19), decoupling BYOL's reliance on batch statistics, a desirable trait for a representation learning algorithm (Brock et al., 2021). At batch size 3072, using LayerNorm in the predictor and projector achieves competitive performance  $(74.10\%)$ , performing slightly worse than using BatchNorm  $(74.47\%)$ . At the larger batch size of 24,576, runs perform significantly worse as the EMA Scaling Rule was not applied.
H.5 Longer training duration with incremental Progressive Scaling
Figure 20: BYOL ViT-B/16 on ImageNet1k for different scalings  $\kappa$ . The baseline model ( $\kappa = 0.75$ , black dashed) uses batch size 3072 and teacher momentum  $\rho_{B} = 0.99$ . We increment the batch size by 3072 every 60 epochs to a final batch size of 24,576 using Progressive Scaling (Definition 3.2).
Here we use the same base hyperparameters as Table 10, except that we train for 480 instead of 300 epochs. To mitigate the student impulse phenomena discussed in Section 3.4, in Figure 20 we investigate increasing the batch size every 60 epochs using Progressive Scaling (Definition 3.2). We observe that this more gradual procedure enables closer tracking of the baseline train loss trajectory. Additionally, this procedure results in a scaled linear probe performance that outperforms the baseline (75.64% compared to the baseline performance of 74.47%). The same procedure can be applied to the LayerNorm variant discussed in Appendix H.4, which produces a similar result (75.09% compared to the baseline performance of 74.10%).
H.6 Building intuition around Progressive Scaling and momentum sensitivity
Our final BYOL ViT results are to help build intuition around Progressive Scaling (Definition 3.2), as well as when the EMA Scaling Rule is most important. In Figure 21 we explore transitioning from the baseline batch size 4096 model to batch size 24,576 in a single transition after 60 epochs. After this transition, we continue training for 240 epochs for a range of momenta:  $\rho \in \{0.8,0.9,0.95,0.97,0.9867,0.994,0.999\}$  without the EMA Scaling Rule.
We observe that after the transition, any  $0.9 \leq \rho \leq 0.994$  produces a linear probe performance that matches or outperforms the baseline at the end of training. This indicates that after the initial training period, BYOL becomes less sensitive to the choice of teacher momentum. Note that without the initial 60 epochs of training with batch size 4096, all models, including those employing the EMA Scaling Rule diverge (see  $B = 24$ , 576 in Figure 6).
We present an illustration for why this might happen in Figure 22. First, we see that using the EMA Scaling Rule always keeps the model within the acceptable momentum region. We also wee that not using the EMA Scaling Rule can keep the model within the acceptable momentum region for a range of batch sizes, depending on how large wide in momenta the acceptable region is at the base batch size. Finally, we see that the momentum value matters much more at low values of momenta
Figure 21: BYOL ViT-B/16 on ImageNet1k for different momenta  $\rho$ . The baseline model ( $\rho = 0.99$ , black dashed) uses batch size 4096. At the 60th epoch we apply Progressive Scaling (Definition 3.2) and transition to batch size 24576. We train for a further 240 epochs without EMA scaling for a range of momenta:  $\rho \in \{0.9, 0.95, 0.97, 0.9867, 0.994\}$ .
(the acceptable momentum region shrinks), whereas at large momenta, this region of acceptability widens.
Figure 22: A hypothetical scenario where there is an upper and lower limit for momenta qualitatively leading to the same result. We assume at base batch size  $B = 1024$  there is an upper  $(\rho_{\mathrm{max}}$ , black dashdot) and lower  $(\rho_{\mathrm{min}}$ , black dashed) limit for valid momenta. We show what happens if we start with  $\rho_B = 0.95$  at a batch size of 4096, and scale with  $(\rho = \rho_B^\kappa$ , blue) and without  $(\rho = \rho_B$ , red) the EMA Scaling Rule.
H.7 Compute usage for ViT BYOL investigation
We now summarize the compute usage for the BYOL ViT experiments. First we detail cost of reproducing Figure 6 in Table 11, Figure 19 in Table 12, and Figure 20 in Table 13.
Table 11: Compute usage for baseline ViT BYOL investigation in Figure 6. Values include node allocation times (typically a small % of corresponding total runtime), giving a practical estimate of reproduction cost. All experiments conducted are using 80Gb A100s, and experiments indicated by  $(\dagger)$  have a faster interconnect.
Batch Size	GPUs	Time (h)	Compute/Run (GPUh)	Runs	Compute (GPUh)
4,096	32	16.6	531.4	2	1,062.7
8,192	48	14.1	678.0	2	1,356.1
16,384	96	8.3	800.3	2	1,600.6
24,576	128	6.6	850.1	4	3,400.4
32,768†	176	4.1	721.7	2	1,443.4
Total	8,863.1
Table 12: Compute usage for ViT BYOL investigation into BatchNorm and LayerNorm variants in Figure 19. Values include node allocation times (typically a small % of corresponding total runtime), giving a practical estimate of reproduction cost. All experiments conducted are using 80Gb A100s, and were run for 480 epochs, except those indicated by  $(\ast)$  were truncated early (see Figure 19 for more details).
Batch Size	Normalization	GPUs	Time (h)	Compute (GPUh)
3,072	BatchNorm	16	47.9	766.0
3,072	LayerNorm	16	48.0	768.7
24,576	BatchNorm	128	14.8	1900.4
24,576*	LayerNorm	128	3.5	451.1
Total	3,886.2
Table 13: Compute usage for ViT BYOL investigation into incremental scaling in Figure 20. Values include node allocation times (typically a small % of corresponding total runtime), giving a practical estimate of reproduction cost. All experiments conducted are using 80Gb A100s for 60 epochs. Stage 0 corresponding to the baseline in Figure 20 is the run detailed in the first row of Table 12, using a batch size of 3,072, Batch Normalization, and 16 GPUs. Computing only the first 60 epochs of stage 0 corresponds to approximately 127.7 GPUh, which would bring the total cost of Figure 20 to 1,432.9 GPUh.
Stage	Batch Size	GPUs	Time (h)	Compute (GPUh)
1	6,144	32	3.5	113.0
2	9,216	48	3.1	149.8
3	12,288	64	2.8	176.0
4	15,360	80	2.3	186.5
5	18,432	96	2.1	202.9
6	21,504	112	2.1	235.8
7	24,576	128	1.9	241.3
Total	1,305.2
Next, the cost of a single momentum ablation presented in Figure 21 is 240 epochs at batch size 24,576, which is  $\approx 240 / 480 \times 1900.4$  GPUh = 950.2 GPUh, giving a total cost over seven runs of 6651.4 GPUh.
Finally, providing a full view of the investigations carried out for the ViT BYOL is given in Table 14.
Table 14: Total compute usage for ViT BYOL investigations.
Compute (GPUh)
Baselines (Figure 6 and Table 11)	8,863.1
BatchNorm and LayerNorm (Figure 19 and Table 12)	3,886.2
Incremental scaling (Figure 20 and Table 13)	1,305.2
Momentum ablations (Figure 21)	6,651.4
All other compute, e.g. code development, runs with errors, and debugging	84,984.1
Total	105,690.0
H.8 ResNet-18 hyperparameter sensitivity analysis
To demonstrate that the EMA Scaling Rule works for a broad range of optimization hyperparameters (i.e. beyond those presented in Figure 5 and Section 3.4), we provide a sensitivity analysis for base teacher momentum  $\rho_{B}$  and base learning rate  $\eta_{B}$  in the challenging setting of BYOL.
Base teacher momentum In Figure 23 we show the effect of changing the base teacher momentum  $\rho_{B}$ , defined at batch size 1024. The EMA Scaling Rule is robust to modifications of momentum down to  $\rho_{B} \approx 0.946$  in this particular setting. Below  $\rho_{B} \approx 0.946$ , matching is poor, although the smallest momentum in this setting corresponds to  $0.841^{4} \approx 0.5$ , which is a particularly small teacher momentum, and is unlike to provide utility over the using the target model (see Appendix E.2).
(a)  $\rho_{B} = 0.841$
(b)  $\rho_{B} = 0.946$
(c)  $\rho_{B} = 0.974$
(e)  $\rho_B = 0.992$
(d)  $\rho_{B} = 0.987$
(f)  $\rho_{B} = 0.997$
Figure 23: ResNet-18 BYOL on CIFAR10 teacher momentum sensitivity  $(\eta_B = 0.08)$  for scalings  $\kappa \in \{2,4\}$  and base teacher momenta  $\rho_{B} \in \{0.841, 0.946, 0.974, 0.987, 0.992, 0.997\}$  defined at  $\kappa = 1$ . The baseline  $(\kappa = 1$ , black dashed) uses batch size 1024, and is scaled from batch size 2048 (left) to 4096 (right) with (blue,  $\rho = \rho_{B}^{\kappa}$ ) and without (red,  $\rho = \rho_{B}$ ) the EMA Scaling Rule. Bands indicate mean and standard deviation across three runs.
Base learning rate In Figure 24 we show the effect of changing the base learning rate  $\eta_{B}$ , defined at batch size 1024. The EMA Scaling Rule is robust over a wide range of learning rates. At the largest learning rate  $\eta_{B} = 0.5$  matching starts to become poor at scaling  $\kappa = 4$ .
(a)  $\eta_B = 0.01$
(b)  $\eta_B = 0.02$
(c)  $\eta_{B} = 0.04$
(e)  $\eta_B = 0.2$
(d)  $\eta_B = 0.15$
(f)  $\eta_B = 0.5$
Figure 24: ResNet-18 BYOL on CIFAR10 learning rate sensitivity  $(\rho_B = 0.992)$  for scalings  $\kappa \in \{2,4\}$  and base learning rates  $\eta_B \in \{0.01,0.02,0.04,0.15,0.20,0.50\}$  defined at  $\kappa = 1$ . The baseline  $(\kappa = 1$ , black dashed) uses batch size 1024, and is scaled from batch size 2048 (left) to 4096 (right) with (blue,  $\rho = \rho_B^\kappa$ ) and without (red,  $\rho = \rho_B$ ) the EMA Scaling Rule. Bands indicate mean and standard deviation across three runs.
H.9 ResNet-18 additional scaling analysis
To demonstrate that the EMA Scaling Rule works for a broad range of scalings  $\kappa$  (i.e. beyond those presented in Figure 5 and Section 3.4), we investigate scaling down to  $\kappa = 1/8$  in Figure 25. We see that the EMA Scaling Rule works well down to the small batch size of 128, although matching is not perfect. We suspect this is due to the presence of Batch Normalization layers in the ResNet-18 architecture, which underperform at small batch sizes (Ioffe & Szegedy, 2015). The synthetic analysis of Section 3.1 instead demonstrated the EMA Scaling Rule holding for scalings spanning factors of  $\kappa$  that differ by 1024, with scaling error insensitive to the value of  $\kappa$  for sufficiently low  $\kappa$  (see Figure 1b).
Figure 25: ResNet-18 BYOL on CIFAR10 lower scaling analysis ( $\eta_B = 0.08$ ,  $\rho_B = 0.992$ ) for scalings  $\kappa \in \{1/8, 1/4, 1/2\}$ . The baseline ( $\kappa = 1$ , black dashed) uses batch size 1024, and is scaled from batch size 128 (left) to 512 (right) with (blue,  $\rho = \rho_B^\kappa$ ) and without (red,  $\rho = \rho_B$ ) the EMA Scaling Rule. Bands indicate mean and standard deviation across three runs.
H.10 Scaling a ResNet-50 BYOL using LARS and Progressive Scaling
Here we investigate whether Progressive Scaling and the EMA Scaling Rule can be used in practice where there is no known optimizer SDE approximation. We use the default 300 epoch configuration for BYOL (Grill et al., 2020) in Figure 26. We see that although trajectories during training do not match, we are able to match or surpass the linear probe performance of the BYOL baseline at the larger batch size if 32,768. This indicates that the contributions of our work have practical utility beyond the theoretical constraints.
The compute usage for the BYOL ResNet using LARS is detailed in Table 15.
Table 15: Compute usage for ResNet 50 LARS investigation in Figure 26. Values include node allocation times (typically a small % of corresponding total runtime), giving a practical estimate of reproduction cost. All experiments conducted are using 80Gb A100s.
Batch Size	GPUs	Time (h)	Compute (GPUh)
4,096 → 32,768 (120 Epochs)	128	14.1	1809.8
4,096 → 32,768 (60 Epochs)	128	12.9	1655.9
4,096	16	32.8	524.9
All other compute, e.g. code development, runs with errors, and debugging	17,654.6
Total	21645.2
Figure 26: ResNet50 BYOL on ImageNet1k using LARS for different configurations of progressive scaling. The baseline (black dashed) uses batch size 4096 and momentum  $\rho_{B} = 0.99$ . We consider progressive scaling (blue) smoothly from epoch 60 for 60 epochs (left) and 120 epochs (right) up until batch size 32,768, scaling the learning rate linearly, and applying the EMA Scaling Rule.
H.11 Preventing collapse phenomena in DINO at scale
Until now, our representatives SSL method has been BYOL for reasons discussed in Section 3.4. Here, we will turn our attention to DIstillation with NO labels (DINO) (Caron et al., 2021), which has the update rule presented in Definition H.4.
Definition H.4 (DINO Update). DINO learns unsupervised features by matching predictions over emergent pseudo-labels of a student backbone and head  $f(\cdot ;\theta)$  to those of an EMA teacher  $f(\cdot ;\zeta)$  through a cross-entropy guided distillation procedure. DINO has a additional centering procedure, which is a form of batch normalization with momentum  $\rho_{c} = 0.9$  which we do not scale using the
Table 16: DINO ViT-B/16 Training hyperparameters.
DINO ViT-B/16
CIFAR10 Linear Probe Top-1 (ρB=0.996)	85.38%
CIFAR10 Linear Probe Top-1 (ρB=0.992)	86.96%
Weight initialization	trunc_normal(.02)
Normalization	Layer Norm
Learning rate schedule	Single Cycle Cosine
Learning rate warmup (epochs)	50
Learning rate minimum value	1×10-6
Training duration (epochs)	280
Optimizer	AdamW
Optimizer scaling rule	Adam
Base (β1, β2)	(0.9, 0.95)
Base learning rate	3×10-4
Base batch size (B)	1024
Base teacher momentum (ρB)	0.992 or 0.996
Base weight decay	0.04
Weight decay scaling rule	Linear
Weight decay skip bias	Yes
Center Momentum	0.9
Center Momentum Scaling Rule	None
Precision	bf16
Augmentation stack	DINO multi-crop (Caron et al., 2020)
EMA Scaling Rule. The update for the parameters of DINO is
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} - \eta \times \frac {1}{B} \sum_ {x \in \mathbb {B}} \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(x; \boldsymbol {\theta} _ {t}, \zeta_ {t}, \mathbf {c} _ {t}\right) \tag {78}
$$
$$
\zeta_ {t + 1} = \rho \zeta_ {t} + (1 - \rho) \theta_ {t + 1} \tag {79}
$$
$$
\mathbf {c} _ {t + 1} = \rho_ {c} \mathbf {c} _ {t} + \left(1 - \rho_ {c}\right) \mathbb {E} _ {x ^ {\prime}} \zeta \left(x ^ {\prime}\right) \tag {80}
$$
$$
\text {w i t h} \mathcal {L} \left(x; \boldsymbol {\theta} _ {t}, \zeta_ {t}, \mathbf {c} _ {t}\right) = H \left(f \left(x _ {1}, \boldsymbol {\theta} _ {t}\right), f \left(x _ {2}, \zeta_ {t}\right) - \mathbf {c} _ {t}\right) + \left(x _ {1} \leftrightarrow x _ {2}\right), \tag {81}
$$
where  $H(\mathbf{a},\mathbf{b}) \equiv -\sum_{m = 1}^{M}p_{m}(\mathbf{a})\log p_{m}(\mathbf{b})$  is the cross-entropy between categorical distributions over  $M$  (emergent pseudo-)classes given logits  $\mathbf{a},\mathbf{b} \in \mathbb{R}^{M}$ ,  $x_{1}$  and  $x_{2}$  are two views of a single variate  $x$ , often produced by augmentations, and  $x_{1} \leftrightarrow x_{2}$  denotes symmetrization over  $x_{1}$  and  $x_{2}$ .
In practice, DINO employs multi-crop (Caron et al., 2021). We omit this detail for clarity of presentation, although we do use multi-crop in the experiments that follow.
Our interest DINO is due to the difficulty in its optimization $^{16}$ , and in particular, preventing collapse phenomena in DINO at batch sizes above 1024, which is an open research problem. In this section, we will show that a combination of the EMA Scaling Rule (Definition 1.2) and Progressive Scaling (Definition 3.2) enable training of DINO beyond batch size 1024 without sacrificing performance.
Hyperparameters Base hyperparameters are presented in Table 16.
Results In Figures 27 and 28 we show the results obtained training DINO on CIFAR-10 with  $\rho_{B} = 0.996$  and  $\rho_{B} = 0.992$  respectively at the reference batch size of 1024. We employ smooth Progressive Scaling (Definition 3.2) between epochs 120 and 180.
At batch size 2048, the training loss matches the reference only when the EMA Scaling Rule is applied, whereas the run without the scaling rule diverges from the reference. The impact of this divergence is emphasized as we consider the larger batch size of 4096. Here, there is also a gap with the EMA Scaling Rule, however is approximately three times smaller than the gap without the EMA Scaling Rule.
<sup>16</sup>For an example, see https://github.com/facebookresearch/dino/issues/43#issuecomment-881453515.
Additionally, we observe that using  $\rho_{B} = 0.992$  yields higher Top-1 accuracy over  $\rho_{B} = 0.996$ , and in our experiments, using the EMA Scaling Rule always performs better in terms of linear probe performance than not using the scaling rule.
Figure 27: DINO ViT-B/16 on CIFAR-10 for different scalings  $\kappa$  and base teacher momentum  $\rho_{B} = 0.996$ . The baseline model ( $\kappa = 1$ , black dashed) uses batch size 1024 and center momentum  $\rho_{c} = 0.9$ , and is scaled up from batch size 2048 (left) to 4096 (right) with (blue,  $\rho = \rho_{B}^{\kappa}$ ) and without (red,  $\rho = \rho_{B}$ ) the EMA Scaling Rule. Between epochs 100 and 180 we scale the batch size using progressive scaling (Definition 3.2).
Figure 28: DINO ViT-B/16 on CIFAR-10 for different scalings  $\kappa$  and base teacher momentum  $\rho_{B} = 0.992$ . The baseline model ( $\kappa = 1$ , black dashed) uses batch size 1024 and center momentum  $\rho_{c} = 0.9$ , and is scaled up from batch size 2048 (left) to 4096 (right) with (blue,  $\rho = \rho_{B}^{\kappa}$ ) and without (red,  $\rho = \rho_{B}$ ) the EMA Scaling Rule. Between epochs 100 and 180 we scale the batch size using progressive scaling (Definition 3.2).