SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Fig. 4, we observed that SuperGLUE is particularly exceptional in terms of the diminishing returns with respect to the model capacity. For example, in any of the other plots (GLUE, SQuAD, TriviaQA), we see that the dense baselines fall well below the lines formed by AltUp's data points. In order to obtain a more direct comparison on SuperGLUE, we compared  $\mathrm{L} + \mathrm{AltUp}$  with a size-matched dense baseline - T5L+4, which adds 4 encoder layers and 4 decoder layers to the T5L model. Note this model has the same number of parameters as  $\mathrm{L} + \mathrm{AltUp}$ , but uses more computation and is roughly  $10\%$  slower than  $\mathrm{L} + \mathrm{AltUp}$ . On SuperGLUE, T5L+4 achieves an average score of 82.57, versus 82.75 for  $\mathrm{L} + \mathrm{AltUp}$ . Therefore, AltUp is not only faster, but more performant when compared to a dense baseline. Note that this is specific to SuperGLUE, and we see larger speedups - up to  $87\%$  - on other datasets.
F Sequence-AltUp Details
Here, we provide the full pseudocode of Sequence-AltUp from Sec. 4 (see Alg. 2).
Algorithm 2 AltUp extension to sequence dimension
Input: A sequence of vectors  $x = (x_0, x_2, \dots, x_{T-1})$ , where  $x_i \in \mathbb{R}^d$ . Transformer layer  $\mathcal{L}$  and stride parameter  $k$ .
Output: A sequence of vectors  $y = (y_0, y_2, \dots, y_{T-1})$ .
1: Prediction: predict the output sequence with a trainable linear map:
$$
\hat {y} _ {i} = a _ {1} x _ {i} + a _ {2} x _ {\lfloor i / k \rfloor * k}
$$
for  $i = 0,1,\ldots ,T - 1$ , where  $a_1,a_2\in \mathbb{R}$  are trainable scalars;
2: Computation: subsample the input sequence with stride  $k$  and apply the transformer layer on the subsampled sequence:
$$
\left(\tilde {y} _ {0}, \tilde {y} _ {k}, \dots , \tilde {y} _ {\lfloor (T - 1) / k \rfloor * k}\right) = \mathcal {L} \left(x _ {0}, x _ {k}, \dots , x _ {\lfloor (T - 1) / k \rfloor * k}\right);
$$
3: Correction: correct the prediction with the computation result:
$$
y _ {i} = \hat {y} _ {i} + b \left(\tilde {y} _ {\lfloor i / k \rfloor * k} - \hat {y} _ {\lfloor i / k \rfloor * k}\right)
$$
for  $i = 0,1,\dots,T - 1$ , where  $b\in \mathbb{R}$  is a trainable scalar.
G Recycled-AltUp Fine-tune Evaluations
We conclude the supplementary material by presenting evaluations with Recycled-AltUp as described in Sec. 4. Table 8 presents the results of our pretrain and fine-tune evaluations on T5 Small, Base, and Large. The pretrain accuracy is the one reported at  $500k$  steps, and we fine-tune for an additional  $50k$  steps for the fine-tune evaluations.
Consistent with our results presented in the main body of our paper, we observe that AltUp and Recycled-AltUp both provide clear and consistent gains over the baseline on virtually all pretrain
Model	Pretrain Acc.	GLUE	SG	SQuAD (EM/F1)	TriviaQA (EM/F1)
S	61.21	75.83	59.52	76.44/84.97	19.03/22.83
S + Recycled-AltUp	61.33	77.24	59.12	77.76/85.64	19.06/22.77
S + AltUp	61.86	76.82	59.60	77.51/85.79	19.27/22.95
B	66.42	84.25	73.56	83.78/91.19	23.1/27.56
B + Recycled-AltUp	66.63	85.60	74.83	84.81/91.93	22.72/27.15
B + AltUp	66.96	85.32	75.80	85.24/92.36	24.35/28.78
L	69.13	87.23	81.21	86.77/93.56	26.15/30.76
L + Recycled-AltUp	69.30	87.91	82.53	87.37/93.88	27.51/32.38
L + AltUp	69.32	88.20	82.75	87.81 / 94.29	27.10/32.04
Table 8: The performance of baseline, Recycled-AltUp, and AltUp on pretrain and fine-tune evaluation metrics. Recycled-AltUp and AltUp were instantiated with  $K = 2$  for all evaluations.
and fine-tune metrics. As conjectured in Sec. 4 Recycled-AltUp generally does not provide the full benefits of AltUp in terms of pretrain and fine-tune accuracies, however, this gap seems to shrink for larger models. Moreover, Recycled-AltUp has the appeal that it practically adds no additional parameters to the model and, as a result, has roughly the same speed as the baseline model (see Fig. 5). Given the lightweight and latency-matching nature of Recycled-AltUp, these improvements directly translate to clear gains over the dense baselines. We envision that Recycled-AltUp's improved speed and reduced parameter count may make it more appealing for certain applications.