SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\det  (\mathbf {I} _ {d} - 4 \mathbf {A} ^ {*}) = \det  \left(\boldsymbol {\Pi} (\mathbf {A} ^ {* *}) (\mathbf {I} _ {d} - 4 \mathbf {A} ^ {* *}) \boldsymbol {\Pi} (\mathbf {A} ^ {* *}) ^ {\top}\right) = \det  \left(\mathbf {I} _ {d} - 4 \mathbf {A} ^ {* *}\right)
$$
and, similarly,  $\operatorname*{det}(\mathbf{I}_d - 8\mathbf{A}^*) = \operatorname*{det}(\mathbf{I}_d - 8\mathbf{A}^{**})$ . In the third transition, we use definition of  $\mathcal{G}(\cdot)$  (51). Note that (60) holds for all  $\mathbf{A} \in \mathbb{D}_d$  such that  $8\mathbf{A} \prec \mathbf{I}_d$  and also  $8\mathbf{A}^* \prec \mathbf{I}_d$  since  $8\mathbf{A}^{**} \prec \mathbf{I}_d$ . We conclude that, when  $\mathbf{B}, \mathbf{C}, D$  are chosen optimally with a given  $\mathbf{A}$ , the minimum of  $\overline{\mathcal{L}}(\theta_{\mathrm{SDE}}; \mathcal{X}, \mathcal{Y}, \mathcal{T}_{\mathrm{SDE}})$  is reached when  $\mathbf{A} = \mathbf{A}^*$ . As we have already deduced, diagonal entries of  $\mathbf{A} = \mathbf{A}^*$  are sorted in the non-descending order. Hence, using Lemma A.4's notation, diagonal entries of  $\mathbf{E}$  are already sorted in a non-ascending sorting order and  $\pi = (1, \ldots, d)$ ,  $\Pi = \mathbf{I}_d$  satisfy requirements of the Lemma. Hence, with  $\mathbf{A} = \mathbf{A}^*$ , the optimal  $\mathbf{B}$  has a form  $(\mathbf{I}_d - 4\mathbf{A})^{1/2}\mathbf{Q}^*$  where  $\mathbf{Q}^* = \mathbf{I}_d(\mathbf{Q}^{(3)})^\top = (\mathbf{Q}^{(3)})^\top$ . Optimal  $\mathbf{C}$  and  $D$  are further determined by (33). (11) follows from (60) and the fact that, as discussed above, we can replace  $\pi(\mathbf{A}^*)_l$  with  $l$  in (60),  $1 \leq l \leq d$ .
B Additional experimental details
B.1 Compute resources and implementation
We use NumPy [25] in Google Colaboratory the variance comparison and kernel classification experiment. For the Transformer setups, we use TPU cluster and JAX [7] library. All tested Transformer variants were trained and tested on a TPU pods containing 4 TPU v3 chips with JAX and on GPUs (V100).
B.2 Variance comparison
We repeat the setup of [31] closely: we draw 5 pairs of sets  $\{\mathbf{x}^{(i)}\}_{1\leq i\leq L}$ ,  $\{\mathbf{y}^{(j)}\}_{1\leq j\leq L}$ ,  $L = 1024$ . On each pair, we compute the relative variance for all pairs of points and for all indicated RF methods. Further, the shifted log-variance is optimized separately on each pair of sets for GERF, ADERF and SDERF.
We take  $M = 1$  since  $M$ 's value is not important in this experiment: bigger  $M$  would just shift the curves below. The reported curves are means over all pairs of points and over all 5 sets.
Table 2: Hyperparameters for the base models for pre-training for all methods
Parameter	Value
# of heads	12
# of hidden layers	12
Hidden layer size	768
# of tokens	512
Batch size	256
M	256
Pretrain Steps	1M
Loss	MLM
Activation layer	gelu
Dropout prob	0.1
Attention dropout prob	0.1
Optimizer	Adam
Learning rate	10-4
Compute resources	8 × 8 TPUv3
Table 3: Dataset used for pre training.
Dataset	# tokens	Avg. doc len.
Books [54]	1.0B	37K
Wikipedia	3.1B	592
B.3 Kernel classification
As in [31], we obtain training, validation and test splits by shuffling the raw dataset and taking  $90\%$ ,  $5\%$ ,  $5\%$  objects respectively. The splits are fixed for all RF methods. We tune  $\sigma$  on a logarithmic grid of 10 values on  $[10^{-2}, 10^{2}]$ . For each  $\sigma$  and each RF type, we try 50 seeds for drawing RFs during validation and testing. Testing is performed for the best  $\sigma$  only. Figure 3 reports averages over 50 seeds. We use orthogonal  $\omega$ 's for all types of RFs as described in [31], since orthogonal random features work better in practice [16, 31].
B.4 DERFs for long-sequence Transformers
B.4.1 Speech modelling
Our Conformer-Transducer variant was characterized by: 20 conformer layers, model_dim = 512, relative position embedding dimensionality rped = 512 and  $h = 8$  heads. We used batch size  $\mathrm{bs} = 2048$  and trained with the adam optimizer on TPUs. For the regular Conformer-Transducer training, we run ablation studies over different number of random features:  $m = 8, 32, 128$ . In the NST setting, we run experiments with  $m = 8$ . We reported commonly used metric: normalized word error rate (NWER).
B.4.2 Natural language processing
We pretrained BERT model on two publicly available datasets (see: Table 3). Following the original BERT training, we mask  $15\%$  of tokens in these two datasets, and train to predict the mask. All methods were warm started from exactly the same pre-trained checkpoint after 1M iteration of BERT
pretraining. We used the exact same hyperparameter-setup for all the baselines (FAVOR++[31], FAVOR+ [16], ELU [26], ReLU [16]) and FAVOR++. The hyperparameters for pretraining are shown in Table 2. We finetuned on GLUE task, warm-starting with the weights of the pretrained model. The setup is analogous to the one from the original BERT paper.