SlowGuess/NAACL_2022_OCR huggingface dataset:
Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3009-3019. Association for Computational Linguistics.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nystromformer: A nystrom-based algorithm for approximating self-attention. CoRR, abs/2102.03902.
Yun Yang, Mert Pilanci, Martin J Wainwright, et al. 2017. Randomized sketches for kernels: Fast and optimal nonparametric regression. The Annals of Statistics, 45(3):991-1023.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2020. Informer: Beyond efficient transformer for long sequence time-series forecasting. CoRR, abs/2012.07436.
A Further experiment Details
As it is not realistic to exhaustively fine-tune all models and search for the best performance under limited computation resources, we instead replace the self-attention module in transformer with the various drop-in attention methods and keep other experimental settings the same. Following (Xiong et al., 2021) we use a 2-layer transformer model with 64 embedding dimensions, 128 hidden dimensions, and 2 attention heads for all experiments. Mean pooling is used in all classifiers.
For comparable computation complexity, we control the number of features used in all methods, which leads to 256 as the number of features in Skeinformer, 256 as  $k$  in Linformer, 256 as the number of landmarks in Nyströmformer,  $(256 / \log n)$  as the factor in Informer, and 256 as the number of features in Performer. Additionally, the number of random blocks and block size in Big Bird are by default 3 and 64, under which setting Big Bird will visit  $640 \cdot n$  elements in the attention matrix while other models visit  $256 \cdot n$  elements. A clearer complexity evaluation on the FLOPs of each method is provided in Appendix.
We use Adam optimizer (Kingma and Ba, 2015) with a learning rate of  $1e - 4$ . Batch size is selected conditioned on the memory requirements of Skeinformer, which leads to 128 for Text Classification, 256 for ListOps, 64 for Document Retrieval, 512 for Pathfinder and 256 for Image. For methods reporting out-of-memory errors, we apply gradient accumulation and report the accumulated steps. Instead of setting a fixed epoch number, we train all models until convergence with a stopping strategy (if better performance is not observed for 10 checking steps on the validation set we will stop the training process).
We conduct all experiments on one Tesla V100 SXM2 16GB. For numerical consistency, all experiment results are averaged across three random runs.
A.2 LRA Dataset
We evaluate on five classification tasks in LRA benchmark (Tay et al., 2020a), excluding Pathfider-X, which fails all baseline models.
ListOps (Nangia and Bowman, 2018): This 10-label classification task requires the models to parse a sequence of length  $2k$  of numbers and operators
and evaluates their capacity of modeling hierarchically structured long sequences.
Text Classification on IMDb review dataset(Maas et al., 2011): This byte-Level binary classification task requires the model to analyze the sentiment of a sequence of length  $4k$  by composing the unsegmented characters into higher-level meaningful units.
Document Retrieval on AAN dataset (Radev et al., 2013): This byte-Level binary classification task requires the model to compress long sequences of length  $4k$  into representations for similarity score calculation in a two-tower setup without cross-attention.
Pathfinder on CIFAR-10 dataset(Linsley et al., 2018): This binary classification task requires the model to decide whether two points are connected by a dashed path on an image represented as a pixel sequence of length  $4k$ , and exams their capacity to capture long-range spatial dependency.
Image Classification (Krizhevsky et al., 2009): This 10-label classification task requires the models to learn the spatial relations between the flattened input pixels of length  $1k$ .
A.3 Validation Loss
We present the loss decay plots on all tasks in Figure 2. In the first subplot for the text classification task, we note all the methods quickly overfit the dataset. In all the other plots, our methods show the ability to both efficiently converge to the long-time limit and find better local minima with lower validation loss.
A.4 FLOPs
We conclude in this subsection the floating point operations (FLOPs) of each candidate model (excluding the ablation models). To ease the notation, given the sequence length  $n$ , we fix  $p = 32$ ,  $d = 256$ . Assuming the matrices  $Q, K, V$  are given and omitting the non-leading term, we report the FLOPs of each model in Table 3. We additionally comment that Reformer is excluded from the table since its FLOPs are not fixed and depend on the frequency of collision after hashing of tokens, which changes with the input sequence.
A.5 Hyper-parameter Sensitivity
Figure 3 shows the accuracy and training time for Skeinformer using different batch sizes (64,128) and learning rates  $(1e - 3,1e - 4,1e - 5)$  on text
Figure 2: Validation loss (Y axis) changes with regard to training time (second, X axis).
Models	FLOPs
Standard	2n2p
Big Bird	5ndp
Performer	3ndp
Nystromformer	4ndp
Linformer	4ndp
Informer	3ndp
Skeinformer	4ndp
Table 3: The leading terms of FLOPs in computing attention.
classification. The results are averaged across random trials. We observe that smaller learning rate offers slower convergence but to a better point.
Figure 3: Testing accuracy and training time (Y axis) changes with regard to learning rate (X axis).
Proof. For each column  $B^{(i)}$ , we first define a discrete random variable  $X_{i}$ , that with probability  $\frac{1}{n}$ ,  $X_{i} = b_{ji}, \forall j \in [n]$ , where  $b_{ji}$  is the  $j$ -th element in  $B^{(i)}$ . Since all the elements in  $B$  are bounded (within the range [0, 1]) due to the row normalization in softmax, we infer that for any  $i \in [n]$ ,  $X_{i}^{2} \in [0, 1]$  is a sub-Gaussian random vari-
able with parameter  $\sigma = \frac{1}{2}$  (Wainwright, 2019). Combine the conclusion with the assumption that  $\mathbb{E}X_i^2\leq C$  , we have
$$
\frac {X _ {i} ^ {2}}{\mathbb {E} X _ {i} ^ {2}} \sim \text {s u b - G a u s s i a n} \left(\sigma^ {2} = \frac {1}{4 C ^ {2}}\right). \tag {7}
$$
Then we uniformly sample  $d$  indexes  $\{j_k\}_{k=1}^d$ 's with replacement, and we estimate the squared norm of each column with the unbiased estimator  $Y_i = \frac{n}{d}\sum_{k=1}^{d}b_{jk}^2$ . We remark  $Y_i$  has the same distribution as  $\frac{n}{d}\sum_{k=1}^{d}X_{i,(k)}^2$ , where  $X_{i,(k)}$  are  $d$  i.i.d. copies of  $X_i$ . Therefore through a linear transform of Equation (7) we can derive that
$$
\frac {Y _ {i}}{n \mathbb {E} X _ {i} ^ {2}} \sim \text {s u b - G a u s s i a n} \left(\sigma^ {2} = \frac {1}{4 C ^ {2} d}\right). \tag {8}
$$
Notice different  $Y_{i}$ 's may not be independent since they all rely on the same  $d$  rows in  $\pmb{B}$ . However, we can still apply the maximal sub-Gaussian inequality (Boucheron et al., 2013) to have:
$$
\mathbb {P} \left\{\max  _ {i \in [ n ]} \left| \frac {Y _ {i}}{n \mathbb {E} X _ {i} ^ {2}} - 1 \right| > \frac {1}{2} \right\} \leq 2 n e ^ {- \frac {C ^ {2} d}{2}}. \tag {9}
$$
If the high probability bound holds that  $\max_{i\in [n]}\left|\frac{Y_i}{n\mathbb{E}X_i^2} -1\right|\leq \frac{1}{2}$ , we directly have that our estimators  $Y_{i}\in [\frac{1}{2}\| B^{(i)}\|^{2},\frac{3}{2}\| B^{(i)}\|^{2}],\forall i\in [n]$ . In that case, the estimated sub-sampling probabilities satisfy that
$$
\begin{array}{l} \hat {p} _ {i} = \frac {Y _ {i} ^ {\frac {1}{2}} \| V _ {(i)} \|}{\sum_ {i ^ {\prime} = 1} ^ {n} Y _ {i ^ {\prime}} ^ {\frac {1}{2}} \| V _ {(i ^ {\prime})} \|} \geq \frac {\sqrt {1 / 2} \| \boldsymbol {B} ^ {(i)} \| \| V _ {(i)} \|}{\sqrt {3 / 2} \| \boldsymbol {B} ^ {(i)} \| \| V _ {(i ^ {\prime})} \|} \\ = \sqrt {\frac {1}{3}} p _ {i}, \forall i \in [ n ], \\ \end{array}
$$
where  $p_i$ 's are the optimal probabilities defined in the main paper.
In that case, to prove the lemma it suffices to pick a big enough sub-sample size  $d$  such that the right-hand side of Inequality (9) is smaller than  $\delta$ . Simply solving the inequality leads to the desired result  $d \geq \frac{2}{C^2} \log \left( \frac{2n}{\delta} \right)$ .