Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
Authors: Ido Amos
Jonathan Berant
Ankit Gupta
Venue: ICLR 2024 oral (2024)
Keywords: Pre Training
Transformers
State Space Models
Long Range Models
Fair Evaluation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Training a model directly on a dataset from sctrach can lead to grossly under-estimated performance. For proper evaluation, one must first pretrain on the dataset and then finetune.

Abstract: Modeling long-range dependencies across sequences is a longstanding goal in machine learning and has led to architectures, such as state space models, that dramatically outperform Transformers on long sequences. However, these impressive empirical gains have been by and large demonstrated on benchmarks (e.g. Long Range Arena), where models are randomly initialized and trained to predict a target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures and that pretraining with standard denoising objectives, *using only the downstream task data*, leads to dramatic gains across multiple architectures and to very small gaps between Transformers and state space models (SSMs). In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points. Subsequently, we analyze the utility of previously-proposed structured parameterizations for SSMs and show they become mostly redundant in the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.

Published as a conference paper at ICLR 2024
Never Train from Scratch: FAIR COMPARISON OF LONG- SEQUENCE MODELS REQUIRES DATA-DRIVEN PRIORS
Ido Amos Tel Aviv University*
Jonathan Berant Tel Aviv University
Ankit Gupta IBM Research
ABSTRACT
Modeling long-range dependencies across sequences is longstanding goal in machine learning and has led architectures, such as state space models, that dra- matically outperform Transformers on long sequences. However, these impressive empirical gains have been by and large demonstrated on benchmarks (e-g. Long Range Arena), where models are randomly initialized and trained to predict target label from an input sequence. In this work, we show that random initialization leads to gross overestimation of the differences between architectures and that pretraining with standard denoising objectives, using only the downstream task data, leads to dramatic gains across multiple architectures and very small gaps between Transformers and state space models (SSMs). In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained, and we improve the best reported results of SSMs on the PathX-256 task by 20 absolute points. Subsequently, we analyze the utility of sevioussy-proposeee structured parameterizations for SSMs and show they become mostly redundant the presence of data-driven initialization obtained through pretraining. Our work shows that, when evaluating different architectures on supervised tasks, incorporation of data-driven priors via pretraining is essential for reliable performance estimation, and can be done efficiently.
INTRODUCTION
suff-superviee pretraining is now widespread across most areas of machine learning, including NLP, speech, and vision (Touvron et al.) 2023 Baevski et al., 2020 Reed et al.) 2022). Given downstream task, it is standard to finetune a pretrained model rather than train "from scratch"", to achieve better performance (Raffel al., 2019). Conversely, when developing new architectures with better inductive biases for particular skills, for example, for capturing long-range dependencies or for better algorithmic reasoning, it is still common train on the task data from scratch with random initialization (Tay et al. 2020a| Delétang et al. 2022, Velivckovi's c et al., 2022: Dwivedi al., (002)). This difference in practice stems not only from the computational overhead required for pretraining massive datasets, but also to decouple the effects of the pretraining data and allow an apples-to-apples comparison, which would otherwise require a "standard" pretraining corpus for each scenario.
prime example of the latter scenario estimating capabilities modeling long range dependencies in sequences, setting where Transformers have reported inadequate performance on benchmarks designed as stress tests, such as Long Range Arena (LRA) (Tay et al.] 2020a). This inefficacy of Transformers has led line of new architectures, suggesting changes to RNNs, CNNs and Transformers themselves, biasing them towards capturing long range dependencies, and achieving impressive performance on LRA, when trained from scratch (Gu et al.) [2022a] Gupta al. [022a Li ct al. 2022] Ma et al. 2022). However, these results do not align with performance of pretrained Transformers ("foundation models"), that have displayed remarkable performance on tasks involving modeling long range dependencies, such as text summarization, code completion and protein folding. (Touvron et al. 2023 Jumper et al. (202T). Despite the significant progress long sequence modeling. the reasons for sub-par performance of Transformers on long sequence benchmarks, such as LRA,
( /iddmmms@mail. www.mmii............m. ii,, jaberanz&es. sas. ic.1,, anpit . sitkanpur@gmail. com).

---

Published as a conference paper at ICLR 2024
S4
Self Pretrained
100
Transformer
Trained from Scratch
90
86.21
86,09 86.75
87.11
80
70
67.82
65
60
55
53,60
45
Mean performance LRA
PathX-256
Figure 1: Evaluation of Transformers and S4 on Long Range Arena when trained from scratch vs. when self pretrained.
remains unexplored, while methods achieving competitive performance resort to tailored changes to the architecture (Ma al., 2022 Zuo al., 2022).
In this work, we shed light on this discrepancy, showing it stems from inadequate training and evaluation practices, and suggest a simple and efficient solution allowing a proper evaluation. While avoiding pretraining on a large corpus is understandable, training from random initialization with downstream supervision alone disregards the role of the pretraining objective itself, leading to different inductive bias than that of pretrained model. In recent line of work, (El-Nouby al. (202T): He et al. (2022); Krishna al. (2023) have demonstrated that, when using denoising objectives, pretraining solely downstream training data (denoted as self pretraining) often leads to gains comparable to the ones from pretraining on large corpora, showing effectiveness on tasks such image classification, segmentation, text classification, etc. This suggests that, rather then training from scratch, more realistic estimate of model performance can be obtained via self pretraining (SPT), with SPT acting as data-driven initialization method, while allowing fair comparison between methods as only the task data is used.
To demonstrate the importance of the suggested method, we empirically show that priors learned through SPT with denoising objectives are highly effective for learning long range dependencies several architectures, eliminating the need for complex hand-crafted modeling biases used
in current solutions (Gu et al., 2022a; Ma et al., 2022 Li et al., 2022] Orvieto et al. 2023). We primarily study Long Range Arena (LRA), a standard benchmark for long sequence modeling and evaluate multiple SPT models. We show that SPT improves the mean absolute performance of vanilla Transformers by more than 30%, for the first time allowing them to match the state-of-the-ar performance on LRA without any architectural changes (Figure I. This is in stark contrast to prior works where Transformers report significantly lower performance compared to the state-of-the-art.
We study the effectiveness of SPT for State Space models (SSMs), novel line of architectures using modified linear RNNs as a replacement for attention layers in Transformers. Incorporating specialized parameterization and initialization of linear RNNs, SSMs such as S4 successfully mitigate the vanishing/cxploding gradient issues and reach impressive performance on long sequence tasks, such LRA (Gu et al. 2022a). We find SPT to also benefit S4 with performance gains in 5 out of 6 LRA tasks. Moreover, with SPT, S S4 solves the challenging PathX-256 task, achieving 20% accuracy improvement compared to training from scratch (Figure Ip. Building these improvements, study the utility of hand-crafted modeling biases in S4 over simpler linear RNNs, finding that the data-driven priors learned via SPT render most of them redundant (Gupta et 2022b). In doing so, we are the first to provide competitive performance with diagonal linear RNNs without any manual modifications (Orvieto al. 2023).
Our findings show that priors beneficial for capturing distant dependencies can be simply learned from the task data via standard denoising objectives without any intrusive changes the model. We examine the benefits of SPT across multiple data scales showing them to become even more pronounced as data becomes relatively searce. Last, for SSMs, we analyze the convolution kernels
2

---

Published as a conference paper at ICLR 2024
learned via SPT shed light the learned priors for capturing long-range dependencies. We demonstrate an interesting phenomena in which, depending on the modality, rapidly decaying kernels can lead improved performance over the slowly decaying ones used in the native S4 model, further highlighting the utility of learning priors from the data itself (Gu et al. 2020).
Our main contributions be summarized follows:
(i) We show that the reported performance various architectures on long range benchmarks is grossly underestimated, and suggest inexpensive data-drived approach to enable accurate evaluation without requiring any additional data.
(ii) We report large empirical gains over the previously-reporte performances on LRA across range of architectures and, in particular, improve upon the best reported accuracy the challenging PathX-256 task by 20 absolute points (67 - 87).
(iii) demonstrate how manually-desigue biases become increasingly redundant with pretraining and that, with modern training and evaluation practices, simpler models can often match the performance sophisticated architectures. We are the first to provide competitive performance on LRA with Transformers and diagonal linear RNNs.
The multi-modal and challenging setup of LRA, along with the scale of improvements due to SPT, advocate the inclusion of pretraining stage while evaluating models in general, for example when designing architectures for multidimensional inputs (Nguyen et al. 2022). algorithmic reasoning
(Diao & Loynd| 2023) graphs (Shirzad et al. 2023)
Our code & data are available at wtttt:t///// //////gittt Sighh...... cww/Tdoonos/nott.t.ooooooooooooe (rooammoo/not--rrmmsssraooooo
EXPERIMENTAL SETUP
Our experiments center around the evaluation of Transformers and SSMs on the Long Range Arena (LRA) benchmark which was proposed for examining the ability of sequence models to capture long-range dependencies (Tay et al. (020a)). contains 6 main sequence classification tasks, each being either binary or 10-way sequence classification.
1. ListOps: Each sequence in the dataset is nested list, with each sublist describing operation (c.g. MAX, MEAN) to be applied on a set of tokens (Nangia & Bowman, 2018). Evaluation of nested lists used single token in their enclosing list thus requiring the understanding of hierarchical structure, the task is 10-way classification with sequence length of 2K.
INPUT: [MAX 4 [MIN 2 3]1 O [MEDIAN 1 8 9 21] OUTPUT:
2. Text: character-level version of the IMDb reviews dataset (Maas et al., 2011 for sentiment
classification, the task is binary classification with sequence length of up to 2048
Retrieval: character-level version of the AAN dataset (Radev et al. 2013) for predicting similarity scores of two documents. The task is binary classification with sequence length of up to 4K, requiring to process 8K tokens for evaluation.
Image: grayscale CIFAR10 images are flattened as 1D sequences and any explicit 2D inductive bias cannot be used. The task is 10-way classification, with sequence length 1024.
5. Pathfinder, PathX: synthetic 2D visual tasks treated as 1D sequences (similar to Image) for testing tracing capabilities (Linsley et al. 2018 Kim et al. 2020). PathX and Pathfinder are similar tasks that differ in sequence length (1024 vs 16384) and are binary classification.
Apart from the aforementioned tasks, we examine an additional variant of PathX called PathX-256 with sequence length 256² 65536 and we the first to report strong results this task. Besides LRA, we experiment with additional datasets that will be described later in Section 3.7
Self Pretraining (SPT) We perform SPT with a causal/autoregressive sequence modeling objective for unidirectional models, and a masked sequence modeling objective for bidirectional models, using only the downstream task training set. For the visual tasks (Image, Pathfinder, Path-X -) the masking ratio for masked sequence modeling is set to 50% following He al. (2022), to 15% for language
3

---

Published as a conference paper at ICLR 2024
Table 1: Long Range Arena. (top) performance of models trained from scratch reported in Tay et al. (2020a), (bottom) performance of self pretrained (SPT) Transformers of sizes comparable to the ones on top. denotes chance accuracy.
Approach
Listops
Text
Retrieval
Image
Pathfinder
PathX
Avg.
Transformer
36.37
64.27
57.46
42.44
71.40
X
53.66
Local Attention
15.82
52.98
53.39
41.46
66.63
X
46.71
Longformer
35.63
62.85
56.89
42.22
69.71
x 52.88
Linformer
35.70
53.94
52.27
38.56
76.34
X 51.14
Reformer
37.27
56.10
53.40
38.07
68.50
X 50.56
BigBird
36.05
64.02
59.29
40.83
74.87
X 54.17
Linear Trans.
16.13
65.90
53.09
42.34
75.30
X 50.46
Performer
18.01
65.40
53.82
42.77
77.05
X
51.18
Transformers Masked SPT
59.75
89.27
88.64
74.22
88.45
87.73
81.34
Transformers Causal SPT
59.15
88.81
90.38
76.00
88.49
88.05
81.81
tasks (Text, Retrieval) following Liu al. (2019), and to 10% for ListOps. For Transformers, full attention as default with the hardware optimized FLASH implementation (Dao et al.] (022). Due to computational constraints, for tasks with sequence length least 16K we split the input to the attention layer into non-overlapping blocks of size 4096 and allow each block to attend itself and its neighbour(s).
bardware-optimizze
Our codebase built on the original S4 repository!! For additional experimental details, such as computational resources for SPT and finetuning, please refer Appendix C.I
RESULTS
In Section 3.1 perform SPT for LRA tasks using the official model configurations. In Section
3.2
we perform SPT for Transformers and S4. Section 3.3 evaluates the role of design choices in SSM in the context of SPT. Section 3.4toonn3344eama examines the utility of SPT across data scales and Section
3.5
examines the utility of PT on large text corpus. Section provides an analysis of pretrained
SSM kernels and how they relate current initialization schemes. Section 3.7 contains additional experiments distinct modalities.
3.1
UNDERESTIMATION OF LONG-RANGE ABILITIES OF TRANSFORMERS
We start by investigating the reliability the historically-reportee model performances on LRA, in the more modern setting of pretraining. Concretely, repeat the Transformer experiments performed by Tay et al. (2020a), except that we first pretrain the model on the task data and then finetune To allow fair comparison with the original results, strictly follow the model configurations used by Tay et al. 20200a). We experiment with two pretraining objectives: (1) next token prediction for unidirectional models (2) masked token prediction for bidirectional models, varying the masking ratio as detailed in Section
As summarized in Table [ we find that both pretraining objectives lead to dramatic performance gains for Transformers compared to the conventional practice of training with random initialization, with the average test accuracy increasing by roughly 30%. Both causal and masked pretraining yield similar results even in cases where there are no clear benefits to using causal model, such as on the visual tasks. Furthermore, even for LISTOPS large performance gains are observed even though, in the original data, the arguments to the list operations are sampled randomly, meaning that inferring missing tokens from context rarely possible.
As the experiments are performed with no architectural changes or additional data, the difference in performances can be attributed the priors learned during SPT, clearly demonstrating its importance for reliable performance evaluation.
4

---

Published as a conference paper at ICLR 2024
Table 2:
Long Range Arena. Self pretrained (SPT) Transformers and S4 compared to existing trained from scratch models. Average performance ("Avg.") ") is reported without PathX-256 to align with prior work. Results for MEGA, SPADE & S4 are taken from original papers with exceptions denoted by f. x denotes computationally infeasible, O denotes unreported results.
Approach
Listops
Text
Retrieval
Image
Pathfinder PathX PathX-256
Avg.
Transformers Rotary
47.90
79.08
82.31
75.04
76.64
84.72
74.28
Transformers Rotary Masked SPT
61.49
91.02
91.57
86.04
94.16
92.98
86.21
S4 Gu ct al. 2022a
59.60
86.82
90,90
88.65
94.20
96.35
67.821
86.09
S4 + Masked SPT
61.25
90.34
88.74
89.36
94.92
96.94
87.11
86.75
SPADE Zuo al.
2022
60.50
90.69
91.17
88.22
96,23
97.60
Q
87.40
MEGA Ma et al.
2022
63.14
90.43
91.25
90.44
96.01
97.98
88,21
Pythia 70M (Rand Init)
41.20
69.29
76.45
52.55
74.31
X
62.76
Pythia 70M
43.05
83.41
84,29
67.41
80.05
X
68.04
3.2 COMPARING S4 AND TRANSFORMERS
In the above set-up we strictly adhered to the model sizes used by Tay et al. (2020a) and consequently the absolute performances are still low compared to the current state-of-the- section, we scale the model sizes and evaluate the utility of SPT for the best performing architectures including S4 (Gu et al.) 2022a). For Transformers,
on LRA. In this
replace the positional embeddings with the more commonly used rotary embeddings (Su et al. 2021) and only train bidirectional models in line with prior works reporting high performance.
As summarized in Table 2 SPT leads to dramatic performance gains for Transformers with per- formance gains ranging from 8 15% across tasks, even surpassing the average performance of well-tuned S4 (86.2 vs 86.1). 1). SPT Transformers surpass the performance of both trained from scratch and SPT versions of S4 on out of tasks. The results Table 2 defy current understanding, with prior works citing the sub-par LRA performance of Transformers as a prime motivating factor for new methods. Yet we show that, while architectural developments indeed lead remarkable performance gains, most of the priors essential to high performance can already be learned from data directly.
In case of S4, while SPT leads to modest gains on most tasks, a substantial gain of 20% is observed on the challenging PathX-256 task with input length of 65K, signiiiantlly improving over the best reported performance of 63.1% by (Dao et al., 2022) who, in addition, used extra data from the Pathfinder-64 task.
The additionally reported models, SPADE and MEGA, are Transformer variants that augment the model with single or several state space layers. SPADE combines the outputs of frozen S4 layer and local attention in the first block, while MEGA incorporates a learned exponential moving average, an instance of diagonal SSMs, into gated attention blocks. To the best of our knowledge, we are the first to show that purely attention-basse methods, without any architectural modifications, can achieve competitive results LRA. While incorporating SSMs can be important in terms of scalability to longer sequences due to their log-linear complexity with respect to input length, we show that in terms of model performance, pretraining leads to biases that are effective as manual designs.
An important aspect of SPT is the use of additional compute compared to the trained from scratch baseline and it is natural investigate if similar gains can be obtained by training from scratch for longer. For all our trained from scratch baselines, we ensured that the validation performance had converged and did not improve for several consecutive epochs. We examine the aspect of the computational overhead of SPT in detail Appendix D where we show that SPT leads to significant gains, even in the setting where the same amount of compute is used for SPT models and the ones that are trained from scratch.
3.3 THE ROLE OF EXPLICIT PRIORS
We have established that SPT allows for a more reliable evaluation of the actual capabilities of architectures and further improves the performance of SSMs such S4. Despite its high performance. S4 has a complex design guided by principled theoretical considerations to enable long range signal
5

---

Published as a conference paper at ICLR 2024
100
Self Pretrained
DLR
ZZZ
Trained from Scratch
90
85
82.75
83.37
82.85
84.47
80.47
Avg. Accuracy
75
80
74.33
77.33
70
65
60
50
45
40
Random Init
Random Init
Structured Inil
Structured Init
Figure 2: Average performance of models when trained from scratch or self pretrained, for different sets of initializations prior to pretraining. See Table 7 for per-task results.
propagation, which can explain the small advantage maintained over SPT Transformers, lacking such an inductive bias. In a series of works, various simplifications to S4 have been proposed while maintaining performance. We will now show that SPT allows for even simpler model (viz. diagonal linear RNN) to match the performance of S4.
We first provide a brief overview of SSMs below and refer to Gu et al. (2022a) for a detailed description. Given an input scalar sequence u. SSMs follow linear recurrence generating a hidden state vector x at timestep n, and produce scalar output sequence y as
n A Bun
A NXN, B C
Yn = Cτ
CE CIXN
By unrolling the recurrence across the timesteps, can be shown that y can be equivalently computed by convolving u with the kernel defined by K₁ = C* A B. Instead of directly using A, B, as learnable parameters, S4 uses an alternate parameterization inspired by theory in continuous time, motivating the transformations:
A A PQ*
(2.1)
A [I - A/2 A) (I (I+ A/2 A)
(2.2)
B = (I A/2 - C=C
(2.3)
K = 0² A
(2.4)
where A, A,P,Q, C, are learnable parameters and A E Diag(C P, E Nx1 In addition to this parameterization, S4 uses principled initialization method aimed towards slow decay of the kernel (w.r.t. timestep k) in order to facilitate capturing long-range dependencies.
Inspired by the success of S4, Gupta et al. (002b) proposed simplification S4 called Diagonal Linear RNN (DLR) defined as
V, = Aē + 1 Un A diag(C
Yn C#
Ce CIXN
where is the all-ones vector. DLR is significantly simpler to compute compared to S4 and the authors reported it to be as performant as state-of-the-art SSMs on a wide variety of token-level tasks. Hence, it is natural to investigate the conditions under which S4 with its more complex design (eq. 2 can be replaced by the simpler DLR. To that end, we evaluate the performance of DLR and S4 on ListOps, Text, Image and PathX tasks they are the hardest and represent all modalities in LRA. For each model, we experiment with two sets of initializations: (1) random initialization where the state space parameters initialized from normal distribution with small standard deviation, and
WWhen the input is sequence vectors, the model is applied each channel separately commonly followed by a FFN to exchange information across channels.

---

Published as a conference paper at ICLR 2024
100
Imsge
Seif Prefrained
image
80
Text
Trained from Scratch
Text
80
70
Relative gain
Actunay
10"
10
10"
10"
Ratio traihing data (log)
Raioo of training data (log)
Trained from scratch and self pretrained (SPT) versions of S4 evaluated on multiple data scales for Image and Text tasks from LRA, originally containing 45K and 25K samples respectively, (left) absolute performances and (right) relative gains due SPT over training from scratch.
Figure 3:
(2) "structured" initialization recommended by the respective authors aimed at capturing long-range dependencies.
The results are summarized in Figure 2 and per-task results provided in Table 7 We find that, when trained from scratch, with both random and structured initializations, DLR lags behind S4 in terms of average performance (77 vs 83) demonstrating that biases incorporated through the specific initialization and parameterization used in S4 are indeed critical to performance. However, the picture radically changes under SPT with SPT, DLR outperforms trained from scratch S4 (83.4 82.8) and is only slightly behind SPT S4 (83.4 vs 84.5). This suggests that the data-driven priors learned through pretraining are almost as effective as the manual biases incorporated in S4.
Results in this section have two additional implications. First, this is the first instance in which vanilla diagonal linear RNNs have been shown to achieve competitive performance on LRA. Prior work by Orvieto et al.) (2023) suggested an additional normalization step in the kernel generation on top of made initialization achieve high performance on LRA. Second, while our discussion revolved around SSMs, many subsequent works designing global convolutions followed similar principles. For example, Li et al. (2022) proposed to generate a decaying convolution kernel from shorter kernels via interpolation, which induces smoothness and can be viewed normalization step. Similarly, Fu et al. (2023) applied a global convolution layer that is transformed by a deterministic function to explicitly induce a smoother kernel. Yet results suggest that these explicit steps are less significant when models are self pretrained.
3.4 SELF PRETRAINING IS EFFECTIVE ACROSS DATA SCALES
As the priors learned via SPT are data-driven, their efficacy dependent on the training set itself, which leads us to examine the performance gains as a function of the dataset size. To this end, given a downstream task, we randomly sample subset of the training set, and study the performance gains for S4 due to SPT under varying sizes of the subset. We restrict the pretraining phase of S4 to fixed number of update steps across all experiments and finetune until convergence.
As summarized in Figure 3 we uncover an interesting phenomenon; while the relative gains from SPT over the trained from scratch baseline S4 are modest when the full task data is available, they become increasingly significant (and as large 30%) on smaller data scales. This shows that priors from pretraining are especially effective when training data is scarce and, in the context of previous sections, implies that the incorporation of the pretraining stage is important for model evaluation regardless dataset size. In Appendix [.] we provide a complementary study on the effectiveness of SPT across model sizes, demonstrating that indeed SPT is effective across multiple model scales for both S4 and Transformers.
3.5 PRETRAINING ON TEXT CORPORA
Given the widespread success of pretrained language models and the large gains due to SPT on the LRA tasks (Table 2 it is natural to ask if similar gains could be achieved by finetuning a language
7

---

800 Published as a conference paper at ICLR 2024
0.10
0.10
Bascline
PT Layer
0.08
0.08
Layer
PT Layer
max
0.06
0.06
0.04
0.04
0.00
200
500
700
900
1000
0.00
0
128
Shift
(a) Text
(b) PathX
0.10
Beseline
PT Layer
0.08
0.08
PayLr
Layer
PT Layer
maxiwo
0.06
0.04
0.04
0.02
0.02
0.00
0.00
100
200 300
400
500 600 Shift
700
800
800
1000
100
300
600 Shift
1000
(c) Image
(d) ListOps
Figure 4: Maximal absolute values of kernels across channels in S4 learned via self pretraining (PT) compared against the standard HiPPO kernels (Baseline). Only odd layers are shown for better visualization.
model pretrained on large text corpus. To answer this, we consider Pythia 70M (Biderman et al. 2023), an autoregressive Transformer pretrained on the Pile (Gao et al.) (2020) as well as randomly initialized version with the same architecture, denoted as "Pythia 70M (RandInit)" in Table To be comparable to existing results and due to the formal requirements of the LRA benchmark, we use Sharacter/pixcl-leved tokenization instead of the original BPE tokenizer and the model is required to adapt to the new tokenization during finetuning.
As shown in Table 2 Pythia 70M generally lags behind our trained from scratch Transformer baseline due to the changed tokenization and difference between the pretraining distribution and the downstream tasks. This further highlights the importance of SPT as it allows the model to specifically learn and adapt to the structure and modality of the given task data. However, the performance of Pythia 70M is significantly better that its randomly initialized version Pythia 70M (Rand init) suggesting that pretraining on large text corpora can be beneficial across modalities.
3.6 THEORETICALLY
VS DATA-DRIVEN KERNELS
Many high performing models such SSMs incorporate manually-crafe priors bias the model towards learning long range dependencies. For example, the initializations used in SSMs such as S4, DSS, S4D, S5 are based on HiPPO theory (Gu et al. 2020). which explicitly determines the decay rate of the convolution kernels over time and provides strong dependence between distant elements in the input sequence. In similar spirit, Li al. (2022) generate convolution kernels modified with fixed weights aimed towards slow decay. On the other hand, kernels learned via SPT have guarantees of slow decay and depend solely on the input distribution and the pretraining objective.
In this section, we analyze the structure of the convolutional kernels learned via SPT and compare them to the HiPPO-based kernels used to initialize existing SSMs such as S4. The convolution operation in S4 has the form
Yo.k Σ A.
E Kaj Teck-1
where e is the channel and is the timestep. Based on this structure, we can estimate the degree of dependence between sequence elements channel I positions apart as [Ka,]-- For easier
8

---

Published as a conference paper at ICLR 2024
Table 3: Additional Experiments. Performance on Speech Commands (SC), S SCIFAR (accuracy) and BIDMC (R2) tasks. Results for trained from scratch S4 taken from Gu et al. (2022a). except for BIDMC (denoted by t) that are reproduced for the more interpretable R2 score.
Approach
SC
SCIFAR
BIDMC
Causal
Bi.
HR
RR
SpO2
S4 Transformers
93.60
96.08
91.13
0.999
0.994
0.9991
84.55
86.93
79.41
0.998
0.981
0.998
$4 SPT
95.09
96.52
91.67
0,999
0.990
0.997
Transformers + SPT
86.13
91.49
90.29
0.992
0.956
0.993
interpretation, we take the maximal absolute value over the channels as Kmax./ = max, [Ke.). For shift 1, Kmax,i bounds the norm of the derivative of ye,k w.r. ke for all positions & and channels
We generate kernels for the pretrained S4 models from Section 3.2 (before finetuning) and compare with the ones used in standard S4. Figure plots Kiax for the Image, Text, PathX and ListOps, all entailing better performance with the pretrained model (Table 2). We observe that the learned kernels exhibit variable decay rates across the tasks and model layers, in contrast to the fixed decay rate of the Bata-agnostic HiPPO kernels. particular, on the Text task the learned kernels are more local compared to HiPPO. For PathX, the vertical grid lines are aligned with the image resolution (128 x 128) showing high correlation between the underlying 2D structure of the data and the kernel peaks. Overall, Figure 4 further highlights the utility of SPT over data-agnostic initializations that cannot adapt to local global structure in task distribution.
3.7 ADDITIONAL EXPERIMENTS
In addition LRA, also tested the utility of SPT on additional datasets, encompassing additional natural data modalities, described as follows:
Speech Commands (SC) Raw speech waveforms of length 16K used in 35-way classification task (Warden] 2018). We test both causal and bidirectional models following Gu et al. 2022b).
SCIFAR Sequential CIFAR-10 10 dataset using RGB channels as features. This is similar to the Image task from LRA that uses grayscale images, except that here richer features are used.
BIDMC A suite of regression tasks, requiring predict respiratory rate (RR), heart rate (HR) and blood oxygen saturation (SpO2) from EKG and PPG signals of length 4K each.
The results shown in Table 3, with additional details in Appendix C.3 further strengthen the claims made throughout this work. On both SC and sCIFAR tasks, SPT leads large performance gains for Transformers and modest gains for $4. The gaps between trained from scratch Transformer and S4 are substantially narrowed with SPT. On the SC task, SPT leads Transformers and we observe the performance gap between causal and bidirectional variants of S4 to be mitigated with SPT. similar, but not identical, observation is made in section 3.1 where masked and causal SPT lead to very similar results on all LRA tasks.
large 5% improvement for
On the sCIFAR task. SPT leads to a dramatic 11% improvement for Transformers, nearly matching the performance of S4 (90.3 vs 91.7) and again pointing towards sub-optimal evaluation when only training from scratch. On BIDMC, the performances of both Transformer and S4 baselines are already close to perfect and it is hard to observe any meaningful improvements due to SPT.
In general, our results suggest that similar under-estimation of model performances might also be prevalent in other scenarios where training from scratch is standard (Delétang et al., 2022 Velivekovi's al. 2022 Dwivedi et al. 2022).
since the model is bidirectional there are two sets of kernels, left to right, and right to left. We take the maximum over both.
9

---

Published as a conference paper at ICLR 2024
4 ACKNOWLEDGMENTS
We thank Amir Globerson for insightful discussions and his support throughout the course this work. This research was partially supported by The Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).
REFERENCES
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with Noint-embedding predictive architecture. 2023 IEEE/CVF Conference on Computer Vision and Puttern Recog- doi: 10. 121100/cp5229992223.0000
nition (CVPR), pp. 15619-15629, IEEE, 6 2023.
URL
https://arxii http:////aaaaaooor///22//0000000 220.//22211.233
Alexei Baevski, Henry Zhou, Abdel-rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Ad- vances Neural Information Processing Systems 33: Annual Conference on Neural In- formation Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, volume neurips. cc/paper/2020/ abs/2006. 1 11477, 2020. URL (11777700666777700007000 hat.abttttttttttmlll...... html Bash/92dleleb. wwwpedddffaaacoccoooo http://22..eeea.cccfffaa12212222
Stella Biderman. Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume abs/2304.01373, pp. -997-2430 PMLR, 4 2023. URL https: httt:s::/prooceeiige
ml
html
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficiem exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Or- leans, LA, USA, November 28 December 9, 2022, volume abs/2205. 2022. doi: 10.48550/ arxiv.2205.14133
URL http: / papers http::/s...pp...cpppppeeepeepeee
httpppppeee..aeeeeeeeeee//////2 html
Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A. Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, volume wbtp/2200000099... OpenReview.net, 2022. doi: 10.48550//rxiv.22077 20200000....200000000.......... URL http:: http://////pppeeeeeeeeeeeeeeeeee http://www...aaaaaao/////////aaa
Cameron Diao and Ricky Loynd. Relational attention: Generalizing transformers for graph-structured tasks. In The Eleventh International Conference on Learning Representations, 2023. URL
https: http/ppeeviiiieee wwt.:iewwwweeeeeeee wttp://oommiiioeeme
wotpereeeeee..ene/ooroouli-----e cF uMmbWiN6
Vijay Prakash Dwivedi, Ladislav Rampášek, Mikhail Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, volume abs/2206.088 164, 6 2022. doi: 10. 00.48850/arxiv.226..08 061 URL http: httt::/ppapers..rt ni nips. httppppperrfllsppppeeeeeeeee
2022 and and__Beedeeeee www.BBnnhhhak....
Alaacldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv.org, abs/21 12.1040, 12 2021. ISSN 2331-8422. URL http:p::///axvv..com
10

---

Published as a conference paper at ICLR 2024
Daniel Y. Fu, Elliot L. Epstein, Eric Nguyen, Armin W. Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. Simple hardware-efficient long convolutions for sequence modeling. In Andreas Krause 0001, Emma Brunskill, KyungHyun Cho, Barbara Engelhardt. Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23- 29 July 2023, Honolulu, Hawaii, USA, volume abs/2302.06646, pp. 10373-10391. PMLR, 2023. doi: 00448550/axxiv22322.6666 URL
mlr. httpseeeddnnaammm...pessss////
Fu2 a..2aa.bm html
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. volume abs/2101.00027, 12 2020. URL
https://arxil ortg 00027
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Re. Hippo: Recur- rent memory with optimal polynomial projections. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Informa- tion Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, volume abs/2008.0766 8 2020. URL haaaa///220666002222211101112260 wbw.r/9tteebaatratt.........
htwp://ooocccii.........oo.oooo. cC cc/paper/2020/
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 2022. OpenReview.net, 2022a. URL https:: wwwp//weneeeeeeeeeeeeeeee. / net/ Forum?id=uYLFoz1Ve
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, volume abs/2206. 1893, 2022b. doi: 10.48550/arxiv. 2206. 1 1893. URL http:: http://ppprrrreee mipppprrr..ppppp....o..pp cc hwtp:cppppppeeeeeeeeee filss httpp/pppp//2/hhaaaa/p
/ (00.2ffae2770006660044111111
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as struc- tured state spaces. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Confer- ence on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, volume abs/2203. 3 2022a. doi: 10.48550/arxiv. URL
httppppesssssssccppppeeeeeeeeees wttp:lleep.ppppep./2/e///// //ash/ Bcw.ccc995553 htt.a//ccw.....ooooo.........oa.
html
Ankit Gupta, Harsh Mehta, and Jonathan Berant. Simplifying and understanding state space models with diagonal linear rnns, arXiv.org abs/2222070 12 2022b. ISSN 2331-8422, doi: 10.48550/ arxiv. mixiv.2212.00761 URLpttpss//ooo.oorr//1....
2ww.222222.. 00768
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988 IEEE, 2022. doi: 10.1109/cvp:52688.222 01553.
06377
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. doi: 1086008844415---0---03333 https: www. are com htt///tiiccee 51686 021 03800-881100 pdf
Nature, 596(7873):583 589, 8 2021. ISSN 0028-0836.
URL

---

Published as a conference paper at ICLR 2024
Junkyung Kim, Drew Linsley, Kalpit Thakkar, and Thomas Serre. Disentangling neural mechanisms for perceptual grouping. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020. OpenReview.net, 2020. URL https:// openreview weeenrevieenee/fforrreee /ww//founm (tt.rummmmHH......4..F..
Kundan Krishna, Saurabh Garg, Jeffrey Bigham, and Zachary Lipton. Downstream datasets make surprisingly good pretraining corpora. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume abs/2209. 14389, pp. 12207-12222. Association for Computational Linguistics. 9 2023. doi: 10.18653/w//2023.acal/ w/w/23.acllloog..oooooo -ong. 682
URL httpps
//doi org/10..g0... wwt...1116./.12../2
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to -qquence-t--eqqeence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. 2020. acl main. 703
pttttps//acllathooooy..oooo
Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. What makes con- volutional models great on long sequence modeling? In The Eleventh International Con- ference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, volume abs/2210.09298.. OpenReview.net, 10 2022. doi: 10.48550arxii22210000999
URL https
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learn- ing long-range spatial dependencies with horizontal gated recurrent units. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada. neuris ht/ wwtpppppee//////aaaaaaaaaaeeeeea pp. 152 164, 2018 URL www.ccaaaccceeeeeeeeeeee Abst html ec8 Sc8956637a9978777
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv.org, abs/1907. 7 2019. ISSN 2331-8422. URL http::////// /ttp///axvvvv... org
abs/1907.11692
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, volume abs/2209. 10655. OpenReview.net, 2022. doi: 00.45500arii..22011655 URL https: /tEpp:///ppeneview hww.:/ee//peeeeee
3iq2E1
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142- 150. Association for Computational Linguistics, 2011. URL htttpp::/www..alweb..oor.
anthology/PIl-1005
Nikita Nangia and Samuel Bowman. ListOps: A diagnostic dataset for latent tree learning. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. pp. 92-99. Association for Computational Linguis-
tics, 2018, doi: 00.18653/11/N11-400 N18-4013
URL https: www. aclweb org/ antooloythooo/////
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals with state spaces. Advances in neural information processing systems, 35:2846-2861, 2022.
Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In Andreas Krause 0001,
12

---

Published as a conference paper at ICLR 2024
Emma Brunskill, KyungHyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.),
International Conference on Machine Learning, JCML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume abs/2303.00649, pp.
PMLR, 3 2023. doi: 10445500uxi.22033.66449 URL httphttspp:///rceeeeeeiii ngs. ml htt.://ww.............oooooeoooa
Dragomir R. Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara. The ACL anthology network corpus. Language Resources and Evaluation, 47:919 944, 2013.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 1532-4435. URL httthttp:/
10 2019. ISSN
074 html
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent. Truns. Mach. Learn. Res., 2022, 2022.
Hamed Shirzad, Ameya Velingker, B. Venkatachalam, Danica J. Sutherland, and Ali Kemal Sinop. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning, 2023.
Jimmy T.H. Smith, Andrew Warrington, and Scott Linderman. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.
net URL http:ttt///epeeeeeeaaaatae hwt.:/eeetf.rrrre/oeeeeem forum? id=Ai BHW BAXqkk
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv.rr,, abs/2104.09864, 4 2021. ISSN 2331-8422.
URL
awta/aas/l110 09864
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, volume abs/2011.04006. OpenReview.net, 11 2020a. URL https V/openreview net htt:://ommmmmdd//eee
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, ACM Computing Surveys, 55:1 28, 2020b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv.org, httph/2377.0922880 7 2023. ISSN 2331 -4222. doi: 10.48550/arxivv2337.0990 https://doi.org// 48550/ /015550/xxxii arwii. 2307 09288
URL
Petar Velivekovi'c, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Ban- ino, Mikhail Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reason- ing benchmark. In International Conference on Machine Learning, 2022. URL https
7/api. www.nt htt.ee/......ororooooooosoa22oo.
13

---

Published as a conference paper at ICLR 2024
Pete Warden. Speech commands: A dataset for limited-vocabulang speech recognition. ArXiv, abs/1804.03209, 2018.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. Efficient long sequence modeling via state space augmented transformer. arXivorg,
abs/2212.08 36, 12 2022. ISSN 2331 doi: 10.48550/arxiv.2212..000 URL https;
48550//00...455////0/// www.xiii..21..... 08136
A RELATED WORK
Modeling Long Range Dependencies Evaluation of long-sequence models commonly includes the LRA benchmark (Tay al. 2020a), suite of tasks demonstrating the inefficacy of various efficient Transformers long sequences. TTy et al.) (020bb.. The first to obtain high performance on LRA was the S4 model (Gu et al. 2022a), an instance of linear RNNs augmented according to complementary theory in continuous time (Gu et al. 2020; 2022b). Following S4, multiple works have proposed
simplifications to S4 Gupta et al. 2022a Gu et al. 2022b Smith et al. 2023 Orvieto et al. 2023 or augmentations other common architectures (Li al. 2022 Fu et al. 2023 Ma et al. 2022 Zuo
eet al. 2022), aimed at replicating biases observed in S4 and achieving similar performance on LRA.
Common all of the above that evaluation on the tasks done from random initialization and
thus does not encompass the biases from a various pretraining objectives, and specifically denoising.
In Gupta al. (2022b), the authors examine diagonal linear RNNs and Transformers on set of long range tasks with dense (token-level) supervision but do not evaluate on LRA.
Pretraining with Downstream Data Pretraining prior to evaluation already the best practice in many domains and is usually performed using upstream corpora that much larger than the intended downstream task. While various objectives could be used for pretraining (Lewis et al.) 2020]]
Baevski al. 2020 Assran et al.
2023)
focused the typical causal and masked language
2019 Liu et al.) 2019). EI-Nouby et al. (2021) were the first to modeling objectives RRaffrd et al.
show that cuffvsupee pretraining on the downstream task data alone can often match performance of supervised ImageNet pretraining, for object detection and segmentation tasks. In Krishna et al. (2023), the authors provide similar study on NLP tasks, showing multiple instances in which pretraining on the task data performs comparably to off-the ffftthe-shelf models pretrained on large text corpora.
B CONCLUSIONS
In this work, we argued against the common practice of training models from scratch evaluate their performance on long range tasks and suggested efficient and effective solution to mitigate this issue self- self-supervised pretraining on the task data itself. Through a comprehensive array of experiments, we showed that our method consistently leads to dramatically improved performance for multiple architectures across data scales, and allows simpler models to match the performance of complex ones. Our work stresses the need to account for the pretraining stage while designing and evaluating novel architectures in the future.
C APPENDIX
C.I ADDITIONAL EXPERIMENT DETAILS & HYPER-PARAMERS
We provide additional implementation notes and hyper-parameters for reproduction purposes. For all experiments listed the main text, pretraining is performed either with cross-entropy (CE) or L1 loss, and the normalization layer for Transformers always uses LayerNorm in the PreNorm setting. When finetuning a pretrained model we use the checkpoint matching highest validation accuracy / R2 score for CE / loss respectively, and perform search over small grid of learning rates and batch sizes with log spacing, e.g. (1e-3 5e-4 le-4) and (8 32 128), with values for best performance reported in Table C.I
14

---

Published as a conference paper at ICLR 2024
Our experiments performed NVIDIA 3090 and V100 GPUs. All models were trained for a maximum of either 200 epochs or 24h on a single GPU (whichever comes first) for pre-training and fine-tuning, with exceptions for Transformers on PathX & Speech Commands experiments that were trained for a maximum of 5 days on GPUs, and Pythia experiments that were trained for a maximum of 2 days on 4 GPUs. When detailing learning rates for State Space models, learning rates for parameters of the State Space layer itself are set differently to predefined values, we always use values provided by respective authors (Gu al. 2022a Gupta al. 2022b).
Additional specifications such as learning rate schedules and optimizers can be found in our repository.
C.I.1 UNDERESTIMATION OF LONG-RANE ABILITIES OF TRANSFORMERS
(e.g. model size, num. layers, num. attention heads etc.) and model configuration (classification head, normalization layer type etc.) are similar those listed in Appendix A in (Tay et al.) (020a) and provided here in Table with the main exception being the pooling method across sequence dimension during finetuning. We use mean-pooling, max max-pooling or the last element of the sequence, instead of using a [CLS] J token, as done in Tay et al. (2020a). The choice of pooling method for SPT models had most impact on the PathX tasks, but varying across max/mean/last pooling with same as Tay et al. (2020b).
Model
Transformers did not exceed random performance,
C.1.2 COMPARING S4 AND TRANSFORMERS
As listed in the main text, in this section there are discrepancies between model sizes of Transformers and S4. For Text & ListOps. the models in official repository of Gf et al. (2022a) are smaller then the Transformers used in LRA, we tried larger S4 models (from scratch and SPT) matching the Transformer in parameter count which did not result in better performance. For ryper-parametery of S4 models we refer to Gu et al. (2022a) Appendix D.2 Table 11, and 5 results of our grid searches.
of Transformers can be found in Table For the PathX-256 task we use the same configuration as PathX, changing the pooling method again from Mean to Max. For pretraining we use learning rate 1e-3, batch size 8 and batch size 64 for fine tuning with the same learning rate, the trained from scratch model's 's learning rate is lc-4 with batch size 16.
C.1.3 THE ROLE EXPLICIT PRIORS
S4 models follow the setup described in C.1.2 2 for structured and random initialization, with results for grid searches in Table For DLR Gupta et al. 2022b), we use the official implementatior and set the State Space layer parameter as the default, with state size 1024. For the remaining hyper-parameters (model size, num. layers, weight decay) we use the same values S4, with the exception of the normalization layer that fixed to LayerNorm. For the random initialization, reset all model parameter value to random normal values and standard deviation 0.1. For all experiments on PathX we use Max-pooling on the final sequence representation, which we found helpful for optimization.
C.1.4 SELF PRETRAINING IS EFFECTIVE ACROSS DATA SCALES
Experiments in this section use similar hyper-parameters to Table 5 For data restriction, we use fractions of the entire training set: (Num. Train Samples) x (0.005, 0.01, 0.05, 0.1, 0.5, 1.0].
The subsets are inclusive, meaning smaller fractions are contained in bigger ones, to avoid biases from difficult sampled subsets. For experiments on Text we pretrain all models for 40K pretraining steps and finetune the checkpoint with best performance the validation set until convergence. For Image we pretrain all models for 10K step and finetune similarly. The learning rate and batch sizes are fixed across all subsample experiments, on Text we use batch size and learning rate 5e-4 for pretraining and 16, 5e-4 for finetuning, for the trained from scratch model we use batch size 16 and learning rates Ic-3, 5e-4 and use the model with best performance on the validation set. For Image we use batch size 50 for all experiments, learning rate le-2 for pretraining, and learning rate to 2e-3 for finetuning and training from scratch.
tattps://gitien com/ wwwm/aggg8/////////.//oo
15

---

Published as a conference paper at ICLR 2024
Table 4: hyper-parameters for Transformers all sections, section 3.1 are listed the 2nd value was used for pretraining, e.g. (FT, PT). LR Learning Rate, BSZ Batch Size, WD- Weight Decay
(denoted by f). where 2 values
psstOps
Features
Depth
Num Attn. Heads
FF size
Pooling
LR
BSZ
WD
PT Loss
Transformer + Masked SPT!
S12
6
1024
Mean
le-4Jee3
64,128
0.1
CR
Transfomer- + Causal SPT
512
6
8
1024
Last
5c-4, So-4
128,32
0.1
CR
Transfommrr Rotary Masked SPT
512
6
8
1024
Mean
1e-4, 1e-3
64,256
0.05
CE
Text
Features
Depth
Num Heads
FF
Pooling
Transfomerr Masked SPT!
1024
1c-4.5e44
64.32
CE
Tranformer Causal SPT
1024
64.32
CE
Rotary Masked
1024
5e-4.5e-4
CE
Retrieval
Features
Depth
Num Attn. Heads
FF size
Pooling
L.
BSZ.
WD
PT Loss
Transfomerr + Masked SPT!
128
4
512
Meun
4,-4,5
16,32
CE
Tranformer Causal SPT*
128
4
4
512
Mean
1e-3,5e33
48,32
CE
Transfrmer Rotary Masked SPT
128
6
512
Menn
5c-4,5e33
16,32
CR
Image
Features
Depth
Num
Pooling
Transformer + Masked SPTT
3
128
Max
1e-3.1003
16,32
LI
Transformer + Causal SPT!
64
3
128
Max
1e-3,5633
32.32
LI
Transformer + Rotary Masked SPT
256
6
512
Mean
1e-3.1e-3
64.32
L1
Pathfinder
Features
Depth
Num Attn. Heads
FF size
Pooling
LR
BSZ
WD
PT Loss
Transfommer Masked SPT!
128
4
8
128
Mean
Se-4,1o-3
16,16
CE
Transfomerr Masked SPT!
128
128
Mean
Ie.3,le3
256,128
CE
Transfomer Rotary Masked SPT
128
512
Mean
64,32
CE
PathX
Features
Depth
Num Attn. Heads
FF size
Pooling
I.R
BSZ.
WD
Loss
Transformer + Masked SPTT
128
128
Max
5e-4.5e
32,8
CE
Transforeer + Masked SPTT
128
4
8
128
Max
Sc-4,lo33
256,8
CE
Transformer + Rotary Masked SPT
128
5
512
Max
Sc-4,lo-3
32,8
CE
Features
Depth
Num Attn. Heads
FF
Pooling
BSZ
Loss
Transfomer Rotary Masked SPT
128
128
Max
Ie-3,le-3
256,8
Li
Transfomer Rotary Cousal SPT
128
1e-3,1o-3
256,32
Li
BIDMC
Features
Depth
Num Atin. Heads
FF size
Pooling
LR
BSZ
WD
PT Loss
Transfomer- - HR + Rotary + Masked SPT
128
4
4
128
Max
5c-4.5e-4
32,16
L1
Transfomer- - RR Rotary + Masked SPT
128
4
4
128
Max
Se-4.5e-4
32.16
L1
Tansformer Sp02 Rotary Masked SPT
128
4
4
128
Max
1e-4,5e44
8,16
L1
Table 5: Hyper-parameters for SSMs and Pythia in Sections 3.2
3.3
3.5
x denotes no evaluation is BSZ), FT Fine Tuning
reported. Values are reported in format (FT LR, FT BSZ, PT (or training on downstream), PT Pre Training, LR Learning Rate, BSZ- Batch Size.
LR, PT
Approach
Listops
Text
Retrival
Image
Pathfinder
PathX
S4 Rand Init SPT
1e-3.16,1o-3,32
1e-3.16,32
5c-4.64,1e-3.32
1e-4,16,1e-3.4
S4 + Rand Init
5c-4,16
lc-3,16
x
lc-4,128
X
5c-4,16
SPT
1e-3,16
5e-4.16,5e-4,8
5e-4,256,50-448
1e-3,512,1e-3,32
1e-3,64,1e-3,8
1e-3,64,1e-3,8
DLR Rand Init SPT
5e-4.64,5e-4.32
5e-4.16,5e-448
1e-3,64,32
56-4,16,5e-4.8
DLR + Rand Init
1c-4,16
lc-3,16
x
1e-3,16
X
5c-4,64
DLR SPT
1e-4,16,1e-,1128
1e-3,16,5e-4,8
x
1e-3,64,1e-3322
50-4,64,5e-4,8
DLR
5e-4.16
5e-4,16
5e4,664
5c-4,64
Pythia
5c-5,64
5c-4.16
lc-4,256
7c-5,64
lc-4,1024
x
C.1.5 PRETRAINING TEXT CORPORA
We use the pretrained Pythia model available at: htttps://uuginggfeet wtter/aacc.ooeeleeeee.eeeee 1ptthia- 70m- deduped, and finetune without any regularization. We perform grid search over learning rates and batch sizes, similar to previous sections, with best values listed in Table
16

---

Published as a conference paper at ICLR 2024
C.1.6 ADDITIONAL EXPERIMENTS
For S4, both trained from scratch and pretrained, we follow the hyper-parameters used by Gu al. (2022a), performing small grid search similar to previous sections. All Transformers in this section use the Rotary PE method, and use hyper-pas parameeetss match the size of S4, reported in Table On the sCIFAR task Transformers use the same setup as the Image task.
Table 6: hyper-parameters for SSMs in section 3.7
+denotes results are cited from Gu et al. (202a))
Approach
SC-Causal
SC-Bi.
sCIFAR
BIDMC-RR
BIDMC-HR
BIDMC-SpO2
S4 + SPT
5e-3,16.1e-3332 1e-4,16,1e-3,32
1e-3,64,ee3,,64 56-4,16,50-4,8 5c-4,256,5e-4,8
5e-4.64.1e-3332
1e-3,6,11e3,644 1e-3,32
1e-3,64,1e-3,44
S4
1e-3,16,1e-3,128
1e-2,32
le-2,32
C.2 THE ROLE OF EXPLICIT PRIORS EXTENDED RESULTS
We report all results for experiments in section 3.3 tion are the same those used in section 3.2
The pretrained S4 models with HiPPO initializa-
Table Evaluation of different State Space models with and without SPT.
Approach
Listops
Text
Image
PathX
Avg.
S4 Rand Init SPT
61.5
90.5
87.25
91.05
82.75
$4 + Rand Init
59.85
88.34
75.56
73.57
74.33
SPT
61.25
90.34
89.36
96.94
84.47
$4
59.60
86.82
88.65
96.35
82.85
DLR + Rand Init + SPT
56.75
89.13
81.83
94.59
80.47
DLR Rand Init
39.55
73.72
56.70
50.04
55.00
DLR + SPT
60.45
89,94
87,12
96.38
83.37
DLR
57.50
79.65
79.83
92.33
77.33
C.3 ADDITIONAL. EXPERIMENTS DETAILS
For experiments in this section we follow the scheme described in Section for the experimental setup. Namely, for masked pretraining we use fixed masking ratio for each task, for SC and BIDMC we use 25% masking and for sCIFAR we use 50%. In the SC task, when causal model is also trained, we pretrain with causal denoising objective. Due to the significant length of SC sequences, 16K, we split the input to the attention layer into non-overlapping blocks of size 4000 and allow each block to attend to itself and its neighbour(s), similar fashion to experiments on PathX.
In BIDMC, we observed that naive finetuning of pretrained models results in very bad performance, which is an artifact of the label and data scales being very different, e.g. common label values for HR task 80 while input is normalized to [-1, The reported performance is after normalizing the labels to lie in [-1, which does not entail any additional information on the label itself.
D COMPUTE REQUIREMENTS OF SELF PRETRAINING
In sections 3.1 3.2 and 3.7 we observed large empirical gains using SPT compared to simply training models from scratch. However, SPT requires additional compute, it is important to ensure that the reported gains are not an artifact of additional resources and that training longer from scratch does not result in similar gains.
In each of our trained from scratch (TFS) baseline experiments, we ensured that either (1) the training accuracy is almost perfect and validation performance stopped improving for multiple epochs, or
17

---

Published as a conference paper at ICLR 2024
(2) the training loss stopped reducing. E.g. TFS Transformer PathX fail exceed 52% training accuracy after 8 epochs (equivalent to days on V100 GPUs), while SPT for 1 day and finetuning for 1 day achieved training accuracy 2 78%. Across experiments, observed case (2) to occur on a small number runs and case (1) to be the dominant paradigm.
This implies that the gains due to SPT cannot be simply explained by the use of additional compute and possible underfitting of TTS models, but rather by improved generalization as a consequence of the pretraining denoising objective.
To validate this further, we conducted a compute-tied study for the Image and Text tasks from LRA, where we fixed the total number of training epochs across SPT and fineutning (FT) and varied the number of epochs allocated for SPT As shown in Table even a modest amount of SPT outperforms or closely matches the TFS baseline.
Table Comparison of SPT and trained from scratch (TFS) models compute-tied setting. Total number of epochs across SPT and finetuning is fixed and the ratio of epochs dedicated to SPT is varied. Training budget set to 30 epochs for Text and 150 epochs for Image.
SPT Epochs
Image
Text
Transformer
S4
Transformer
S4
0% (TFS)
75.04
87.83
79.08
87.51
20%
84.45
87.15
90.20
89.50
40 %
84.95
87.72
90.56
89.10
60 %
84.32
87.63
90.65
88.87
Despite the efficiency of SPT presented in Table [ the performance still lags behind the unrestricted setting suggesting. However, in our experiments we observed that the SPT phase reaches close the peak performance relatively early in the run, and leads to optimization benefits during finetuning, compared to the trained from scratch model. In Figure 5 we provide the training performance for trained from scratch and SPT models, demonstrating the above claims. This suggests that the computational requirements of our evaluation scheme can be potentially reduced, which we leave for future work.
100.0
100.0
96.0
87.5
90.0
Accurncy
92.5
90.0
87.5
70.0
86.0
65.0
Finetuning
80.0
Trining from Stmtch
Solf Preiraning
80.0
20%
40%
63%
80%
100%
20%
40%
60%
80%
100%
Training Progression
Training Progression
(a) Image
(b) Text
Figure 5: Training accuracy on the downstream and denoising task of Transformers on Image and Text from LRA, across epochs, showing the denoising task is solved relatively quickly (orange) and that finetuned models optimize faster (solid vs dashed blue curves). Training budget is set to 30 epochs for Text and 150 epochs for Image.
'The performance of shown here is the model we retrained using the specified number of epochs different from Table 2 which is cited from the original work.
18

---

Published as a conference paper at ICLR 2024
Table 9: Performance on Image task across model sizes with SPT & trained from scratch.
Approach
100K
300
1M
3M
10M
Transfyrmer+Rotary
68.51
68.51
71.50
75.04
77.88
Transtyrmer+Rotary + Masked SPT
74.43
76.36
84.83
86.04
86.54
81.36
83.63
84.81
88.65
85.73
S4 + Masked SPT
83.45
86.39
88.67
89.36
88.72
E SELF PRETRAINING IS EFFECTIVE ACROSS MODEL SCALES
demonstrated the effectiveness of SPT across data scales. We now demonstrate that SPT is effective across model sizes as well. We focus on the Image task, which exhibits the largest gap across scales (i.e. the gap between Table [ and Table 2). and evaluate both S4 and Transformer with rotary positional embeddings with model sizes spanning four orders of magnitude. While extensive research has been dedicated study the effect of model scale on the performance of Transformers, less literature is available on scaling-laws for state space models (e.g. S4) The results in Table show that the utility of SPT maintained across model scales, consistently outperforming the trained from scratch variants.
In section 3.4
'along with model width and depth, state space models have an additional hyperparameter; the state dimension. We found it difficult scale up the model size without increasing the state size.
19


Reviews:
[{"review_id": "T6sDaw3NTC", "reviewer": "Reviewer_YbGG", "summary": "The paper demonstrates that random initialization of model weights on long sequence benchmarks leads to severe underestimation in performance of transformer architectures. The results show that pre-training on the training data with autoregressive/masked prediction objectives results in much better initializations and final performance. With self pre-training the performance gap between state space models specifically designed for handling long sequences and traditional transformers is much smaller than shown in prior work. Moreover, the self pre-training improves the performance on state space models as well. Overall the paper points out an important baseline which should be adopted more broadly when evaluating different architectures for long sequence tasks.", "questions": "* Figure 3, in the smallest data setting it seems like SPT does not provide monotonically increasing gains as the data size reduces. Is there guidance on what scale of data SPT ends up resulting in poor initializations that the ones used in state space models.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper does a fairly through evaluation of SPT and its impact on evaluation on various benchmarks. The evaluation clearly shows gaps in the current evaluation of different architectures on long sequence tasks. \n* In addition to providing guidance on evaluation practices the experiments also show the effectiveness of data driven initialization across both transformers and state space models. It is also interesting to see SPT to provide better initialization at smaller data scales for state space models. \n* The paper also demonstrates that simplified state space models can perform competitively to their complicated counterparts when initialized with SPT", "weaknesses": "* The paper largely compares the different models on the effectiveness in terms of benchmark accuracy. It would be good to include commentary on SPT computational costs relative to initializations of state space models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8N9kUskQ1H", "reviewer": "Reviewer_fh2t", "summary": "This paper provides a suite of experiments to show that self pretraining (SPT), i.e., pretraining with denoising objectives on only downstream data, most often closes the performance gap between Transformers and state space models (SSMs) on the Long Range Arena benchmark. In the case of Transformers the performance gains from the incorporation of SPT range from 8 to 15% across tasks.\n\nThe experiments also show that in the case of SSMs, manually-designed biases become increasingly redundant when SPT is incorporated.\n\nMore generally, the results suggest the evaluation of different architectures on supervised tasks should incorporate SPT for reliable performance estimation.", "questions": "**Q1.** What can be said about the results of Transformers and S4 on PathX-256?\n\n **Q2.** What is the point of the experiment with Pythia? For instance, what is the single Pythia row in Table 2 to be compared with?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**S1.** The presentation of the main ideas, related work and experimental results is clear.\n\n**S2.** The incorporation of SPT is efficient and extremely effective compared to only training from scratch.\n\n**S3.** The experimental results are thorough and support the main claims in the paper.", "weaknesses": "**W1.** There are no results on computing requirements for SPT and, e.g., how to best combine SPT with supervised fine-tuning.\n\n **W2.** The results on PathX-256 suggest SPT failed to close the gap in this case. This seems to warrant further investigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxEk7Kmak1", "reviewer": "Reviewer_7n9E", "summary": "This paper studies the effectiveness of self-training (SPT) using its own downstream data with Transformer and state space models (in particular S4) for long-range sequence modeling. Specifically, the studies include the impact of SPT on Transformer, S4, and Diagonal Linear RNNs for long-range sequences. Also, the effect of SPT across data size and data-driven vs HiPPO kernel initialization are analyzed. The experiments show that SPT overall improves the performance on Long Range Arena benchmarks, speech commands, CIFAR, and some regression tasks.", "questions": "- Self-training may be effective not only across the data scale but also the model size. Have you tried the same experiments with different model sizes? It would be interesting to see the phenomenon.\n\n- It looks like the hybrid models like SPADE and MEGA outperform other models including transformer+SPT and S4+SPT for many Long Range Arena tasks. Would SPADE+SPT or MEGA+SPT further improve the performance? Is there a reason this comparison is not included in the paper?\n\n- Regarding the experiment about pretraining on text corpora: I appreciate the idea of comparing results with a pretrained model on a large language dataset. However, the current results are meaningless since the downstream task datasets are very different from Pythia 70M. I'm not sure we are able to find the right dataset to cover all the tasks for Long Range Arena. What about separating this experiment from Long Range Arena and showing the comparison for another downstream task on the language domain?\n\n- Are S4/Transformers trained with the same number of epochs as S4+SPT/Transformers+SPT (including pretraining+finetuning)? If it's not, these results should be added. I'm asking to make sure the single-trained models are not undertrained.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly written. \n- The experimental setup sounds and the results are informative.\n- The analysis of conv. kernels learned via SPT compared to the HiPPO kernels is novel and interesting.", "weaknesses": "- Most experiments are performed on Long Range Arena which is relatively small or synthetic. \n\n- The main self-pretraining results (Tables 1 and 2) are not with the latest Transformers and SSMs. \n\n- Some experimental analysis is lacking. See my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ym0J0WiQDe", "reviewer": "Reviewer_yToF", "summary": "This work questions the common procedure of testing sequence model architectures by directly training on downstream supervised tasks (e.g. the LRA benchmark). They propose to pretrain models on the downstream task data (self pretaining or SPT) before fine-tuning on the task. This significantly closes the gap between many sequence models on long-range sequence tasks. The work also investigates the importance of manually designed biases to help methods capture long-range dependencies and finds that SPT makes these biases less important.", "questions": "1. Could you clarify the pretraining vs fine-tuning procedure and how much time is spent on both for each method? Please let me know if I have missed this in the main paper. Appendix B.1 says models were trained for 200 epochs or 24 h \"for pretraining and fine-tuning\", but it is unclear if this means 200 epochs of pretraining and 200 epochs of fine-tuning? If so this would seem to be much more time/epochs than some of the baseline methods were trained.\n\n2. In Table 2, was the Transformer + Rotary embeddings trained without SPT? This seems unlikely. Perhaps there is a typo in the description of the Transformer methods in this table?\n\n3. In Table 2, the X \"denotes computationally infeasible or unreported results\". These are 2 drastically different things and 2 separate symbols should probably be used.\n\n4. The methods evaluated for Figure 2 still seem to use complex valued parameterizations even though they are randomly initialized. Is this still necessary when using SPT? Since complex values can be problematic when scaling large scale systems, it would be interesting if SPT also removed the need for this in SSMs/linear RNNs.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Benchmarks such as Long Range Arena (LRA) are constantly used for new sequence modeling architectures. Questioning some of the assumptions and usefulness of benchmarks such as this is a fresh and interesting direction.\n\n- Most of the experimental results are strong and convincing. The proposed SPT really does seem to improve the performance of methods such as Transformers previously considered to be unable to solve LRA. This is compelling since SPT is more in line with how large models are often trained.\n\n- The investigations of explicit priors and effects across data scales are also interesting angles to explore.", "weaknesses": "- The paper does not seem to take the cost of SPT vs training directly on the downstream task into account, or at least does not make this axis of comparison clear. \n  - The details are unclear in how much time is spent pretraining vs fine-tuning with the SPT procedure and how this compares to the typical training method for these tasks. These points should be discussed clearly in the main paper.\n  - It is stated in Section 3.2 that sub-par LRA performance is often cited as a prime motivating factor for new methods, but it seems efficiency is often just as important a reason new methods are proposed\n  - If one method can be directly trained on the task, while another has to first be pretrained and then fine-tuned, one would need to compare the cost/time/compute/etc to achieve a certain performance to determine if one method is superior, at least in many settings (note this point is less relevant in large scale language and vision settings where pretraining is the norm). \n  - Further exploration and clarification on these points would improve the paper.\n\n- Even when trained with SPT, it seems from the results that methods such as S4 with structured biases consistently outperform methods with less structured biases across almost every task (I believe Text and Retrieval in Table 2 are the only exceptions to this). The reviewer agrees these differences are not as drastic as it seems when using the traditional procedure, but it still seems the structured biases are helping and the traditional procedure is somewhat predictive of the ordering. Or is this just an artifact of the experiments? A potential discussion on this point could be useful. \n\n- Table 1 lists many efficient attention methods that were originally evaluated on LRA. It would have been interesting to see a couple of these methods also trained with SPT to confirm empirically that they also do not perform as poorly when using SPT. (I suspect this is the case, but currently have to guess since no result is provided).\n\n- The tasks considered in this paper are standard, but nonetheless the addition of less synthetic or larger scale tasks and experiments would also make the paper more compelling to a broader audience\n\n- No code appears to be included (it seems it will be made available based on the anonymized link, but would have been nice to explore during the review).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)