Reservoir Transformers
Authors: Sheng Shen
Alexei Baevski
Ari S. Morcos
Kurt Keutzer
Michael Auli
Douwe Kiela
Venue:  (2021)

Abstract: We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear reservoir layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

Under review as a conference paper at ICLR 2021
RESERVOIR TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
We demonstrate that transformers obtain impressive performance even when some the layers are randomly initialized and never updated. Inspired by old and well- established ideas in machine learning, explore variety of non-linear "reser- voir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.
INTRODUCTION
Transformers Vaswani et al. 2017 have dominated natural language processing (NLP) in recent years, from large scale mac hich translation (Ott et al. 2018) pre-trained (masked) language mod- eling (Devlin et al. 2018 Radford et al. 2018), and are becoming more popular in other fields as well, from reinforcement learning (Vinyals et al. 2019) to speech recognition (Baevski et al.) 2019) and computer vision (Carion et al. 2020). Their success is enabled in part by ever increasing com- putational demands, which has naturally led an increased interest in improving their efficiency.
Scalability gains in transformers could facilitate bigger, deeper networks with longer contexts (Ki- laev et al., 2020) Wang al.) 2020; Beltagy al., 20200 Kaplan al. 20200] Tay et al. 2020b). Conversely. improved efficiency could reduce environmental costs (Strubell et al. 2019) and hope-
fully help democratize the technology.
In this work, we explore simple question: if some layers of the transformer are kept frozen- i.e., never updated after random initialization- can we match the performance of fully learned trans- formers, while being more efficient? Surprisingly, the answer is resoundingly yes; and what is more, we find that freezing layers may actually improve performance.
Beyond desirable efficiency gains, random layers are interesting for several additional reasons. Fixed randomly initialized networks (Gallicchio & Scardapane 2020) converge to Gaussian pro- cesses in the limit of infinite width (Daniely et 20016). have intriguing interpretations in metric learning (Rosenfeld & Tsotsos, 2019; Giryes et al. 2016), and have been shown to provide excel- lent "priors" either for subsequent learning (Ulyanov al. 2018) pruning (Frankle & Carbin 2018). Fixed layers allow for efficient low-cost hardware implementations (Schrauwen et al.) (007) and can be characterized using only a random number generator and its seed, which might have repercussions in distributed training and enables highly efficient deployment to edge devices. The strong performance of networks with fixed layers also sheds new light on the inner workings of BERT (Devlin et al.) 2018). and layer-wise interpretations of such models (Rogers et al. 2020 Ten- ney et 2019). It appears that "not all layers are created equal" (Zhang al. 2019 an extent that some layers can simply remain random and fixed.
true such
These ideas have a long history in machine learning. By Cover's theorem (Cover 1963). any high- dimensional non-linear transformation is more likely to be linearly separable than its lower-or-equal- dimensional input space. By Cohnsson-Lindentrann jections distort Euclidean distances very little under mild assumptions, which is useful e.g. for dimensionality reduction and random indexing (Sahlgren) 2005). Fixed random layers in neural networks pre-date deep learning by far (Gamba al. 1961 Baum 1988). Indeed, random kernel 2008 2009).
(Johnson & Lindenstrauss) 1984). random pro-
methods have been an impactful idea in machine learning (Rahimi & Recht
One way think of such layers is as "reservoirs" (Lukoševičius & Jaeger 2009 where highly
non-linear high- high-dimensional black box representation is provided to a lightweight "readout" net-
work, as in echo state networks (Jaeger 2003) and liquid state machines (Maass et al. 2002). The

---

Under review as a conference paper at ICLR 2021
benefit of such approach is that the reservoir has fixed parameters and is computationally efficient, as it can be pre-computed and does not (necessarily) require backpropagation.
In NLP, Wieting & Kicla 2019) showed that random sentence encoders present strong bascline for text classification, with subsequent work showing applications in a variety of NLP tasks (Enguchard 2019 Garg et al. 2020 Pilault et al. 2020). To our knowledge, this work is the first examine this phenomenon in transformers, and the first to recursively alternate reservoirs with subsequent transformer layers acting as readout functions. We introduce "reservoir transformers", wherein fixed random reservoir layers are interspersed with regular updateable transformer layers. The goal of this work not necessarily models on a more solid footing by providing empirical evidence of their capabilities even when some their parameters are fixed. Our contributions are as follows:
et.
a new state of the art, but to put our understanding of transformer
We introduce a new area under the convergence curve metric for measuring performance- efficiency trade-off's, and show that replacing regular transformer layers with reservoir lay- ers leads to better results on that metric.
We show that the addition of reservoir layers in fact leads to improved test set generalization variety of tasks in variety of settings.
We show that pre-trained masked language modelling architectures like BERT and RoBERTa (Liu et al.) 2019) can benefit from having some of their layers frozen, both during pre-training well when fine-tuning on downstream tasks.
In addition, we experiment with different types of reservoir layers, including convolutional and recurrent neural network- based ones. We also show empirical evidence that the back- ward pass can be entirely skipped by approximating top-layer gradients using an approach we call backskipping, with a relatively small sacrifice performance.
APPROACH
This paper is based on a very simple idea. Neural networks are trained via backpropagation, which involves consecutive steps of matrix addition and multiplication, i.c.,
aJ DJ aJ a... 0L, OLD a0,"
0+11 + 02 -1 a0, 00, aL, aL OL Or
for some objective J, parameterization 0 and learning rate 77 with the gradient computed via the chain rule, where L, is the i-th layer of the neural network and x is the input. Let L Transformer(X) X) be single layer in Transformer network (Vaswani et al. 2017), i.e.
H = NulliHeaadeefAAtt(aayyannnnnt + X
(2)
L FFN(LaycrNorm(H)) H
Now, during every "backward pass". we compute the Jacobian for parameters 8² at layer L, which used update the parameters of T., of, well compute the next layer's Jacobian, thus paok-prop the gradients. In this work however, for some of the layers, we still backpropagate through them compute gradients for earlier layers, but we never update their parameters. As result, these layers stay fixed at their random initialization, saving computational resources.
2.1 BACKGROUND
Naturally, never updating some of the parameters is computationally more efficient, as some matrix addition operations can be skipped in the backward pass, but why is this not detrimental to the performance of the network?
In the early days of neural networks, the bottom layers often kept fixed as "associators" (Block) 1962), or Minsky & Papert (2017) called the Gamba perceptron (Gamba et al. 1961 Borsellino & Gambal 1961). Fixed random networks (Baum) 1988 Schmidt et al. 1992 Pao al. 1994) have
2

---

Under review as a conference paper at ICLR 2021
been explored from many angles, including "random kitchen sink" kernel machines (Rahimi & Recht, 2008; ) "extreme learning machines" (Huang et al. 2006) and reservoir comput- ing (Jaeger 2003: Maass al. 2002. Lukoševičius & Jaeger 2009). In reservoir computing, in- put data are represented through fixed random high-dimensional non-linear representations, called "reservoirs", which are followed by regular (often but not necessarily linear) "readout" network to make the final classification decision.
The theoretical justification for these approaches lies in two well-known results in machine learn- ing: Cover's theorem (Cover 1965) on the separability of patterns states that high-dimensional non-linear transformations are more likely be linearly separable; and the Johnson- Lindssn--indestrauuss lemma (Johnson & Lindenstrauss) 1984) shows that random projections distort Euclidean distances very little under mild assumptions.
Practically, random layers can be seen as cheap way to increase network depth. There are inter- esting advantages this approach. Fixed layers are known to have particularly low-cost hardware requirements and can be easily implemented on high-bandwidth FPGAs with low power consump- tion (Hadaeghi et al., 2017; Tanaka et al., 2019). or on optical devices (Hicke et al. 2013). This might yield interesting possibilities for training in a distributed fashion across multiple devices, as well as for neurmorphic hardware (Neftci et al., 2017). This approach also facilitates lawerclatency deployment of neural networks to edge devices, since weights can be shared simply by sending the seed number, assuming the random number generator is known on both ends.
2.2 RESERVOIR TRANSFORMERS
This work explores inserting random non-lime non-linear transformations, or what we call reservoir layers, into transformer networks. Specifically, we experiment with variety of reservoir layers:
Transformer Reservoir: The standard transformer layer described above, but with all parameters fixed after initialization, including the self-attention module.
FFN Reservoir: A transformer-style fixed feed-forward layer without any self-attention, i.e., #FNILayeNNom(PPevious (Pryeeiouusaayerr) + Previous layer.
BiGRU Reservoir: A fixed bidirectional Gated Recurrent Unit (Cho et al. 2014) layer, which is closer spirit to previous work on reservoir computing, most of which builds on recurrent neural network architectures.
CNN Reservoir: A fixed Convolutional Neural Network (LeCun et al. 1998 layer, specifi- cally light dynamical convolution layers (Wu et all, 2019),, which are known to be compet- itive with transformers in sequence-to-sequenec tasks.
We find that all these approaches work well, to certain extent. For clarity, we focus primarily on the first two reservoir layers, but include a broader comparison in Appendix A
In each case, contrary to traditional reservoir computing, our reservoir layers are interspersed throughout regular transformer network, or what we call reservoir transformer. good jus- tification for this approach is that while random projections are not learned and might introduce noise, subsequent normal transformer "readout" layers might allow us recover from any adverse effects of randomness. For example, previous work has shown that ResNets, with all of their param- eters fixed except for the scale and shift parameters of batch normalization, can still achieve high performance, simply by scaling and shifting random features (Frankle et al.) (020). Adding noise to the parameters neural networks is also known to help convergence and generalization (Jim et al.)
1995 1996, Gulcchre al. 2016 Noh et al. 2017
EVALUATION
We evaluate the proposed approach on variety of well-known tasks in natural language processing, namely: machine translation, language modelling and masked language model pre-training
In this work, we not necessarily interested in obtaining the state of the art any task in improving overall task performance via this method. The main objective is to examine effi- ciency, i.e. the relationship between compute time and task performance. This is closely related
3

---

Under review as a conference paper at ICLR 2021
valid BURU AACE
33.5
33.0
0.36
Transformer Reservoir PPN Reservoir
32.5
Trandformer
Reservoir
PFN Reservoir
Updatable Encoder Layera
Updatable Encoder Layera
Figure 1: Validation BLEU AUCC and test BLEU for IWSLT (high is good). Comparison of regular transformer and reservoir transformer with FFN or Transformer reservoir layers added.
efforts in Green AI, which concerned with the trade-offss between compute, data, and per- formance (Schwartz et al. 2019). We propose a new metric for our purposes, the area under the convergence curve (AUCC): similarly how the area under the receiver operating characteristic (Bradley 1997 AUC-ROC) measures a classifier's performance independent of the classification threshold, AUCC measures model's performance independent of the specific compute budget.
Specifically, AUCC is computed as follows:
Σ gl(f(z).)) ZEEED
(3)
where f is the network and g is the evaluation metric, measured until convergence time î, which the maximum convergence time of all models included in the comparison. Note that time here is wall-clock time, not iterations. By convergence, we mean that validation performance has stopped improving, and hence the convergence curve whose time. Runs are averaged over multiple seeds and reported with standard deviation. We normalize raw AUCC scores by their maximum score to ensure more easily interpretable
measure plots the desired metric over
1] range.
One potential downside of this approach is that the AUCC metric could lead to higher scores for model that converges quickly but ultimately worse performance, if measured in a small window. We account for this by making sure that / is set sufficiently high. We include the raw validation curves in the appendix and also report test set generalization in each experiment.
3.1 EXPERIMENTAL SETTINGS AND IMPLEMENTATION DETAILS
We evaluate on IWSLT de-en (Cettolo et al. 2015) and WMT en-de BBoarr et al. 2014 for ma- chine translation; enwiki8 (LLC 2009) for language modelling; and experiment with RoBERTa
Liu et al. 2019) in our pretraining experiments. For IWSLT, we follow the pre-proe pre-processing steps in Edunov al. 2018 The train/val/test split is 129k/10k/ 11//10//6.86. 8k sentences. For WMT, we follow the pre-processing steps in Ott et al. 2018). The train/val/test split is 4.5M/16.5k/3k sentences. For en-
wiki8, we follow the pre-processing steps in Dai al. (001)). The train/val/test split 1M/54k/56k
sentences. For RoBERTa pretraining, we follow the pre- pre-processing steps in Liu al. (2019).
We use Volta V100 GPUs for WMT and enwik8, 32 V100 GPUs for RoBERTa and a single V100 for IWSLT. The hyperparameters for IWSLT14 and WMT16 were set to the best-performing values from Ott et al. (2018) and Kasai et al. (2020) respectively. The enwik8 experiment settings followed Liu were conducted using fairseq (Ott et al. 2019). Our code and experimental settings will be made
Bachlechner al. (2020) and the RoBERTa experiments followed Liu et al. (2019). All experiments
open source at (ANONYMIZED-GTTHH)) B-
4

---

Under review as a conference paper at ICLR 2021
Model
Layers
Frozen
Max BLEU
Train time
Ratio
# Params Trainable (Total)
Train Time each epoch seconds)
until max (in hours)
0.07
2.548
26.8M
122.73
8
34.59 = 0.11
2.557 +: 0.05
1
31.1M
142.28 = 1.87
Transformer
10
34.56
0.05
3.173 1:
161.66 + 1.54
0.12
3.521
39.5M
172.45
1.98
34.37 0.12
2,422 +
0,95
22. 6M (26, 8M)
120.59 = 1.32
Reservoir
8
2
34,80
+ 0.07
2,450 +: 0.06
0.96
26.8M (31.1M)
134.49 + 1.76
34.70
+ 2,831
0.89
3M (35.3M)
144.42 +
12
34.78 0.04
3,476 +: 0.04
0.98
35.3M (39.5M)
159.43 +: 1.67
6
2
34.43 + 0.15
2.120 +: 0.04
0.83
22.6M (25. 8M)
107.71 = 1.73
8
34.56 +: 0.16
2,203 +: 0.06
0.86
26.8M (29. 1M)
120.07 +: 1.65
FFN Reservoir
0,79
31.1M
130.11 1,43
12
2
34,76 #: 0.03
3.241 + 0.04
0.92
35.3M (37.5M)
156.32 #: 1.87
6
2
34,59 + 0.15
2.364 + 0.08
0.92
22.6M (26.8M)
119.30 = 1.36
LayerDrop
34.58 = 0.16
2.554 +
0.99
26.8M (31.1M)
138.62 = 1.44
34.57
+ 0.07
3,404
0.06
1.07
31.1M (35.3M)
140.88 1.62
12
2
33.65 = 0.24
3,251
# 0.04
0.92
35,3M (39.5M)
160.85 = 1.49
Table I: Wall-clock time (averaged over multiple runs) saved for IWSLT for different model types and encoder depths. Max BLEU is for validation. Number of layers is for encoder, decoder depth is kept fixed at 2. Ratio is computed compared to comparable number of layers in the normal case.
All the experiments in this paper were run with 3 random seeds and the mean and standard deviation are reported. For the relatively small IWSLT, the 7 value in the AUCC metric was set to 4 hours. For WMT, which larger, we set it to hours. For enwiki8, it was 30 hours; and for the RoBERTa pre-training experiments, it was set to 60 hours.
The projection weights in random layers were initialized using orthogonal initialization (Saxe et al. 2013)), which makes sense since random orthogonal projections should be most information- preserving, and which was found to work well empirically for initializing fixed random representa- tions in previous work (Wieting & Kiela 2019). Biases and layer
parameters using their respective PyTorch defaults (based on Xavier init; Glorot Bengio 2010).
initialized
We intersperse reservoir layers in alternating fashion starting from the middle. Specifically, alternate one reservoir layer with one transformer layer, and place the alternating block in the middle. For example: a 7-layer encoder LLLLLLL in which replace three layers with reservoirs becomes LRLRLRL, and with two becomes LLRLRLL. See Appendix [C]ii a study comparing this strategy alternative approaches (e.g., freezing in the bottom, middle or top).
4 EXPERIMENTS
In what follows, we first show our main result: reservoir transformers often have better AUCC met- rics, less training time per epoch, less convergence time until the best validation performance achieved, and even improved test set generalization metrics, on variety of tasks. As strong base- line method, we compare to LayerDrop (Fan et al. 2019). LayerDrop can also be seen as a method that dynamically bypasses parts of the computation during Transformer training in attempt to im- prove efficiency, and is a suitable comparison to examine our methods.. We also examine whether can minimize the expectation that we do not have to pass the true gradients through the reservoir for further efficiency.
the gradients of upper layers in the transformer network such
4.1 MACHINE TRANSLATION
Machine translation (MT) is one of the core tasks of NLP. We demonstrate two well-known MT datasets, IWSLT' 14 German- German-English and WMT' 16 English-German German, that reservoir transformers obtain a better AUCC. For the raw validation plots over time that were used to calculate the AUCC, please refer to Appendix
Following Kasai al. (2020), the architecture of the network is N-layer reservoir transformer encoder. followed by a regular shallow one- or two-layer decoder. This design choice has been shown to lead to very good speed and efficiency trade-offs, and serves good baseline for
5

---

Under review as a conference paper at ICLR 2021
valid BLBU AICE
27.50
26.75
Transforerr
Trandformer
0.94
Reservoir
Reservoir
PPN Reservoir
26.25
PFN Reservoir
15
15
20
30
Updatable Encoder Layera
Updatable Encoder Layera
Figure 2: Validation BLEU AUCC and test BLEU for WMT (high is good). Comparison of regular transformer and reservoir transformer with FFN Transformer reservoir layers added.
Transformer
Transformer
Reservoir
Reservoir
bpr RICE
1.6
1.4
Updatable Decoder Layers
Updlatable Deceder Layers
Figure 3: Validation BPC AUCC and test BPC on the enwik8 language modelling task (low is good). Comparison of regular and reservoir transformers for varying depths.
experiments. Moreover, shallow decoders make it easier to decide where to place reservoir layers (in the encoder) and makes more straightforward to identify where performance gains come from.
Figure D shows the results for IWSLT. On the y-axis show validation AUCC for the BLEU metric; on the x-axis we show the number updatable layers in the encoder. The performance of regular transformer encoder with 6 layers and reservoir transformer encoder with 6 layers plus N additional reservoir layers are plotted for the same x-axis value to show the total number of updated layers. Plots for the total number of layers (updatable plus not-updatable, so essentially shifted versions) are shown in Appendix D Table[]shows the time it took to achieve the maximum validation BLEU score and how that relates the regular transformer, demonstrating that reservoir transformers consistently converge faster in terms of wall-clock time, up to 22% much with the same number of updateable layers. We save as much as 27% time until convergence a 24 layer model on WMT, as shown in Table One other noticeable point is that we can see that the T Reservoir achieves similar performance LayerDrop on IWSLT and WMT terms of wall-clock per epoch and wall-clock time to the best performance. However, both tasks, FFN Reservoir performs much terms of efficiency per epoch and achieves better/similar performance in
better than LayerDrop less time in each case. As point of reference, half hour gain IWSLT translates gain of several days in the training of bigger transformer models like GPT-3 (Brown et al. 2020).
We observe that reservoir transformers consistently perform better than, competitive to, reg- ular transformers, both in terms of validation BLEU AUCC as well as test time BLEU, for all examined encoder depths.

---

Under review as a conference paper at ICLR 2021
56
96-
Transformor
Transformer
T Remervolr
T Remerooir
FFN Reservoir
78
FFN Reservoir
91
Transformer (hozen finctuned)
Transformrr thozes finctaned)
10
10
14
Updatable Decoder Layers
Updatable Deceder Layers
Figure Downstream RoBERTa performance on SST-2 (left) and MultiNLL-matched (right).
Figure 2 shows similar trend for WMT. WMT much larger and requires a much deeper encoder, as illustrated by the fact that a certain minimum depth is required for reservoir transformers to achieve a comparable validation AUCC. At test time, reservoir transformers outperform regular transformers for almost all encoder depths. The FFN reservoir transformer seems to work best in both cases, which is surprising because it does not have any self-attention component at all. This finding shows that self-attention, or the mechanism learned if present. Once the context features have been gathered, a random projection via a fixed FFN module appears be beneficial, at least for MT.
summarize context information, should be
4.2 LANGUAGE MODELLING
To examine whether the same findings hold for other tasks, we evaluate on the enwiki8 (LLC) 2009) language modelling task. We examine the BPC (bits per character) rate for variety of network depths (since the task is language modelling, these layers are the decoder). The results show that we obtain consistently better BPC for lower depths, except for the 64-layer regular transformer, which appears to be particularly optimal for this task. We observe similar trends during test time.
4.3 MASKED LANGUAGE MODEL PRETRAINING
2019) models from scratch at a variety of depths, both in the normal and reservoir setting. We find that these networks show minor differences in their best perplexity and similar AUCC perplexity (see Appendix DD. We then examine the performance of these mod- els when fine-tuned on downstream tasks, specifically the well known SST-2 (Socher et al., 2013) and MultiNL Williams et al. 2017) tasks. When fine-tuning the reservoir models, we keep the reservoir layers fixed (including them in fine-tuning did not work very well, see Appendix D).
We train RoBERTa (Liu et al.
Figure shows the results of fine-tuning We observe that the reservoir transformer outperforms normal RoBERTa all depths in both tasks. At lower depth, the improvements are substantial. As sanity check, we also experiment with freezing some of the layers in normal RoBERTa during fine-tuning (Transformer frozen finetuned) and show that this helps a little but is still outperformed by the reservoir transformer.
These findings suggest that you can train RoBERTa model without updating all of the layers, achieve similar perplexity at similar computational cost, but with better downstream performance. The fact that some layers can be kept random and entirely fixed during training, without sacrificing any performance, raises intriguing questions for "BERTology" (Rogers al. 2020 and for the study of what different layers transformers learn.
We report results for

---

Under review as a conference paper at ICLR 2021
Beserval
0.5
3.4
24
Figure 5: IWSLT comparison of normal frozen v backskipped
4.4 BACKSKIPPING
With the reservoir transformers as described above, we obtain better efficiency by skipping the "gra- dient application" matrix addition step in some of the layers (i.e., updating the weights). One step further would be to investigate skipping the entire backward pass for reservoirs altogether, which would save from having do the much more expensive matrix multiplication for these layers that is required for the propagation of gradients. We report on preliminary experiments where in the backward pass replace the gradients for the layer L4 going into the reservoir Li+1 with noisy estimate (Jaderberg et al. 2017 Czarnecki et al. 2017). Promisingly, Oktay et al. (2020) recently asked "why spend resourees on exact gradients when we're going to use stochastic optimization?" and show that you can do randomized auto-differretiatiio quite successfully.
Here, rather than minimizing the actual gradients Jag via continuous-action REINFORCE (Williams) 1992). That is, L; becomes a policy Ta l where we sample actions ~ N (a,, 1). We train to minimize the gradient prediction loss via MSE, (a))) and the REINFORCE loss Eg [log(a) (R (a))), where the value
BL,, we minimize their expectation and train
i... En o network V acts as the baseline. R is defined as the mean of the gradients of the top layer L,42, with the sign flipped. Thus, simply put, we train to minimize the expectation of the true gradients the layer directly following the reservoir. We employ an annealing scheme where we first train the value network and propagate the true gradients during warmup. Afterwards, we anneal the probability of backskipping rather than performing true backward pass (multiplying the probability by 0.99 every iteration until we only backskip). We experimented with setting to the negation of the total loss well but found the current reward to work better. We call this approach backskipping.
Figure 5 shows the results as validation BLEU over time. We observe that this approach helps especially during the earlier stages of training. Although it does not match the performance of the approach with true gradients quite yet, actually performs competitively. Backskipping looks promising approach to further reduce computational costs, and would be more efficient from hardware perspective since the circuitry for such layers (which do not need to propagate gradients) effectively be hardwired entirely.
RELATED WORK
Recent work has shown that modern NLP models are able to function with different numbers of layers for different examples (Elbayad et al. 2019 Fan et al. 2019); that different layers special- ize for different purposes (Zhang et al. 2019); that layers can be compressed (Li et al.] 2020);; and, that layers can be reordered (Press et al.) 2019). There is a growing body of work efficient self-attention networks (Tay et al.) 2020b). such as linear attention (Wang et al.) (020). on how to process long context information (Beltagy et al.] 2020) and on approximations to make trans- formers more scalable (Kitaev et al. 2020 Katharopoulos et al. 2020). BigBIRD (Zaheer et al.) 2020) provides random keys as additional inputs to its attention mechanism. Locality sensitive hashing (LSH) as employed e.g. in Reformer (Kitaev et al. 2020) utilizes fixed random projec- tion. Performer (Choromanski et al. 2020) computes the transformer's 's multi-heas attention weights fixed orthogonal random projection. Closely related to this work, Tay et al. (2020a) showed
8

---

Under review as a conference paper at ICLR 2021
that randomized alignment matrices in their "Synthesizer" architecture sufficient for many NLP tasks. While these works focus on random attention, we show that entire layers can be random and fixed. We also show that entire layers can be replaced by fixed random projections that do not have any attention whatsoever.
Beyond transformers, random features have been extensively explored. Examples of this include
FreezeOut (Brock et al. 2017), deep reservoir computing networks (Scardapane & Wang. 2017]
Gallicchio & Micheli 2017), well applications in domains as varied as text classification (Con-
neau et al. 2017: Zhang & Bowman 2018; Wieting & Kiela 2019) or music classification (Pons
& Serra 2019). It is well known that randomly initialized networks can display impressive perfor-
mance on their own (Ulyanov et al.) 2018 Rosenfeld & Tsotsos 2019; Ramanujan et al., 2020),
which underlies, for example, the recently popularized lottery ticket hypothesis (Frankle & Carbin 2018 /Zhou et al. 2019). We know that learning deep overparameterized networks appears to help in general ([. & Liang 2018 Du et al. 2019). Our method represents an easy and cheap way to add both depth and parameters transformer networks.
6 CONCLUSION
This work demonstrated that state-of-the-art transformer architectures can be trained without updat- ing all of the layers. This complements long history in machine learning of harnessing the power of random features. In most cases, "reservoir transformers" achieve better performance-efficien trade-offs as measured by our newly introduced AUCC metric, and better test generalization, on variety of tasks and in variety of settings. Future work includes further investigating hybrid networks and hackskipping architectures, as well as utilizing pruning strategies at inference time, in order to try to obtain even better performance/effcciency trade-offs.
REFERENCES
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley. Rezero is all you need: Fast convergence at large depth. arXiv preprint arXiv:2003.04887, 2020.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910.05455 2019.
Eric B Baum. On the capabilities of multilayer perceptrons. Journal of complexity, 4(3):193-215, 1988.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051555 2020.
Hans-Dieter Block. The perceptron: model for brain functioning. i. Reviews of Modern Physics, 34(1):123, 1962.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Kochn, Johannes Lev- cling, Christof Monz, Pavel Pecina, Matt Post, Herve
Radu Soricut, Lucia Specia, and Aleš Tamehyna. Findings of the 2014 workshop statistical machine translation. In Pro- ceedings of the Ninth Workshop on Statistical Machine Translation, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics.
A Borsellino and A Gamba. An outline of a mathematical theory of papa. Il Nuovo Cimento (1955- 1965), 20(2):221-231, 1961.
Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1 145-)1999 1997.
Andrew Brock, Theodore Lim, James M Ritchic, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint arXiv:1706 706604493, 2017.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint a005 14 165, 2020.

---

Under review as a conference paper at ICLR 2021
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. 12X1::2005.12872, 2020.
End-to-end object detection with transformers. arXiv preprint
M. Cettolo, J. Nichues, S. Stüker, L. Bentivogli, and Marcello Federico. Report on the 11 th iwslt evaluation campaign iwslt 2014. In Proceedings of IWSLT, 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: |406. 1078, 2014.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamas Sarlos, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for pro- teins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03355, 2020.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data, arXiv preprint arXiv:/70Ss 02364, 2017.
Thomas M Cover. Geometrical and statistical properties of systems of linear inequalities with ap- plications in pattern recognition. IEEE transactions on electronic computers, (3):226-334, 1965.
Wojeiech Marian Czarnecki, Grzegorz Šwirszez, Max Jaderberg, Simon Osindero, Oriol Vinyals, and Koray Kavukcuoglu. Understanding synthetic gradients and decoupled neural interfaces. arXiv preprint arXiv: / 703.00522, 2017.
Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xe XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances In Neural Information Processing Systems, pp. 2253-2261, 2016.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pp. 1675- 1685, 2019.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedingg of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), New Orleans, Louisiana, June 2018. Associa- tion for Computational Linguistics.
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. arXiv preprint arXiv:I910.1007) 2019.
Joseph Enguehard, Dan Busbridge, Vitalii Zhelezniak, and Nils Hammerla. Neural language priors. arXiv preprint arXiv: 1xX.v:99/0003492 2019.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv: 1909. / 1556, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803 2018.
Jonathan Frankle, David J Schwab, and Ari Morcos. Training batchnorm and only batchnorm: On the expressive power of random features in enns. arXiv preprint arXiv:2003.001: 2020.
10

---

Under review as a conference paper at ICLR 2021
Claudio Galliechio and Alessio Micheli. Echo state property of deep reservoir computing networks. Cognitive Computation, 9()):337-3533 2017.
Claudio Gallicchio and Simone Scardapane. Deep randomized neural networks. In Recent Trends in Learning From Data, pp. 43-68. Springer, 2020.
A. Gamba, L. Gamberini, G. Palmieri, and R. Sanna. Further experiments with papa. Il Nuovo Cimento (1955-1965), 20-2): 1961.
Ankush Garg, Yuan Cao, and Qi Ge. Echo state neural machine translation.
arXiv preprint
arXiv:2002.118877 2020.
Raja Giryes, Guillermo Sapiro, and Alex M Bronstein. Deep neural networks with random gaussian weights: A universal classification strategy? IEEE Transactions on Signal Processing, 64(13); 3444 3457, 2016.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and stafistics, pp. 249-256, 2010.
Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In International conference on machine learning, pp. 3059-3068, 2016.
Fatemeh Hadaeghi, Xu He, and Herbert Jaeger. Unconventional Information Processing Systems, Novel Hardware: A Tour D D'Horizon. 2017.
Konstantin
Hicke, Miguel Escalona-Moran, Daniel Brunner, Miguel Soriano, Ingo Fischer, and Claudio Mirasso. Information processing using transient dynamics semiconductor lasers sub- ject delayed feedback. Selected Topics in Quantum Electronics, IEEE Journal of, 19:1501610- 1501610, 07 2013. doi: 10.1 10.109//STQE.223
Guang-Bin Huang, Qin-Yu Yu Zhu, and Chee-Kheong Siew. Extreme learning machine: theory and applications. Neurocomputing, 70(1-3):48-501,, 2006.
Max Jaderberg, Wojeiech Marian Czarnekki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Inter- national Conference Machine Learning, pp. 1627-1635. PMLR, 2017.
Herbert Jaeger. Adaptive nonlinear system identification with echo state networks. In Advances in neural information processing systems, 2003.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Kam Jim, Bill G Home, and Lee Giles. Effects of noise on convergence and generalization in recurrent networks. In Advances in neural information processing systems, pp. 649-656, 1995.
Kam-Chuer Jim, C Lee Giles, and Bill G Home. An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Transactions on neural networks, 7(6):1224-1438, 1996.
William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics, 26(189-206)) 1984.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah A Smith. Deep encoder, shal- low decoder: Reevaluating the speed-quality tradcoff in machine translation. arXiv preprint arXiv:2006. 10369, 2020.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers mns: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006. 16236, 2020.
11

---

Under review as a conference paper at ICLR 2021
Yoon Kim. Convolutional neural networks for sentence classification. 0rXiv:1408.5882. 2014.
arXiv preprint
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0445 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):22788223 2324,, 1998.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157- 8166, 2018.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gon- zalez. Train large. then compress: Rethinking model size for efficient training and inference of transformers. arXiv preprini arXiv:2002. 2020.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 16922 2019.
MultiMedia LLC. Large text compression benchmark. 2009.
Mantas Lukoševičius and Herbert Jaeger. Reservoir computing approaches to recurrent neural net- work training. Computer Science Review, 3(3), 2009.
Wolfgang Maass, Thomas Nätschläger, and Henry Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural computation, 14 (11):2531-2560, 2002.
Marvin Minsky and Seymour A Papert. Perceptrons: An introduction to computational geometry. MIT press, 2017.
Emre Neftei, Charles Augustine, Somnath Paul, and Georgios Detorakis. Event-driven random pack-propagation: Enabling neuromorphic deep learning machines. Frontiers in neuroscience, 11:324, 2017.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. Advances Neural Information Pro- cessing Systems, pp. 5109-5118 2017.
Deniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson, and Ryan Adams. Randomized automatic differentiation. arXiv preprint arXiv:2007.10412, 2020.
Myle Out, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv: 1806.00187 2018.
Myle Ott, Sergey Edunov, Alexci Bacvski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Yoh-Han Pao, Gwang-Hone Park, and Dejan Sobajic. Learning and generalization characteristics of the random vector functional-link net. Neurocomputing, 6(2):163-180, 1994.
Jonathan Pilault, Jaehong Park. and Christopher Pal. On the impressive performance of randomly weighted encoders in summarization tasks. arXiv preprint arXiv: 2020.
Jordi Pons and Xavier Serra. Randomly weighted cnns for (music) audio classification. In ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 336-340. IEEE, 2019.
Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. arXiv preprint arXiv: 1.03864, 2019.
12

---

Under review as a conference paper at ICLR 2021
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2018.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, pp. 1177-1184, 2008.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in neural information processing systems, pp. 1313- 1320, 2009.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 11893-1 1902,, 2020.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What know about how bert works. arXiv preprint arXiv:2002.12322/ 2020.
Amir Rosenfeld and John Tsotsos. Intriguing properties of randomly weighted networks: Gener- alizing while learning next nothing. In 2019 16th Conference Computer and Robot Vision (CRV), pp. 16. IEEE, 2019.
Magnus Sahlgren. An introduction to random indexing. In Methods and applications of semantic in- dexing workshop the 7th international conference terminology and knowledge engineering, 2005.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint arXiv: 1312.6/20, 2013.
Simone Scardapane and Dianhui Wang. Randomness in neural networks: overview. Wiley Inter- disciplinary Reviews: Data Mining and Knowledge Discovery, 7(2)=e1200, 2017.
Wouter Schmidt, Martin A Kraaijveld, and Robert PW Duin. Feedforward neural networks with random weights. In Proceedings of the 11th International Conference on Pattern Recognition, 1992. Vol. I.. Conference B: Pattern Recognition Methodology and Systems, pp. 1-4, 1992.
Benjamin Schrauwen, Michiel D' d'Hanne, David Verstraeten, and Jan Campenhout. Compact hard- ware for real-time speech recognition using liquid state machine. pp. 1097 1102, 09 2007. doi: 201109/0NNNN2207777711111
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. arXiv:19077 10597, 2019.
Green ai. arXiv preprint
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 1906.0224,, 2019.
Gouhei Tanaka, Toshiyuki Yamane, Jean Benoit Héroux, Ryosho Nakane, Naoki Kanazawa, Seiji Takeda, Hidetoshi Numata, Daiju Nakano, and Akira Hirose. Recent advances in physical reser- voir computing: review. Neural Networks, 115:100 123, 2019.
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00744 2020a.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint arXiv:2009.06732 2020b.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint /rXiv:/905.05950, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9446-9454, 2018.
13

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 59985-908, 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom I.. Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukeuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782): 350-354, November 2019.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification. arXiv preprint 19N1:1991.10444 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.
Ronald J Williams. Simple statistical folloing algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.104300 2019.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv2000.11066. 2020.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 19Xxv0/902.0199 2019.
Kelly Zhang and Samuel Bowman. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP, 2018.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances Neural Information Processing Systems, pp. 3597-3607, 2019.
14

---

Under review as a conference paper at ICLR 2021
34.0
valid BLRU AACE
32.5
1542 BLEU
33.0
Transfomer
32.5
Transfomer
RReeervoir
T RReservarr
FFN Reservoir
FFN Reservolr
GRU Reservoir
32.0
GRU Reservoir
Conv Reservoir
Cosv Reservoir
Updatable Encoder Layera
Updatable Encoder Layera
Figure 6: IWSLT comparison of different hybrid architectures with different reservoir layers.
24.6
0.99
34.2
woldd BLEU AUCE
tart BLED
34.0
8.98
33.6
33.4
Transformer Reservoir FFM Recervoir
Transformer
12.2
Reservoir
FFN Recervoir
Updatable Encoder Layers
Updatable Encoder Layers
Figure 7: IWSLT validation AUCC and test BLEU with 6-layer decoder.
A HYBRID NETWORKS AND NON- TRANSFFORRM RESERVOIRS
We investigate whether reservoir layers need be transformer-basee based (or transformers-withon attention, i.c. FFN), We examine two different alternatives: bidirectional Gated Recurrent Units (Cho et al. 2014) and Convolutional Neural Networks (LeCun et al. 1998 Kim 2014), specifi- cally light dynamical convolutions (Wu et al. 2019). Figure shows the results for these hybrids: depending on the setting, they may obtain a better AUCC than the regular transformer, but this is less consistent than with the other reservoir layers, most likely because these layers have different computational properties. It's possible that these hybrids simply require further tuning, as we found e.g. up-projecting help for BiGRUs, but studying this is outside of the scope of the current work.
B DEEP DECODERS
We show that the same results hold for 6-layer decoder on IWSLT (although less pronounced for AUCC, probably because the decoder is computationally heavier). See Figure and Table
C FREEZING STRATEGY
We explored different strategies for the placement of reservoir layers and found the "alternating" strategy reported in the main body the paper to work best. Generally, we found repetitive appli- cation of reservoirs yield diminishing returns, might be expected. See Figure 8
15

---

Under review as a conference paper at ICLR 2021
Model
Layers
Frozen
Max BLEU
Train time
Ratio
# Params
Train Time each epoch seconds)
until max (in hours)
Trainable (Total)
0.05
39.5M
177.84
8
34,99 = 0.08
2.161 + 0.03
1
43.7M
206.59 = 3.47
Transformer
10
34.98 0.04
2.345 1: 0.02
17.9M
236.72 :
34.78 0.11
2.535
52.0M
265.90
34.73 0.11
1.838 0.01
0,92
35.3M
(39.5M)
166.11 2.21
Reservoir
8
2
35.07 +: 0.05
1.912 +: 0.03
0.88
39.5M (M)
190.08 + 3.73
35,02 0.01
1,970
0.84
43.7M (47.9M)
204.42 + 2.89
12
35.06
0.02
2,499 +: 0.02
0.95
47.8M (52.0M)
236.41 +: 4.35
6
2
34.85 + 0.10
1,729 +: 0.03
0.87
35.3M (37.4M)
161.72 = 2.32
8
34,99 +: 0.11
1.751 + 0.02
0.81
39.5M (41 (1)6M)
180.21 + 2.68
FFN Reservoir
0.03
0.81
43.7M
12
2
35.16 :: 0.04
2.395 +: 0.01
0.94
47.8M (49.9M)
216.08 #: 2.57
6
2
34,51 + 0.12
1,908 +: 0.04
0,96
35.3M (39.5M)
169.62 : 3.16
LayerDrop
34.77 0.11
2.023 0.02
0.94
39.5M (43. (43.7M)
186.71 = 2.17
34,06
0.05
1.912 0.02
0.97
43.7M (47.9M)
205.52 + 3.31
12
2
34.08 = 0.13
2,524 # 0.01
0,99
47.8M (52.0M)
222.45 = 2.21
Table 2: Wall-clock time (averaged over multiple runs) saved for IWSLT for different model types and encoder depths. Max BLEU is for validation. Number of layers is for encoder, decoder depth is kept fixed at 6. Ratio is computed compared to comparable number of layers in the normal case.
Model
Layers
Frozen
Max BLEU
Train time
Ratio
# Params Trainable (Total)
Train Time each epoch (in hours)
until max (in hours)
12
24.46 + 0.04
15.15 0.15
75 63.6M
0.505 + 0.005
Transformer
24.52
16.05 0.18
0.643
24
24.69 +: 0.05
17.61 0.85
113.4M
0.877 + 0.029
32
24.83 + 0.04
18.42 2 0.28
138.6M
1.036 0.010
12
24.26 + 0.08
14.11 + 0.21
0.93
72.4M (75 6M)
0.472 + 0.007
24.50
0.28
75.6M
Reservoir
24
25.11 + 0.07
15.89 + 0.74
0.90
100.8M (113.4M)
0.776 + 0.024
24.66 +: 0.04
0.24 16.38
0.88
126.0M (138.6M)
0.998 0.009
24.42 + 0.05
14.01 0.09
0.92
72.4M (7L
0,441 0.003
24.65 + 0.07
14.53 1.53:2.11
75.6M (83.9M)
0.524 0.006
FFN Reservoir
24
24.93 + 0.04
12.62 1.53
0.71
100.8M (109.2M)
0.743 + 0.018
24.98 +: 0.03
13.96 0.19
0.73
126.0M (134.4M)
0.964 0.007
24.27 + 0.03
14.61 # 0.14
0.96
72.4M (75.6M)
0.489 0.006
LayerDrop
24.15 0.06
15.55 1.5520.50 0.54
0.97
75
0.597 0.017
24
24.37 + 0.05
16.25 4 0.36
0.92
1M0.8M (113.4M)
0.823 + 0.013
23.84 + 0.03
15.27 0.38
0.83
126.0M (138.6M)
1.028 0.012
Table 3: Wall-clock time (averaged over multiple runs) saved for WMT for different model types and encoder depths. Max BLEU is for validation. Number of layers is for encoder, decoder depth is kept fixed at Ratio is computed compared to comparable number of layers in the normal case.
D ROBERTA RESULTS
Here we present the additional RoBERTa results for convergence plot and AUCC in various decoder depth setting in Figure 10 As stated in the main paper, the difference of AUCC / Convergence Plot between RoBERTa model with or without Reservoir layers are limited. Moreover, we plot the downstream task performance for SST-2 and MNLI compared to the pretraining wall-clock time Figure 9 can seen that the FFN Reservoir can achieve up to 25% and 10% pretraining time savings while matching the best performance of vanilla transformers for MNLI-m and SST2, respectively.
E RESERVOIR RESULTS FOR TOTAL LAYERS
Here present the shifted Reservoir Results for IWSLT14, WMT16, Enwik8 and RoBERTa fine- tuning in Figure| II
12 1 14 respectively. We show the same results also hold when it comes to replace normal transformer blocks with Reservoir blocks at least for MT.
16

---

Under review as a conference paper at ICLR 2021
Medel
IWSEI-Dee2
IWSL T-Deo
WATF-Del
Lapers
Train liase
Max BLEU
# Layers
Train time
Maa BLEU
I Layers
Trais time
Max BLBU
unii 4 mas lie booms)
(90%)
usiil 95% usa Csa hours)
(95%)
unil
95% mus (1e houes)
(959)
Trancfomeer
10
0,508 0.02
0.05
13.96 0.08
10
0,98 0004
53.52 0.90
24
5.262
0.003
23.50 0883
12
1087 1.06
33.07
20,09
12
1J037 0077-8000 0.033
33.07 60.11
32
6,212
= 0.222
25.91 = 0004
0.500:1 1.02
0.01
0.999 0001
0..6
0.065
Reservoz
50
01:29 0.00
35.05
0.00
0.238 0103
50,57
0.06
24
4.223 0.773
25255 000
13
0.962 4: 1:00
33:03
6.11
12
0,958 - 0:01
33.40 0: 0.29
57700 0.214
25.71 0 0.03
0,521 1..
12.85
0.03
0003
01.6
0.144
0.614 0:01
33.05
0.15
0.08
4.15
12
0511 I:00
33.36
00.00
12
0.780 0/12
03:00:4 0.06
32
4.884
0.323
23,82 = 0.02
0.00
0:706 0001
V108 00.05
0.098
Layerimop
0.901
1.00
33.12
0.04
3.581 = 0.076
0.914 0.01
32.35 0.06
0,93 0002
32.94 0.10
5.900 0,21
32.97 0.04
Table 4: Wall-clock time (averaged over multiple runs) saved for IWSLT/WMT for different model types and encoder depths. 95% Max BLEU is for validation.
Medel
JWSLI-Deer
LWSL.-DDec
WATE-Decl
e Layers
Trais time
Max BLEU (995)
# Layers
Train time
Max BLEU
Layers
Trois time
Max BL
until 9915 max (Er boors)
until 99% mox Cin hours)
(995)
until
99% max (e hroes)
(999))
Transforaer
10
1.475 0.00
1.506-8 0.04
34.33
34.25 0.04
10
1,622 0:15
1,390 0.02
34.75
34.64 0.05
24
12.023 0.072
13.412 d: 0.557
24.49 0.07
12
2.250 0.00
34.24 0.11
12
1,748 27388.000 0.01
34.65
0.00
32
15.10 0.332
24.50 = 0.02
RRssevoi
10
1.500 0.06
34.36 0.02
1,49 0/14
1,330 0.03
34.72 0.08
24
10.413 2: 0.590
9.221 0.073
24.50= 0.03
11
2,045 11.0
14.55
0.07
12
1.41 00422000
54.57 0.00
32
11. 11055-18800 01322
24:49 L 0.01
0.00
PFN Reservor
1.101
2261 0:01
14:16- 0.001
1.776 0001
5461 0.00
9.336
0.978 0.546
36.91 000
13
7765 1 (N.05
14:45
0.00
12
1/400 0:01
54.87
0.00
32
10:504 01541
76:56 0001
1.363:2 0.05
34.58
1253 0.0
0.10
3.312 0.039
LayesDoop
0.06
34.55
0.07
135 000
0.00
10:155 0.678
36.07 0.09
13
2.001 à 0.00
33.45
0.73
12
1.433 0.07
3197 0:122
32
10 161 10000-00 0.77
25.11-000 0.03
Table 5: Wall-clock time (averaged over multiple runs) saved for IWSLT/WMT for different model types and encoder depths. 99% Max BLEU is for validation.
F VALIDATION PLOTS
Here we present the validation plots for training 8-layer encoder, 2-layer decoder model for IWSLT14, a 24-layer encoder, 1-layer decoder model for WMT14, a 48-layer decoder model for enwik8 and 12-layer decoder model for RoBERTa for detailed steps to calculate the AUCC. It can be clearly observed that given the configurations from Section 3.1 all the models have converged. So when we compute the area under the convergence curve, this depicts the training efficiency of the model (basically time x performance) until convergence. Specifically, we set T sufficiently high for computing the AUCC, which 4h for IWSLT, 20h for WMT, 30h for enwik8 and 60h for RoBERTa pretraning, From the training plot in the appendix, we can see that each model has converged at that point. The Reservoir model in Figure [5] has 2 layers frozen for IWSLT14, 8 layers frozen for enwik8, and 4 layers frozen for WMT14 and RoBERTa.
G ROBERTA PROBING
We follow Jawahar et al. (219)) and investigate what the frozen layers in the Reservoir Transformer have actually "learned" (while being forzen) as measured by probing tasks, reported in Table The results are gathered over 3 random seeds for reporting the mean and standard deviation. From the table, we can see that generally probing performance is quite similar between Transformer and the T Reservoir model. We also noticed that the representations collected after the frozen layer (3, 5, 7, 9) in the T Reservoir actually have significantly better performance over the regular Transformer representations across all the probing tasks. This has interesting repercussions for the study of "BERTology". as it clearly shows, somewhat confusingly, that even completely random and frozen layers represent linguistic phenomena.
17

---

Under review as a conference paper at ICLR 2021
0.99
0.98
BLEU AUCE
0.97
Transformer
Reservoir
0.93
Mid Reservair
Top Reservorr
0.92
Layers
Figure 8: IWSLT with 2-layer decoder using different freezing strategy.
95-
84
91
Reservoir
Pretraiting Wallslock Time
Protraining Wallslock
Figure 9: RoBERTa Reservoir Results, Pre- Pre-training versus downstream task plot for 12 layer RoBERTa. MNLI-m (left). SST-2 (right).
1.000
0.75
0.40
0.23
Tandorrew
0.60
Fint Amorrsk
Speciatde Deveder Lagens
Figure 10: RoBERTa Reservoir Results, Training plot for 12 layer RoBERTa (left). AUCC result (right).
18

---

Under review as a conference paper at ICLR 2021
1.00
voled BLEU AUCC
BLEU
33.5
Transformer Reservoir Reservair
32.5
Trandformer
Resercoir
Recervoir
10
12
10
12
Total Encoder Layers
TTttl Encoder Layers
Figure 11: Validation BLEU AUCC and test BLEU for IWSLT (high is good). Comparison of regular transformer and reservoir transformer with FFN or Transformer reservoir layers added.
1.00
23.00
0.50
valie MLUU AUCE
0.58
9.97
27.25
27.00
0.55
26.50
0.54
26.35
12.5
17.5
20.0
22.5
25.0
20.0
12.5
17.5
20.0
22.5
25.0
20.0
Layars
Figure 12: Validation BLEU AUCC and test BLEU for WMT (high is good). Comparison of regular transformer and reservoir transformer with FFN or Transformer reservoir layers added.
vette Spe MNCE
1.6
1.8
1.2
76
Total Decodir Layers
Total Decodirr Layers
Figure 13: Validation BPC AUCC and test BPC on the enwik8 language modelling task (low is good). Comparison of regular and reservoir transformers for varying depths.
19

---

Under review as a conference paper at ICLR 2021
26-
54
94
92
80-
Transformer
Transformer
78
FFN Transfumer (froen finetuned)
Trandformer (hozen finetuned)
Total Decoder
Total
Figure 14: Downstream RoBERTa performance on SST-2 (left) and MultiNLI-mathedd (right).
busorvl
Tradkee
Figure 15: IWSLT with 2-layer decoder validation plot (upper left). WMT with 24-layer decoder validation plot (upper right). Enwik8 with 48-layer decoder validation plot (lower left). RoBERTa with 12-layer decoder validation plot (lower right).
Model
Layer
Sentlem (Surface)
TreeDepth (Symachic)
TopConst (Syntactic)
BShhift (Synacticc)
Tense
SabjiNum (Semanic)
Nbj (Semantic)
SOMO (Semantic)
Coordlar (Sementic)
(Semaninc)
84.56 11.4
32.30
54, 0:35
49
0.01
50.95
0.32
76.26
0.09
50.01 0.19
76.35
33.63
58.38
50.12 0.17
52.84
0.65
78655 0.19
0.53
0.16
32.00
54.41
0.10
50.02
0.01
51.72
0.59
77.00
0.13
51.32
0.64
56.13
0.30
32.50
50,06 0:21
69.76 0.36
51.63
1.17
76.47
0.00
52.41
1.24
31.14 0.45
44.76 0.38
74.52
0.31
0.19
0.16 75.66
52.05
72:90
TranSoemer
16.65
0.13
78
0.30
0.38
23.46
25.19 1.02
77.42 0.27
80.35
0.05
0.59
21.17
31:21
58.42
66.77
22.54
41.01
86.13
62.01 0.59
11
71.37
30.22
45.58 0.35
84
57.25
0.59 59.41
0.15 84.46
12
71.66
0.12
33:43
64.38 0.20
87.38
0.00
63.01
65.72
31.60
50.38
30.00
0.15
76.47
34.70
61:41 0:42
60
0.65
0-77
33
0
50.75
89.28 1009
36.42
67.36
0.45
75.64
0.52
0.18
HIL53
0J02
$2.50
57.16
74.31
0.32
32.42
0.83
55.19
0:33
A5
0.00
79.56
0.00
15
0.08
53.68
0..5
75.02
0.19
36.30
0.08
88.03
0.22
38.34
0.64
68.65
0.29
82:15
0.12
$6.80
6.02
R1.27
0.33
$7.95
0.24
88.82
0.91
58,85
0.10
Reservoir
85.82
74.55
0.37
1L37
33.13
37.63
1: 0.29
0.13
70.43
52.70
1: 0:31
0.05
84.12
29:11
0.13
0.35
56.85
55.70
1 0.36
6.07
82.86
77.45
0.30
0.03
57.26
61.17
0.19
0.21
88.79
75.35
0.17
0.66
61.33
51.55
0.95
1.30
71.69
0.71
30.32
0.01
43.04
0.30
19.22
0.12
54.75
0.09
79.25
0.11
59.53
0.166
76,80
0.41
57.34
0.14
9
NE.86
IL12
37.59
0,03
68.53
0.37
85.55
0.12
$7.98
0.22
84.13
0001
63.06
0.01
N2.55
0.31
66.07
0.05
10
69.25
1123
25.5%
1 0.35
29.20
0.5%
78.57
0.00
55.02
0.03
75.68
0.16
$7.55
1,57
74.70
0.02
55.02
0.64
11
65.77
70
0.05
10.57
+ 0.01
47
56
000
81 30
0.00
86.
75
0.02
73-7
0.05
40
+ 0.17
80.50
0.15
62.50
0.11
12
70.61
= 0.88
34 + 0.20
64.19 + 00
84.53
0.00
63.45 + 0.16
84 - 0.14
63.75
+ 0.14
62.08 + 0.03
64.73
0.06
Table 6: RoBERTa Probing Results. The line in bold text are the the frozen layers in the T Reservoir.
20


Reviews:
[{"review_id": "meerRhluas6", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper studies how transformers train when some layers are kept fixed at the randomly initialized parameters. The authors observe that transformers can be effectively trained with a significant percentage of their layers \"frozen\". It is also argued that as a result transformers can be trained more efficiently as for the frozen layers gradients with respect to the parameters need not be computed.\n\n### Strengths\n\n- The proposed idea is very simple and easy to understand and implement.\n- The authors perform thorough experiments on several datasets and tasks.\n\n### Weaknesses\n\n1. The AUCC metric is not measuring the efficiency independent of the time budget as it is claimed. For instance given a high enough T_hat the best performing model will always have the highest AUCC. Contrary to that, given a low enough T_hat the fastest to converge model will always be better. I agree with the authors that measuring the efficiency of a neural network is a hard problem but I don't think that the AUCC metric is a good solution. Time to X% of best score is probably much more informative. In addition, time to best score is also not very informative as it depends heavily on the learning rate schedule and the random oscillations of training.\n2. From the provided comparison with respect to the number of updateable layers we cannot deduce information regarding the efficiency. For instance in Fig. 2 for WMT how many \"frozen\" layers are used for the reservoir transformers? What is the wall-clock time per epoch for each model?\n3. From the comparison in the supplementary material with respect to all the layers in the model we observe that the performance is on par with standard transformers but what is the improvement in efficiency? Namely, how much faster are the reservoir transformers for a given number of layers?\n\n### Reasons for recommendation\n\nThe paper proposes a very simple idea and then evaluates it experimentally. Therefore, the experimental section should be thorough and should lead to clear conclusions. Due to the the newly proposed metric as well as the lack of comparison between training time and achieved performance lead me to propose rejection.\n\n### Miscellaneous\n\n- Figure 14 bottom right should probably read \"Validation PPL\".\n\n### Post-rebuttal update\n\nI would like to thank the authors for their additions to the paper. I believe that the extra metrics improve the reader's ability to extract conclusions from the experiments significantly.\n\nHaving said that, I believe that the extra experiments and numbers do not paint a clearer picture. For instance, in IWSLT and WMT indeed there is fairly consistent evidence that FFN Reservoir is more efficient to train than fully trainable transformers. However, for enwik8 the T Reservoir outperforms everything significantly when looking at figure 15 but judging from figure 13 we see that the best case scenario has been selected for T Reservoir (namely 32 or 48 layers). Even more importantly perhaps, the story is completely different when looking at the test set evaluation where FFN Reservoirs perform better and actually the T Reservoir performs the worst among all methods. Similarly on RoBERTa pretraining, fully trainable transformers seem to achieve the lowest validation perplexity and with the highest efficiency.\n\nTo summarize, I believe that the reservoir transformers could be a useful tool for improving either the efficiency or the generalization of transformers in low-data regimes or both. However, a distillation of the experiments and conclusions is required in order for this to be shown from the paper. Namely, even with all those numbers I still cannot judge which reservoir layer will be better, in what sense it will be better (accuracy or efficiency) and why is it going to be better.\n\nDue to the above, I tend to keep my score but because the additions of the authors provide significantly more information (and in my opinion value) to the paper, I will increase my score to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "js7i-QUGGlR", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. The authors have experiments on four types of reservoir: Transformer Reservoir , FFN (feed-forward layer) Reservoir, BiGRU Reservoir and CNN Reservoir. And the results show that the Reservoir can achieve competitive/better performance or than normal Transformer on machine translation, language modeling, and MLM pre-training. As pointed out by the authors,  deep reservoir computing networks (Scardapane & Wang, 2017; Gallicchio & Micheli, 2017) have been explored before. The main novelty of this paper is only the Reservoir exploration of Transformer structure. Although the method is interesting and the experiments are well-done, the work \"REDUCING TRANSFORMER DEPTH ON DEMAND WITH STRUCTURED DROPOUT\" seems more straightforward and effective. The authors should have some comparisons with this method. As there's still residual connections, how about simply adding some noise to the previous layer? Overall, although the experiments are interesting, it doesn't provide novel theory on it and misses some comparison to previous works. It's hard to evaluate the significance of this work.\n\nPro:\n1. The reservoir on Transformer is interesting and has not been explored before.\n2. The authors prove the idea by many different tasks.\n\nCons:\n1. The reservoir operation was explored on other structures.\n2. No strong baseline provided, such as \"REDUCING TRANSFORMER DEPTH ON DEMAND WITH STRUCTURED DROPOUT\" or some other structure pruning based methods.\n3. No novel theory to explain the method.\n\n\n######update\n\nI like the experiment added in the revision. However, it is only tested on a IWSLT which is a smaller dataset and can be influence by many hyper-parameters. It's not clear what frozen layers mean for LayerDrop and it need to be clarified with more details. I didn't find clear comparison with stronger baselines on WMT in the revision.\n\nAs pointed our by the authors, I think the theory mainly come from previous works. \n\nI have also read other reviews. Overall, I would like to keep my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7Ic7sYcpjmC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores the concept of randomization in Transformer architectures. Essentially, some of the layers in the encoder network are replaced by fixed layers, which makes the computation faster straightforwardly in the prediction phase. The paper also illustrates a method, called backskipping, to reduce the cost of backpropagating gradients through the fixed layers. The proposal is well supported by a number of experiments in the area of Machine Translation.\nPROs:\n- I find the paper very clear and well written & readable. I enjoyed reading it.\n- The research proposed in the paper seems novel enough to me, and goes towards a fruitful research direction, that one of randomized neural algorithms (although this is nowadays established in the ML community).\n- The experiments offer a nice perspective on the advantages of the proposed model\n\nCONs:\n- I find inappropriate the use of the term \"reservoir\" in this work. In my understanding, that is essentially a fixed (untrained) recurrent layer. The aspect of the dynamics is fundamental in this respect. In the paper, instead, the use of \"reservoir\" stands mainly for \"randomized\". All in all, I don't see reservoir layers in the proposed model, and I suggest to take this into account (perhaps changing the name and the title)\n- The experiments show that using untrained layers gives a positive trade-off between accuracy and compute times. In the experimental comparison, I think it would - however - important to see how this trade-off behaves/scales wrt to the number of trainable weights (when avoiding training in some of the layers the number of trainable weights is reduced, and the overall complexity of the system is reduced too, having a sort of regularization effect).\n- The experiments show that it is possible to avoid training in some of the layers of a Transformer. However, in randomized neural networks, a pivotal role is played by the scaling parameters of the involved weight matrices. Orthogonal matrices are used in the experiments, which is good, but I wonder how (and if) the scaling of such matrices affect the performance. Ideally such scaling should be hyper-parameters and be chosen on a validation set.\n\na few minor points:\n- please, if possible, clarify more explicitly on the concept of \"converging faster\"\n- please, clarify on the data splitting (in tr/vl/ts) for the used datasets\n- please, introduce the datasets before section 3.1 (where they are already mentioned without intro).\n\n-- EDIT:\nI would like to thank the authors for the nice work during the review process. I am pretty satisfied with that and I feel serene to increase my rating to acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were split between accept (7) and borderline reject (two 5's). All three reviewers acknowledged that the proposed approach is simple and intuitive (but this paper follows, for the most part, the concept of reservoir operation and apply it to transformers). The main criticisms were insufficient experiments (R5) and the lack of a clear conclusion (R2). I found these concerns to be valid and did not find strong reasons to overturn their recommendations. More comprehensive experiments (especially on WMT) and clear conclusions (accuracy or efficiency) would make this paper much stronger.
Decision: Reject