Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
Authors: Subhabrata Dutta
Tanya Gautam
Soumen Chakrabarti
Tanmoy Chakraborty
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Transformers
Dynamical systems
Time-evolution
Self-attention

Abstract: The Transformer and its variants have been proven to be efficient sequence learners in many different domains. Despite their staggering success, a critical issue has been the enormous number of parameters that must be trained (ranging from $10^7$ to $10^{11}$) along with the quadratic complexity of dot-product attention. In this work, we investigate the problem of approximating the two central components of the Transformer --- multi-head self-attention and point-wise feed-forward transformation, with reduced parameter space and computational complexity. We build upon recent developments in analyzing deep neural networks as numerical solvers of ordinary differential equations. Taking advantage of an analogy between Transformer stages and the evolution of a dynamical system of multiple interacting particles, we formulate a temporal evolution scheme, \name, to bypass costly dot-product attention over multiple stacked layers.  We perform exhaustive experiments with \name\ on well-known encoder-decoder as well as encoder-only tasks. We observe that the degree of approximation (or inversely, the degree of parameter reduction) has different effects on the performance, depending on the task. While in the encoder-decoder regime, \name\ delivers performances comparable to the original Transformer, in encoder-only tasks it consistently outperforms Transformer along with several subsequent variants.

Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
Subhabrata Dutta
Tanya Gautam
Soumen Chakrabarti IIT-Bombay India
Jadavpur University
IIT--Dllhi India
India subha0009@gmail.oo com
(anya18048Gi1itd.aad
soumen@cae iittendceeiitb..... ac in
Tanmoy Chakraborty
IiIT-Delhi
India
tanmoy@iiitd.as ac.
Abstract
The Transformer and its variants have been proven to be efficient sequence learners in many different domains. Despite their staggering success, critical issue has been the enormous number of parameters that must be trained (ranging from 107 to 10th) along with the quadratic complexity of dot-product attention. this work, we investigate the problem of approximating the two central components of the Transformer - multi-head self-attention and point-wise tion, with reduced parameter space and computational complexity. We build upon recent developments in analyzing deep neural networks as numerical solvers of ordinary differential equations. Taking advantage of an analogy between Trans- former stages and the evolution of a dynamical system of multiple interacting particles, we formulate temporal evolution scheme, TransEvolve, bypass costly dot-product attention over multiple stacked layers. We perform exhaus- tive experiments with TransEvolve on well-known encoder-decoder as well as encoder-only tasks. We observe that the degree of approximation (or inversely, the degree of parameter reduction) has different effects on the performance, de- pending the task. While in the encoder-decoder regime, TransEvo llve delivers performances comparable to the original Transformer, encoder-only tasks it con- sistently outperforms Transformer along with several subsequent variants. Code is available in: https::/ /sttss://gtthb. www.ccccc ---GSS-IIIII TTW.ID///TanEEveeee..
transforma-
Introduction
Neural networks have evolved from early feed-forward and convolutional networks, recurrent
networks, to very deep and wide "Transformer' networks based on attention Vaswanii et al. 2017
Transformers and their enhancements, such as BERT [Devlin et al. 2019 T5 Raffel et al. 2020
and GPT [Brown et al. 2020) are, by now, the default choice in many language applications. Both their training data and model sizes are massive, BERT-base has 110 million parameters. BERT-large, which often leads to better task performance, has 345 million parameters. GPT-3 has 175 billion trained parameters. Larger BERT models already approach the limits of smaller GPUs. GPT-3 is outside the resource capabilities of most research groups. Training these gargantuan models is even more challenging, with significant energy requirements and carbon emissions [Strubell et al. 2020).
In response, growing community of researchers is focusing post-facto reduction of model sizes, which can help with the deployment of pre-trained models low-resource environments. However, training complexity is also critically important. A promising recent approach to faster training uses
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

way of viewing layers of attention as solving ordinary differential equations (ODEs) defined over dynamical system of interacting particles [Lu et al. 2019 Vuckovic et al. 2020 We pursue that line of work.
Simulating particle interactions over time has a correspondence to 'executing" successive layers of the Transformer network. In the forward pass at successive layers, the self-attention and position-wist feed-forward operations of Transformer correspond to computing the new particle states from the previous ones. However, the numeric function learned by the i-th attention layer has zero knowledge regarding the one learned by the (i 1)-th layer. This is counter-intuittee whole evolution is temporal in nature, and this independence leads to growing numbers of trainable parameters and computing steps. We seek to develop time-evolution functionals from the initial condition alone. Such maps can then approximate the underlying ODE from parametric functions of time (the analog of network depth) and do not require computing self-attention over and over.
due to the fact that the
We propose such a scheme, leading to a network/method we call TransEvolve. It can be used for both encoder-decoder and encoder-only applications. We experiment on several tasks: neural machine translation, whole-sequence classification, and long sequence analysis with different degrees of time- evolution. TransEvolve outperforms Transformer base model on WMT 2014 English-to-Frencts translation by 1.4 BLEU score while using 10% fewer trainable parameters. On all the encoder-only tasks. TransEvolve outperforms Transformer, as well as several strong baselines, with 50% fewer trainable parameters and more than 3 training speedup.
2
Related Work
Our work focuses two primary areas of machine learning understanding neural networks as dynamical systems and bringing down the overhead of Transformer- based models in terms of training computation and parameters.
Neural networks and dynamical systems. Weinan 2017] first proposed that machine learning
systems can be viewed as modeling ordinary differential equations describing dynamical systems.
Chang et al. [2018) explored this perspective to analyze deep residual networks. |Ruthotto and Haber [2019] later extended this idea with partial differential equations. [u et al.) [2018] showed that any parametric ODE solver can be conceptualized as a deep learning framework with infinite depth. Chen let al. 2018 achieved ResNet-comparable results with drastically lower number of parameters and memory complexity by parameterizing hidden layer derivatives and using ODE solvers. Many previous approaches applied sophisticated numerical methods for ODE approximation to build better
neural networks [Haber and Ruthotto 2017, Zhu and Fu 2018 Vuckovic et al. [2020) developed a mathematical formulation of self-attention as multiple interacting particles using a measure-theoretic perspective. The very first attempt to draw analogies between Transformers and dynamical systems was made by Lu et al. [201]] They conceptualized Transformer as a numerical approximation of dynamical systems of interacting particles. However, they focused on a better approximation of the ODE with a more robust splitting scheme (with the same model size as Transformer and the dot-pre product attention kept intact). We seek to parameterize the temporal dynamics to bypass attention up to certain degree.
Efficient variations of Transformer. Multiple approaches have been put forward to overcome the
quadratic complexity of Transformers Wang et al. 2020a
Choromanski et al.) 2020 Peng et al.
2021 Xiong et al. 2021 Liu et al., 2018). Kitaev et al. [0200] sought to use locality-sensitive
hashing and reversible residual connections deal with long range inputs. Some studies explored
the sparsified attention operations to decrease computation cost [Liu et al. 2018 Ho et al.
2019 Roy
al. 2021]. These sparsification tricks can be done based on the data relevant to the task [Roy et al.
2021 Sukhbaatar et al. [219)] or in a generic manner [Liu et al. 2018 Ho et al. [019). Wang et al.
2020al observed sell-attention to be low-rank, and approximated an SVD decomposition of attention
to linearize it. |Peng et al. [2021) used random feature kernels to approximate the softmax operation on the attention matrix. Chhromanskii et al.) (2020) sought to linearize Transformers without any prior assumption of low-rank or sparse distribution, using positive orthogonal random features. [Le-Thorp jet al. (022T) achieved remarkable speedup over Transformers by substituting the attention operation with an unparameterized Fourier transform. A bulk of these works are suitable for encoder-only tasks and often incurs slower training (e.g, Peng et al. (202T) observed 15% slower training compared to Transformer). Our method overcomes both of these drawbacks.

---

Table 1: List of important notations and their denotations used.
Notation
Denotation
d.d
Hidden and temporal dimension of the model Sequence of d-dimensional vectors input to the 1-th encoder block
X
T
d'-dimensional map of depth (time) I
H
Output of softmax attention at (-th encoder block
Wg. Wi
Query and key projection matrices
W W
Temporal query and key projection matrices
W.
Attention output projection matrix Query-key dot-product from initial values Time-evolution operators for attention Random rotation matrices at depth I
Ao
A A2, A3 UA, Vi
Transformer pruning and compression, Knowledge distillation has been used to build liggtt student model from large, trained Transformer models (Behnke and Heafelld [2020] |aanh et al. 2019, Wang et al. 2020b]. J. |Michel et al. [2019) experimented with pruning different attention heads of BERT to observe redundancy in computation. However, these methods still require a trained, parameter-heavy model to start with. Tensorization approach has shown efficient compression of Transformer-basee language models [Ma et al. 2019
Khrulkov et al. 2019
Transformers as Dynamical Systems
single block of the Transformer encoder (Vaswani et al. 2017 is defined as a multi-head self- attention layer followed by two feed-forward layers, along with residual connections. The j-th head of self-attention operation in the 1-th encoder block, with j E (1,..., m). on a given length-n sequence of d dimensional input vectors X = X E Rdin Jiz1 can be defined as:
J = Softmax; (X W (X
where W W E Rddxd, and W E Rdxds are linear projection layers and dN = d/m. Conven- tionally, dy de. Each HJ is aggregated to produce the output of the multi-head self-attention as follows:
H' Concat ( (H)),,) X
where Wo E Rddx is linear projection. The subsequent forwadd transformation can be defined as:
x (HH'W) in + B)r1)) + 12 + H'' H
where W E Rdddd W 12 E Rddrrxd BBn E Rd# B1yy2 Rd, and a() is a non- non-linearity (Relu Vaswani et al. 2017] or Gelu in Devlin et al. 2019))
As Lu et al. [2099] argued, Equations 13 bear a striking resemblance with systems of interacting
particles. Given the positions of a set of interacting particles as x(1) (x; the temporal
evolution of such system is denoted by the following ODE:
d (1) ((x)())x((1))) + G(r:(1, dt
with the initial condition (t,()) 8, E R.d. The functions F and G are often called the diffusion and convection functions, respectively - the former models the inter-dependencies between the particles time while the latter models the independent dynamics of each particle. Analytical solution of such an ODE is often impossible to find, and the most common approach is to use numerical approximation over discrete time intervals [to to + ôt.... .t.....ty + Lôt). Following Euler's method of first order approximation and the Lie- Trotter splitting scheme, one can approach the numerical solution of Equation from time to + öt as:
(t) = (t) + (t) = x1(1) + F(z.(1)x(1),,)) (t 8t) (t) (tG(E 2,(t) (t),(t))
Equation 5 can be directly mapped to the operations of the Transformer encoder given Equations
with X x(t). H {t) and x x(t ôt). F(.,,t) is instantiated by the
3

---

Figures a [b) summarize the procedure. Given two input sequences X and Y (in case of self- attention, they are the same), we first compute Ao, A1, A2, A3 using the linear projection matrices where m projections W W W. W and Softmax(() operations in the multi-head self-attention. G t) corresponds to the projections W7 y1 W7 W B1 B1 and the a(-) non-linearity in Equation 3
From the described analogies, it quickly follows that successive multi-head self-attention and the point-wise feed-forward operations follow a temporal evolution (here time equivalent the depth of the encoder). However, Transformer and its subsequent variants parameterize these two functions in each layer separately. This leads to a large number of parameters be trained (ranging from 7 x 10" in neural machine translation tasks to 175 x 10 in language models like GPT-3). We proceed to investigate how one can leverage the temporal evolution of the diffusion and convection components to bypass this computational bottleneck.
Time-evolving Attention
As Equation computes x(t) iteratively from a given initial condition x(to) = 8 = {s:][= one can reformulate the diffusion map. F(,, x(t),t) as J's. ()), i.e., as functional of the initial condition and time only. When translated to the case of Transformers, this means one can avoid computing pairwise dot-product between n input vectors at each layer by computing a functional form at the beginning and evolving it in temporal (depth-wise) manner. We derive this by applying dot-product self-attention on hypothetical input vectors with augmented depth information, as follows.
f(s
Let X' = X E Rd+d"} } be a set of vectors such that X = Concat X T'), where X { 3 and T = (4) Tar (1)) WG. Wa E x((+a") are the augmented query and dd''', [
W = 1051 d''' The pre-softmax query-key dot product between X! and X;, is given by dly = (X
key projections, respectively, such that W =
We can decompose d++d' addd'
W/ as concatenation of two matrices Wg, W, such that W, [wyskks_1] and W,, = [ubli,j=d1111
Similarly, we decompose W into W4 and We. Then d'y can be re-written as:
d/g (X?W (X,WW)) + WT) + -(TWW)(X3WW) (TWW))("W (6)
= als + T + +T A2j + Az の T
where A.; X.W, W.T. A2j W. W. XºT As W.W. and @ signify hadamard product.
Detailed derivation is provided in Appendix[A
It is evident that al is the usual dot-product pre-softmax attention between vector elements X². x9. For the complete sequence of vector elements X, we write Ao [aszlej. A₁ {A1s); and A = {A2j); j-- By definition, T is vector function of the depth 1. To construct T as vector function of Z, we formulate
d'l w' cos( 1 p). ...... ................................ w cos ++1
d'l 2P
r = w/ sin P wf sin
2P
where W {wo] j learnable parameters depth 1, and P d'L Such choice of T 2n
intuitive in the following sense: let AT B for some arbitrary A [ary] E Roxd B = {b,} E R?, and C = [ct} E RR. Then we get
jl a.j J+ p
C b, + Σ aay sin
>
cos
In other words, any fced-forwarts transformation on T' gives us a vector in which each component is Fourier series (thereby, approximating arbitrary periodic functions of I with period P). This enables us to encode the nonlinear transformations that X undergoes in successive blocks. With this, A1, A2, As constitute the time-evolution operators map depth information the attention, computed only from the initial conditions X X9.
Wg. Ws, b We, We, as described above. Additionally, we normalize Ao by a factor Vaym
the number of heads, similar to Transformer. Then, at any subsequent layer I (l...., L). instead

---

(b) Initial attention
(a) Time-evolving encoder block of depth L
(c) Timo-evolving encoder at depth
computation
X
Y (In case of self-attention,
Ao
A
A2
A3
X
X = Y)
XY
X
Y
T
Initial attention
W
W
W
W
Ao A2
Softmax()
Encoder at depth=1 1
W.
0
A
A3
X
H
U1E, V
Dot-product
X
RcLU()
Element-wist
Encoder at depth=L
U3 V/2
addition
X
Figure I: Dissecting the primary functional components of TransEvolve. (a) An L-depth encoder block starts with computing (b) the initial condition matrix Ao and the evolution operator matrices A1, A2, Ag from the input sequence. These four are then used in (c) each encoder at depth along with vector function of depth, T apply the attention operation the output from the previous step. This product of attention is then passed to the feed-forward transformation actuated by the depth- dependent, random rotation matrices U/, U2, Vl, V BrrasEvvovve-rndoom see Section 6). In another variation, we use learnable feed-forward layers (TransEvolve-fullfe
of computing the query, key and value projections from the input X we use the previously computed Ao, AI, A2, Ag along with T', Then the time-evolved attention operation becomes
H' Softmax(A, A - T' + As(T' @ T')xx'
(9)
For the sake of brevity, we have not shown the concatenation of multi-headed splits as shown in Equation 2 Therefore, one should identify W in Equation 9 with Equation and not as value projection as in Equation[]] We do not use value projection in our method. Also, in the multi-headed case, all the matrix-vector dot products shown in Equation 9 are computed on head-splitted dimension of size d/m.
Complexity and parameter reduction. Given a d-dimensional input sequence of length n,, com- puting the pre-softmax attention weight matrix in a single Transformer encoder requires O(n d) multiplications, In our proposed method, the complexity of calculating dot- dot-product attention (corre- sponding a in Equation 6 invokes computations of similar order Onn (d + d')). However, this is needed only once at the beginning. The subsequent attention matrices are calculated using the components with in Equation 6 Both A and TA, require O(nd") computation, and A₃T requires only (d'). Therefore, if one tends to use L successive attention operations, our proposed method computationally equivalent to a single original dot-product self-attention followed by multiple cheaper stages. In addition to this, attention weight computation using Equation Eguation[@eliiinatee elliminttes the need for query, key, and value projections each self-attention block. Thereby, ensures parameter reduction of (Ld)) for total stacking depth of L.
5

---

5 Time-evolving Feed-forward
The Transformer counterpart of the convection function ((,,t) is the point-wise feed-forward trans- formation Equation B The complete operation constitutes two projection operations: first, the input vectors are mapped to a higher dimensional space (R" - R along with a non-linear transformation, followed by projecting them back to their original dimensionality (R41, + Rd). At the (-th Transformer encoder, these dimensionality transformations are achieved by the matrices W' and W W respectively (see Equation 3). To construct their temporal evolution, we attempt to decompose them into time-evolving components.
Recall that any real m x n matrix M can be decomposed as M = UEVT where U E Rmxm, V E R2x= orthogonal, and Σ R=*" is rectangular diagonal matrix. However, computing exact orthogonal matrices with large dimensionality is computationally infeasible. Instead, we construct approximate rotation matrices U E Rdxd as:
sinnw
sin(w
di 14 2P cos(wi1
cos(w (wt di 1 1$2P)
U
(10)
Vd
sin(waa ))
sin(w
d 2P
cos(w /)
cos(w
dl dg
and way E (O d.). We discuss the properties of such a matrix U in the Appendix B where p= 2n
We construct four matrices Uf E Rdxd Vi E xdrr, U. E R and V E Rdxd as described in Equation 10 Also, we construct two rectangular diagonal matrices Σ, E Rddxdys and Dg E 1R xd with learnable diagonal entries. With these matrices defined, one reformulate the point-wise reed-forward operation (Equation 3 as:
x = V/ + B1) B2
(11)
This reformulation reduces the number of trainable parameters from {(ddss) Odd).
6 Proposed Model: Tvolve
Equipped with the particle evolution based definitions of attention and point-wise feed-forward operations, we proceed to define the combined architecture TransEvolve. The primary building blocks of our model are the initial attention computations following Equation 6 shown in Figure [ and attention operation at depth followed by fced-forward transformations (shown together as the encoder at depth / in Figure c)). An L-depth encoder block of TransEvolve, as shown in Figure [a), consists of number of attention and feed-forward operations stacked successively, preceded by initial attention computation.
Variations in temporal evolution of feed-forward operation. While the re- parameterization of the point-wise feed-forward operations described in Section 3 seems provide astonishing reduction in the parameter size, it is natural to allow different degrees of trainable parameters for these operations. This allows to explore the effects of time-evolving approximations of attention and point-wise feed-forward separately. We design two variations of TransEvolve. In the original setting, the point-wise feed-forward operation is applied using random rotation matrices, following Equation T1 We call this BransBvolve-random ve-randomFF. In the other variation, the time evolution process is applied for the attention operations only while the feed-forward operations are kept be the same as Transformer (Equation 3b. This variation is denoted as
henceforth.
Different degrees of temporal evolution. Recall from Section 3 that Equation 5 a numerical ap- proximation of Equation 4 so as to evaluate x(L+ōl) iteratively from x(D). When we attempt to approx- (() fxx(t), +)) expected grow as tol grows. However, it not feasible to compute the exact approximation error due to the complex dynamics which, in turn, varies from task to task. To compare with the origi- nal Transformer, we seek to explore this empirically. More precisely, the L-depth encoder (decoder) block with our proposed evolution strategy is expected to approximate L encoder (decoder) layers of Transformer. Following the approximation error argument, TransEvolve is likely to bifurcate more from Transformer as L, gets larger. We experiment with multiple values of I. while keeping the total number of attention and feed-forward operations fixed (same as Transformer in the comparative task). To illustrate. for WMT Beglish-to-Germal translation task, Transformer uses 6 encoder blocks
imate F(z (t) x(t) t) as f fx((t)) t)). the error [F(x, x(t),

---

followed by 6 decoder blocks. Converted in our setting, one can use 1 encoder (decoder) each with depth L 6 encoders (decoders) each with depth 3 (given that the latter requires more parameters compared to the former).
Encoder-decoder. To help compare with the original Transformer, we keep our design choices similar to Vaswani et al. [2017]: embedding layer tied to the output logit layer, sinusoidal position embeddings, layer normalization, etc. The temporal evolution of self-attention described in Section can be straightforwardly extended to encoder-decoder attention. Given the decoder input X and the encoder output Xenc. we compute the initial dot-product attention matrices Age and A corresponding to decoder self-attention and encoder-decoder attention, follows:
dec (Xaee (x A
A = (x Wed (Xenc W
(12)
If X dec and Xene are sequences of length Nider and Penn. respectively, then Age and Ag are Odee Vadeo and Nene, Ridee matrices, respectively. We also compute the corresponding time- evolution operators A Agec .Ageee and A A² AA similar to Equation Then at each depth / in the decoder block, the time-evolved attention operations are as follows:
W:ldec + X H' = Softmax( Ao dec + A dec + doc A2 dec + Ag (Tácc O (dce)) Weo
(13)
H' Softmax(A
Ardri Te + A3 A5 3 ) Wheee H
where T and Thd are two independent vector maps of depth / representing the time-evolutions of the decoder self-attention and the encoder-decoder attention. For the sake of brevity, we have shown the full multi-head attentions in Equations 12 and 13 without showing the concatenation operation (as in Equation 2)
Encoder-only. For many-to-one mapping tasks (e.g., text classification), we only use the encoder part of TransEvolve. The sequential representations returned from the encoder are applied with an average pooling along the sequence dimension and passed through normalization layer to the final feed-forward layer predict the classes.
7
Experiments
7.1 Model Configurations
We set = d' across all the variations of TransEvolve. For the encoder-decoder models, we experiment with the base version of TransEvolve with d 512. For encoder-only tasks, we use small version with d = 256. In both base and small versions, the number of heads is set to
As discussed in Section 6 one can vary the degree of temporal evolution (and the number of trainable parameters) in Tvolve by changing the value of L. the L-depth time-evolving encoder (decoder) blocks. To keep the total depth of the model constant, we need to change the number of these blocks as well. We design two such variations. Recall that the total depth of the Transformer encoder-decoded model is 12 (6 encoders and decoders); in TransEvolve, we choose (i) 1 encoder (decoder) block with depth (denoted as SonnsEvvvv--llFFF. and (ii) 2 encoder (decoder) blocks each with depth 3 (denoted by LaanssEvolv--anndofFFee etc.), This way, we obtain 4 variations of TransEvolve for our experiments.
etc.).
7.2 Tasks
We evaluate TransEvolve over three different areas of sequence learning from texts: (i) sequence- to-sequence mapping in terms of machine translation, (ii) sequence classification, and (iii) long sequence learning. The former task requires the encoder-decoder architecture while the latter two are encoder-only.
learning benchmark, we use WMT 2014 English-to- Ger-aan (En-De) and English-to-Frende (En-Fr) translation tasks. The training data for these two tasks contain about 4.5 and 35 million sentence pairs, respectively. For both these tasks, we use the base configuration of our proposed models. We report the performance of the En-De model on WMT 2013 and 2014 En-De test sets (newstest20133 and newstesr2014), En-Fr model is tested only WMT 2014 test set. Implementation details these tasks are described in the Appendix.
Machine Translation (MT). As a sequence-to-sequeee
Sequence classification. We evaluate the performance of the encoder-only version of our model on two text classification datasets: IMDB movie-review dataset [Maas et al. 2011 and AGnews topic
7

---

En- De
En-Fr
Table 2: Performance of TransEvolve variants on Seglish-to-Germaan (En-De) and English-to-Freme (En-Fr) translations in terms of BLEU scores. Transformer results are taken from the original paper (Vawwani et al. 2017
Model
WMT
WMT
WMT
#Params
2013
2014
2014
Transformer BASE
25.8
27.3
38.1
65M
TransEvolve- randomFF-
22.5
23.1
32.6
27M
TransEvolve randomFF-2
24.2
23.8
33.4
33M
TransEvolve- fullFF-1
25.3
25.8
38.0
53M
TransEvolve- fullFF-2
26.2
27.2
39.5
59M
classification dataset Zhang et al. for training and testing purposes. This is binary classification task (positive/negative reviews). The AGnews dataset is a collection of news articles categorized into 4 classes (Science & Technology, Sports, Business, and World), each with 30000 and 1900 instances for training and testing, respec- tively. For both these tasks, we use the small version of our model. Detailed configurations and hyperparameters are given in the Appendix.
2015]. The IMDB dataset consists of 25000 movie reviews each
Long sequence learning, To test the effectiveness of our model for handling long sequences, we use two sequence classification tasks: character- classification of the IMDB reviews and latent tree learning from sequences of arithmetic operations and operands with the ListOps dataset [Nangia and Bowman 2018). For the IMDB reviews,
the maximum number of tokens (characters) in the training and testing examples to 4000 following (Peng et al. 2021]. In the ListOps dataset, an input sequence of arithmetic operation symbols and digits in the range 0-9 given input; it is 10-way classification task of predicting the single-digit output of the input operation sequence. We consider sequences of length 500-2000 following (Peng al. 2021] Again, use the small version of TransEvolve for these two tasks. Further experimental details are provided in the Appendix.
7.3 Training and Testing Procedure
All experiments are done on v3-8 Cloud TPU chips. We use Adam optimizer with learning rate scheduled per gradient update step as = irmaz x satps 0.5 warmup_stepp x step). For 16000. For the remaining two tasks, these
vi va the MT task, we set [Tmoz = 1.5 and warmuppsley values are set to 0.5 and 8000, respectively. For the translation tasks, we use label smoothing coefficient 0.1. Training hyperparameters and other task-specific details described in the Appendix. For the translation tasks, we report the BLEU scores averaged from 10 last checkpoints, each saved per 2000 update steps. For encoder-only tasks, report the average best accuracy five separate runs with different random seeds. Comprehensive additional results are provided the Appendix.
8 Results and Discussion
Machine Translation. Table 2 summarizes the performance of TransEvolve variants against Transformer (base version) on English-German and English-Frende translation tasks. TransEvolve- randomFF versions perform poorly compared to fullFF versions.
Text classification accuracy of TransEvolve vari- ants on AGnews and IMDB dataset. Scores of Transformer, Linformer, and Synthesizer are taken from TTy 2020a
Table 3:
However, with less than 50% of the parameters used by Transform- ers, they achieve above 85% perfor- mance of that of Transformer. With all random rotation matrices replaced by standard feed-forward layers, Trans- Evolve with single layers deep en- coder (decoder) block performs com- parably to Transformer on the En- Fr dataset. Finally, with blocks of depth encoders, TransEvolve- fullFFF22 outperforms Transformer on WMT 2014 En-Fr dataset by 1.2 points BLEU score, despite having 10% lesser parameters. On the En-De translation task, this model performs comparable Transformer, with 0.5 gain and 0.1 drops BLEUU scores on WMT 2013 and 2014 test datasets, respectively.
Model
AGnews
IMDB
Transformer
88.8
81.3
Linformer [Wan al. 2020a
86.5
82.8
Synthesizer [Tay al. 2020a
89.1
84.6
TranaEvolve-randome
90.6
87.3
TransEvolve-randoofF17
90.8
87.5
TransEvolve-fullFF- TranaEvolve-fullFF-2
91.1
86.8
90.5
87.6
8

---

Table Accuracy (%) of TvaneEvolve on the long range sequence classification tasks. Speed is
measured w.r.t. Transformers on the character-level IMDB dataset for input sequences of length 1k, 2k, 3k and 4h: All results except Tvave variants are taken from Peng et al. 2021
Tasks
Speed
Models
ListOps
charlMDB
1k
2
3k
4
Transformer
36.4
64.3
1.0
1.0
1.0
1.0
Linformer Wang et al.
2020a
35.7
53.9
1.2
1.9
3.7
5.5
Reformer
Kıtacy ct al.
2020
37.3
56.1
0.5
0.4
0.7
0.8
Sinkhorn Tay al. 2020b
17.1
63.6
1.1
1.6
2.9
3.8
Synthesizer Tay aL.
2020a
37.0
61.7
1.1
1.2
2.9
1.4
Big Bird [Zaheer et al.
12020
36,0
64.0
0.9
0.8
Linear attention Katharopoolss
2020
65.9
1.1
3.7
Performers [Choromanski et al. 2020
18.0
65.4
1.2
1.9
3.8
5.7
Random Feature Attention (KFA) Peng al. 2021
36.8
66.0
1.1
1.7
3.4
5.3
TransEvolves F-.
43.2
65.3
1.2
1.3
1.2
1.2
FransSEvolve-randdoFF....
39.1
66.1
1.1
1.2
TransEvolve-fullFF. TransErolve-fullFF2
42.2
65.7
37.8
65.6
1.1
1.1
1.0
1.1
Text classification. Table summarizes the accuracy of TransEvolve on text classification tasks. It should be noted that these results not comparable to the state-of- of-th-o-- results these two datasets; all four models mentioned here use no pretrained word embeddings to initialize (which is essential to achieve benchmark results for these tasks, mostly due to the smaller sizes of these datasets) or extra data for training. These results provide a comparison of purely model-specific learning capabilities. Upon that, SannsEvoove-fllFFFF achieves the best performance both datasets. For topic classification on AGnews, it scores 91.1% accuracy, outperforming Synthesizer (the best baseline) by 2%. The improvements are even more substantial on IMDB. TransEvolve. fullFF-2 achieves an accuracy of 87.6% with improvements of 3% and 6.3% upon Synthesizer and Transformer, respectively.
Long sequence tasks. TransEvolve shows remarkable performance in the arena of long sequence learning as well. As shown in Table TransEvolve outperforms Transformer along with previous methods both ListOps and character-level sentiment classification IMDB reviews. On ListOps, wwweevvvveeeradndnn
establishes a new benchmark of 43.2% accuracy - beating Reformer (existing state-of-the-art) by 4.9% and Transformer by 5.8%. Moreover, all four versions of Trans- Evolve show a gain in accuracy compared to the previous models. is to be noted that we use the small version of TransEvolve (with 256 hidden size) for these tasks, while Transformer uses the base version (512). So all these improvements are achieved while using 25% of Transformer's parameter size. On the char-IMDB dataset, while only existing state-of-the-art, i.e., Random Feature Attention (RFA), other variants of TransEvol ve also turn out be highly competitive. TransEvolve ve-.dvee--a-FFaaaaa achieves 66.1% accuracy, improving upon RFA by 0.1% and Transformer by 1.8%. While all the variants of TransEvolve run faster compared Transformer, they do not show any additional speed-up for longer sequences like RFA or Performers. This behavior is reasonable given the fact that TransEvolve still performs softmax on (n²) sized attention matrix each depth. Moreover, the speedups reported Table are with the same batch size for Transformer. Practically. the lightweight TransEvol Eveev--oleeeeeee ve-eraddomP models can handle much larger batch size (along with larger learning rates). This results in a more than 3x training speedup for all the lengths compared to Transformer. The relative gain in speed compared to Transformer or Reformer is achieved due to linear computation of pre-softmax weights (except for the initial attention calculation) and a reduced number of parameters. This is also supported by the fact that randomFF versions run faster than fullFF ones, and speed decreases with the increased number of shallower encoder blocks.
outperforms the
Effects of model variations. Random rotation matrices, instead of standard ferw-forward layers and varying the number (conversely, the depth) of TransEvolve encoder blocks, show task deseddepnndent effect on performance. TransEvolve versions with single 6-layer deep encoder perform better than successive 3-layer deep encoders in the case of ListOps. other encoder-only tasks, there are no clear winners among the two. Similar patterns can be observed among SoanssEoove--rnddowe and TransEvolve. FulllF..... In the case of machine translation though, straightforward that having lesser number of feed-forwan parameters (randomFF vs. fullFF) and/or fewer encoder blocks

---

with deeper evolution deteriorate the performance. In general, the performance of TransEvolve variations degrades with the decrease in the total parameter size on translation tasks. Encoder-only models use only self-attention, while the encoder-decoder models use self, cross, and causal attentions. From the viewpoint of multi-particle dynamical systems, the latter two are somewhat different from the former one. In self-attention, each of the 'particles' is interacting with each other simultaneously at a specific time-step. However, in causal attention, the interactions are ordered based on the token positions even within a specific timestep. So while decoding, there is an additional evolution of token representations at each step. Moreover, in encoder-decoder tasks as well, TransEvolve outperforms the original Transformer in two datasets (En-De 2013 and En-Fr 2014) clearly, while providing comparable performance in another (En-De 2014). It is the random matrix versions that suffered the most in these versions. The way designed the random matrix feedforward transformations incorporates the depth information via evolution while reducing the parameter size. In encoder-only tasks, this evolution scheme looks comparable to depth-indeppnndet,, parameter-dense full versions in expressive power. Moreover, each of the tasks presents different learning requirements. Intuitively, ListOps character-level sentiment classification tasks have smaller input vocabulary (hence, fewer amounts of information to encode in each embedding), but longer intra-token dependency (resulting in need for powerful attention mechanism) compared to sentiment or topic classification the word level. Random matrix versions provide depth-wise information sharing that may facilitate the model to better encode complex long-range dependencies, but they might remain under-parann paremettraaeeeeee to transform information-rice hidden representations. This complexity trade-off can be the possible reason behind randomFF versions, outperforming fullFF both the long-range tasks while vice versa in text classification tasks.
These variations indicate that TransEvolve is not just a compressed approximation of Transformer. Particularly in the case of encoder-only tasks, the depth-wise evolution of self-attention and feed- forward projection helps TransEvolve ve learn more useful representations of the input with much fewer number of trainable parameters. This also suggests that the nature of the dynamical system underlying a sequence learning problem differs heavily with different tasks. For example, the staggering performance of TransEvolve veansssolve-randdmFP.. 1 on the ListOps dataset implies that the diffusion component of the underlying ODE is more dominant over the convection component, and the error (t), x(t), f)- (t) fx((o)) t)) remain small with increasing |t tol. One may even conjecture whether the Hölder coefficients of the functions F' and G (in Equation 4 underlying the learning problem govern the performance difference. However, we leave this for future exploration.
9 Conclusion
Transformer stacks provide a powerful neural paradigm that gives state-of-the-art prediction quality, but they come with heavy computational requirements and large models. Drawing on an analogy between representation evolution through successive Transformer layers and dynamic particle inter- actions through time, we use numerical ODE solution techniques to design computational shortcut to Transformer layers. This not only lets us save trainable parameters and training time complexity, but can also improve output quality in a variety of sequence processing tasks. Apart from build- ing perterrperrorning model, this novel perspective carries the potential to uncover an in-depth understanding of sequence learning in general.
Limitations. It is to be noted that TransEvolve does not get rid of quadratic operations completely, as in the case of linear attention models like Linformer or Performer. It still performs costly softmax operations on quadratic matrices. Also, at least for once (from the initial conditions) TransEvolve computes pre-softmax Onn2) dot-product matrices. However, the significant reduction in model size compensates for this pre-existing overhead. Also, the speedup achieved by TransEvolve is majorly relevant in the training part and many-to-on mapping. In the case of autoregressive decoding, Trans- Evolve does not provide much gain over the Transformer compared to the linear versions (Peng et al.) 2021). Since the linearization approaches of the attention operation usually seeks to approximate the pair-wise attention kernel k(,r, y) as some (possibly random) feature map (x)))()), one may seek to design temporal evolution of the kernel by allowing temporally evolving feature maps. We leave this as potential future work.
10

---

Acknowledgement
The authors would like to acknowledge the support of Tensorfow Research Cloud. for generously providing the TPU access and the Google Cloud Platform for awarding GCP credits. T. Chakraborty would like to acknowledge the support of Ramanujan Fellowship, CAI, IIT--eelii and iitd Foundation set up under the NM-ICPS scheme of the Department of Science and Technology, India. S. Chakrabarti is partly supported by a Jagadish Bose Fellowship and grant from IBM.
References
Maximiliana Behnke and Kenneth Heafield. Losing heads in the lottery: Pruning transformer attention in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2674, Online, November 2020. URI https
Association for Computational I. Linguistics. doi: 11883//1112200mmmmffpp
220paogggy222000 em emnlp-manaa.... main. 211
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ily Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: ptttpp://pproceddnngg wwurips.cc/papeeeeee (ataapper/220//ahhh Abb2224444-Abttaatt.htt.a.
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL http::ttp//// httt:ttt/tpoenreeeet //opnrevee/ee
Sttp://fommiid-yJS---000 0gR-
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Gareett, editors, Advances in Neural Information Processing Systems
31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 6572-6583, 2018. URL.tttps: http:::sss/p/ceeeddnneeo cc/ cc/paper/2018/
httpr///21//hhhhaaaeeaa hash/ (1116//0000000066222222260666660
html
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. CoRR, abs/2009.114944 2020.
URL https::/arxiv..... wtt./////////..///e///////////ee 14794
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingnistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171- 4186. Association for Computational Linguistics, 2019. doi: 111865/v1/n19-4444 URL http://dd...rg/10111166/////////
1423
Eldad Haber and Lars Ruthotto Stable architectures for deep neural networks. CoRR, abs/ Abb///705.0341, 2017. /ttLLtttp//axxii.oro////ee
03341
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimen- sional transformers. CoRR, abs/19 12180, 2019. URL http:://///axii http:///axxxvvvoo/bbbaa wtrg/abs//990 1912, 12180
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119
11

---

of Proceedings of Machine Learning Research, pages 1156-5165. PMLR, 2020. URL http:
//proceedings.om .lr. ppees//u118//atlarrpoolllo// html
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan V. Oseledets. Tensorized
embedding layers for efficient model compression. CoRR, abs/1901.10787,, 2019. URL. h
7/arxiv. org/ www.aaaa./a. 10787
Nikita Kitaev, Lukasz Kaiser, and Anselm Lovskaya. Reformer: The efficient transformer. In 8th Inter- national Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL BtR.lattpss//opeerreee/ wttp:/ooummmddddxggi.eteee..tee
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. Fnet: Mixing tokens with fourier transforms. CoRR, abs/2105 033844, 2021. URL. httL'ttts:://xxiv..oom
2105. 03824
Peter J. Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: /tttere//opeervview net/ forum?id HygovbWWC
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Leurning Reseurch, pages 3276-3285 PMLR, 10-15 Jul 2018. URL
http://procceediggs mlr. ppress/v0// 1w100//u1dd..... html
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong. Tao Qin, Liwei Wang, and Tie- Van Liu. Understanding and improving transformer from multi-particle dynamic system point of view. CoRR, Abs/1900.02762, 2019. URL http:t/////:iii org/ abs/ 1906. 02762
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Aché-Buuc, Emily B. Fox, and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pages
2019. URL https: /ttp//eeeocc.....pp/eepp neurips cc/ cc/cccccppaoo (paper/209//haen Abstract html
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 201 1. Association for Computational Linguisties. URL. http //www.aclweb. acwweb. org/ www.wnnthhhago.. PP11:s/P111100111
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché-BBuc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, neurips cttpccccppp///////
BC, Canada, pages 14014-14024, 2019. URL httetts://proceeeee 2tt.b/0cc0022220000005556666 Abstract.htm.
Nikita Nangia and Samuel R. Bowman. Listops: A diagnostic dataset for latent tree learning. In Silvio Ricardo Cordeiro, Shereen Oraby, Umashanthi Pavalanathan, and Kyeongmin Rim, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 2-4, 2018, Student Research Workshop, pages 92-99. Association for Computational Linguistics, 2018. doi: 10.18653/v1/n18-4000 URL https: 1ttp://0.11663///1111101001
10. www.18665vvii11
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. Random feature attention. CoRR, abs/2103.02145, 2021. URL. https:://////.. hopps://aree /ttp///axvvv././///// 02143
www.wi.wrg.................ooole www.oogg//........ooooo..o... hts//bb//100 2103
12

---

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn, Res., 21:140:1-140:61 2020. URL http: v21/20-074.h2
org/ papers/
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Trans. Assoc. Comput. Linguistics, 9:53-68, 68, 2021. URL https://trannee org/o jtg/ index.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 1-13, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL httpp///////a
boggabs/19910 01008.01188
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for modern deep learning research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34 (09), pages 13693-13696, 2020.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive atten- tion span in transformers. tors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 331- 335. Association for Computational Linguistics, 2019. doi: 10.18653/11/19--0333 URL https://doi. org/10. 11w11//1///1---// -132
Anna Korhonen, David R. Traum, and Lluís Mårquez, edi-
Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models. CoRR, abs/2005.00743,, 2020a. URL https
//arxiv org/ abs 2005./200..000 00743
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse Sinkhorn attention. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9438-9447. PMLR, 13-18 Jul 2020b. URL http::///ppcreeediigg mml.iiinnnnmm.......mmmmm..m...m wttp://iinnn......./ee/eiiieaa.e wwt.spere//////yyy00000 html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:: httpss::///occee//eee Paper. pdf
neurips.cc/papeete. httpp///2007//1ee/fff22222222662
James Vuckovic, Aristide Baratin, and Remi Tachet des Combes. A mathematical theory of attention, 2020.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, 0bss22006004766 2020a. URL httt::://axxtt:: wwweexiiv.org//bb www./rgg///aaeeeee www.aab.//00000. 04768
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020b. URL htttpa://proceedings http://rrroeiiiigggnrrrrpp.eeeee www.iee.uc/puaaeeeeee potrripps..c/pppp///////aaaaa pttp:////ppprr//2/////aaa httpa//25544dee1111baaaaacacca.e
Abstract..tataa html
E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 511::1-11, 2017.
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: nyström-based algorithm for approximating self-attention. CoRR, abs/2102.03902, 2021. URL https: ///ari/v/viii org/ absa 03902
13

---

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang. and Amr Ahmed. Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http::tss:///eree pptpeeeds///rooceeingg www.eeii....ppppeeeeeeeeeeeeeeee haah:/666522112222887///55111133 awt.aa/9aa3366bbbbb Abstract html
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657, 2015. URL /ttps:/tt//ppr wtw.eeddiinnnneeriss....... www.neuuri.... ct/pppperr/// 2015/hash/220eee
html
Mai Zhu and Chong Fu. Convolutional neural networks combined with runge-kutta methods. CoRR, abs/ 008.0000111... 2018. httppp::/aaxtvooorr///////00...
14


Reviews:
[{"review_id": "k3xElQKWiiu", "reviewer": "Reviewer_N59G", "summary": "This paper develops an analogy between Transformer networks and dynamical systems of multiple interacting particles. They follow the work of Lu et al. 2019 using Lie-Trotter splitting scheme to derive differential equations approximating terms corresponding to operations that compose the Transformer network internals, namely multi-head self-attention and feed-forward transformation. \n\nThis paper develops the described technique beyond previous work by noting their analogous dynamical system motivates architecture choices that would require many fewer parameters than the associated Transformer. The authors investigate performance and memory gains by exploiting time-dependence in the dynamical transformation, achieving similar results to analogous weight-sharing between layers. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work builds on existing literature that relates familiar deep-learning structures to solutions of corresponding dynamical systems. The Lie-Trotter splitting scheme in particular establishes the connection to Transformer networks in literature. These previously published results are well-cited and presented in this paper. As well, this work develops these techniques beyond previously published methods by investigating various parameter sharing formulations that are naturally motivated by their dynamical system. They show improvements in complexity and demonstrate practically through experimental results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors clearly reiterate their technical contributions relative to existing Transformer networks. Primarily, they address the limitations of their complexity reductions, e.g. not affecting the quadratic cost in initial conditions. The authors do not discuss potential negative societal impact. As basic research it is difficult to suggest improvement here. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Z7vPdIfyHUp", "reviewer": "Reviewer_2RYY", "summary": "This paper avoid computing pairwise dot-product between n input vectors at each layer by computing a functional form at the  beginning and evolving it in a temporal (depth-wise) manner.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major questions\n1.Why this point of view relates to the ODE viewpoint, can you formally written in an ODE setting to show the two models have the similar \n\n2. I believe [1] is highly related to your paper, I think it would be essential to discuss and compare with it to show the dynamically improving the attention weights is ensential.\n\nMinor points\nThe notation in the paper is somehow confusing, I would suggest the authors to organize the presentation in next version, there are too many notations doesn't defined in the section 4.\n\nI'm not a NLP guy, thus I would like to let other reviewers to evaluate the experiments, in my limited experience, the experiments looks good\n\nFrom my perspective, this is a very good direction to reduce the quadratic computation power for transformer , the drawback is the unclear motivation from the dynamic system viewpoint to the proposed structure.\n\n[1] Ying C, Ke G, He D, et al. LazyFormer: Self Attention with Lazy Update[J]. arXiv preprint arXiv:2102.12702, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.75"}, "extra_text": {}}, {"review_id": "AH-q6-83gGM", "reviewer": "Reviewer_YDLp", "summary": "This paper introduces TransEvolve, a new efficient transformer architecture that approximates the multi-headed self-attention and point-wise feed-forward transformations in the standard transformers.  Drawing from recent work by Lu et al., 2019 which established the connection between transformers and modeling the dynamics of a multi-particle dynamic systems, TransEvolve approximate multiple layers of the transformer with a temporal evolution scheme that bypasses costly dot product operations.  Experiments on encoder-decoder tasks show TransEvolve to be competitive with vanilla transformers while being slightly more efficient.  On encoder-only tasks, TransEvolve yields better predictive performance with half as many parameters.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality \nThe connection between transformers and multi-particle dynamical systems was established in Lu et al., 2019.  This paper leverages this connection to inform the design of more efficient transformer architectures.  The main points of novelty in this work is the design temporal approximation schemes for the multi-headed attention and point-wise feedforward components of transformers architectures.\n\n## Quality\nThe overall quality of the paper is sufficient.  I do think the experimental results can benefit from additional discussion to provide insights into why TransEvolve performs much better than existing methods for encoder-only tasks. I found L341-354 interesting and believe the paper would benefit greatly from additional discussion to help readers gain more intuition for how TransEvolve behaves.   \n\nQuestions:\n- Did you try other values of L besides the two settings?  How does TransEvolve perform with even higher L > 6 or lower cumulative L < 6?\n- Lu et al., 2019 introduced an architecture called Macaron Net motivated by a different splitting scheme.  Did you look into approximations for that architecture? It seems like  Macaron Net does outperform vanilla transformers on WMT14 by a healthy margin.\n\n\n## Clarity\nThe paper is well written overall but I would have benefited from more intuitive explanation of the approximations presented in Sections 4 and 5.  I did appreciate the complexity summaries provided for the two approaches in the end of each section.  \n\n_Clarification needed:_\n- L343-346: \"In the case of machine translation though, it is straightforward that having a lesser number of feed-forward parameters and/or fewer encoder blocks with deeper evolution deteriorate the performance\" this seems to indicate that performance increases with parameter size.  But the next sentence states \"In general, the performance of TransEvolve variations changes inversely with the total parameter size on translation tasks.\" Am I misinterpreting something here?\n\n_Typos:_\n- L349-350: \"learn more useful representations of the input with a much fewer number of trainable parameters\" -> learn more useful representations of the input with much fewer trainable parameters\n\n\n## Significance\nThe contributions of the paper are important owing to the new perspective provided through the temporal approximations derived from taking a multi-partical dynamical systems view of multi-layer attention.  That said, I am unsure how useful TransEvolve will be in practice given the apparent need for careful tuning (i.e. there isn't a setting from randomFF-1, randomFF-2, fullFF-1, and fullFF-2 that worked well across all experiments) and the mixed experimental results (i.e. TransEvolve seems to only improve upon existing approaches in the encoder-only setting).  Additionally, as the authors noted, TransEvolve is still less efficient than linear attention models since it incurs quadratic cost to compute the initial states of the system.  \n\n\n===========Post Author Response===========\nAfter reading the author's response, I have decided to change my score to 7.  I strongly encourage the authors to include the intuitive explanation of why TransEvolve performs better on encoder only tasks and discussion of potential directions for future work in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors sufficiently discuss the computational limitations of TransEvolve compare to existing approaches.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "25A5WQJLyWq", "reviewer": "Reviewer_8Tqc", "summary": "The authors introduce the TransEvolve approach based on the reformulation of transformer architectures in terms of the temporal evolution of many particle systems. Doing so, they can significantly reduce model parameters and computational cost, while achieving competitive or better performance than existing methods on a series of benchmark tasks.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The use of ODE formalism to cast the transformer into a dynamic framework is an interesting concept, at it allows to formulate multi-head attention and feed forward operations as the evolution of time series, eliminating the need for several costly computations and reducing the total number of parameters.\n\nThe experiments are performed on a series of benchmarks covering a range of potential applications. In almost all tasks, at least one TransEvolve variant improves upon the performance of other models demonstrating the utility of the approach. However, here the text could benefit for a more in-depth discussion on the reasons for the performance difference observed between the random matrix and full feed forward variants.\n\nOverall the paper is written in a clear manner, the architecture and development of the model is easy to follow and results are presented in a concise fashion.\n\nTransEvolve offers a different view on transformer architectures. The theoretical connections to dynamical systems could be exploited to better analyze and potentially improve these kinds of models.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Potential limitations and societal impact have been sufficiently addressed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was universally well-liked by the reviewers. The reviewers found the theoretical developments to be elegant, with significant practical applications yielding a convincing new model in the transformer family. Several reviewers commented that the exposition might be improved by including more intuitive explanations.
Decision: Accept (Spotlight)