Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
Authors: Yiping Lu
Zhuohan Li
Di He
Zhiqing Sun
Bin Dong
Tao Qin
Liwei Wang
Tie-Yan Liu
Venue:  (2020)
Keywords: Transformer
Ordinary Differential Equation
Multi-Particle Dynamic System
Natural Language Processing

Abstract: The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible code can be found on http://anonymized

Under review as a conference paper at ICLR 2020
UNDERSTANDING AND IMPROVING TRANSFORMER FROM A MULTI-PARTICLE DYNAMIC SYSTEM POINT OF VIEW
Anonymous authors Paper under double-blind review
ABSTRACT
The Transformer architecture is widely used in natural language processing. De- spite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a cnrvection-diffsiinnn equation in multi-particle dynamic system. In particular, how words in sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's per- spective, the rich literature of numerical analysis can be brought guide in designing effective structures beyond the Transformer. As an example, we pro- pose to replace the Lie-Trotter splitting scheme by the Strang-Marchut splitting scheme, a scheme that is more commonly used and with much lower local trunca- tion errors. The Strang-Marchuk Marchuk splitting scheme suggests that the self-attention and position-wise feed-fowwrd network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wer -iit FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to brand new architecture. Such an FFN-attention- FFN layer "Macaron-like"" and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible code can be found on http: : / annonmi zed
INTRODUCTION
The Transformer is one of the most commonly used neural network architectures in natural language processing. Variants of the Transformer have achieved state-of-the-art performance in many tasks including language modeling (Dai et al., 2019; Al-Rfou al., 2018) and machine translation (Vaswani et al., 2017; Dehghani et al., 2018; Edunov et al., 2018). Unsupervised pre-trained models based on the Transformer architecture also show impressive performance many downstream tasks (Radford et al., 2019; Devlin et al., 2018).
The Transformer architecture is mainly built by stacking layers, each of which consists of two sub-layers with residual connections: the self-attention sub-layer and the position-wise feed-forward network (FFN) sub-layer. For given sentence, the self-attentio sub-layer considers the semantics and dependencies of words at different positions and uses that information to capture the internal structure and representations the sentence. The position-win FFN sub-layer applied to each position separately and identically encode context at each position into higher-Ieved representations. Although the Transformer architecture has demonstrated promising results in many tasks, its design principle is not fully understood, and thus the strength of the architecture is not fully exploited. As far as we know, there is little work studying the foundation of the Transformer or different design choices.
In this paper, we provide a novel perspective towards understanding the architecture. In particular, are the first to show that the Transformer architecture is inherently related to the Marti-Particle

---

Under review as a conference paper at ICLR 2020
Dynamic System (MPDS) in physics. MPDS is well-established research field which aims at modeling how a collection of particles move in the space using differential equations (Moulton, 2012).
In MPDS, the behavior of each particle is ally modeled by two factors separately. The first factor is the convection which concers the mech- anism of each particle regardless of other particles in the system, and the second factor is the diffu- sion which models the movement of the particle resulting from other particles in the system.
Inspired by the relationship between the ODE and neural networks (Lu et al., 2017; Chen et al., 201 8a), we first show that the Transformer lay- ers can be naturally interpreted as a numerical ODE solver for first-order convection-diflusion equation in MPDS. To be more specific, the self- attention sub-layer, which transforms the seman- ties at one position by attending over all other positions, corresponds the diffusion term; The position-wise FFN sub-layer, which is applied to each position separately and identically, corresponds to the convection term. The number of stacked layers in the Transformer corresponds the time dimension ODE. In this way, the stack of self-attention sub-layers and position- wise FFN sub-layers with residual connections can be viewed as solving the ODE problem numerically using the Lie-Trotter splitting scheme (Geiser, 2009) and the Euler's 's method (Ascher & Petzold, 1998). By this interpretation, we have a novel understanding of learning contextual representations of sentence using the Transformer: the feature (a.k.a, embed- ding) of words in a sequence can be considered as the initial positions of collection of particles, and the latent representations abstracted in stacked Transformer layers can be viewed as the location of particles moving in high-dimensional space at different time points.
Particles move in the space along time [Semantics encoded stacked neural network layers)
Particle (Ward)
Figure I: Physical interpretation of Transformer.
Such an interpretation not only provides a new perspective on the Transformer but also inspires us to design new structures by leveraging the rich literature of numerical analysis. The Lic-Trotter splitting scheme is simple but not accurate and often leads to high approximation error (Geiser, 2009). The Strang-Marchue splitting scheme (Strang, 1968) is developed to reduce the approximation error by simple modification to the Lie- Te-rroeeer splitting scheme and is theoretically more accurate. Mapped to neural network design, the Burang-Marchuk splitting scheme suggests that there should be three sub-layers: two position-wise feed-forward sub-layers with half-step residual connections and one self-attention sub-layer placed in between with full-step residual connection. By doing so, the stacked layers will be more accurate from the ODE's perspective and will lead to better performance in deep learning. As the IFN-attention-- -FN layer is "Macaron-like" we call it Macaron layer and call the network composed of Macaron layers the Macaron Net.
We conduct extensive experiments on both supervised and unsupervised learning tasks. For each task, we replace Transformer layers by Macaron layers and keep the number of parameters to be the same. Experiments show that the Macaron Net can achieve higher accuracy than the Transformer on all tasks which, in a way, is consistent with the ODE theory.
2 BACKGROUND
2.1 RELATIONSHIP BETWERN NEURAL NETWORKS AND ODE
Recently, there are extensive studies to bridge deep neural networks with ordinary differential equations (Weinan, 2017; Lu et al., 2017; Haber & Ruthotto, 2017; Chen et al., 2018a; Zhang et al., 2019b; Sonoda & Murata, 2019; Thorpe & van Gennip, 2018). We here present a brief introduction such a relationship and discuss how previous works borrow powerful tools from numerical analysis to help deep neural network design.
A first-order ODE problem is usually defined as solve the equation (i.e., calculate sr(() for any which satisfies the following first-order derivative and the initial condition:
dr(t) = f(x, 2(to)= = w,
(1)
dt
2

---

Under review as a conference paper at ICLR 2020
in which xxt) Rd for all 2 tg. ODEs usually have physical interpretations. For example, axt) can be considered as the location of a partiele moving in the d-dimensional space and the first order time derivative can be considered as the velocity of the particle.
Usually there is no analytic solution to Eqn (1) and the problem has to be solved numerically, The simplest numerical ODE solver is the Euler's method (Ascher & Petzold, 1998). The Euler's method
dz(() discretizes the time derivative de by its first-order approximation =(t2)-=(hh) (1)( By 12-21 doing so, for the fixed time horizon T = to + yI, we can estimate ax(T) from To xxto) by sequentially estimating 1 = x(h++) using
1 21 (x),
where / = 0.... ,L 1, t = tu + y is the time point corresponds to 21, and y (T to) is see, this is mathematically equivalent to the ResNet architecture (Lu et al.,
the step size. As 2017; Chen et al., 20) The function 7 4x) can be considered as a neural-netwon block, and the second argument f the function indicates the temporal discretization by Euler's 's method naturally leads to the residual connection.
parameters in the 1-th layer. The simple
Observing such a strong relationship, researchers use ODE theory to explain and improve the neural network architectures mainly designed for computer vision tasks. Lu et al. (2017); Chen et al. (2018a) show any parametric ODE solver can be viewed as a deep residual network (probably with infinite layers), and the parameters in the ODE can be optimized through backpropagation. Recent works discover that new neural networks inspired by sophisticated numerical ODE solvers can lead to better performance. For example, Zhu et al. (2018) uses high-precision Runge- Kutta method to design neural network, and the new architecture achieves higher accuracy. Haber & Ruthotto (2017) uses a leap-frog method to construct reversible neural network. Liao & Poggio (2016); Chang al. (2019) try to understand recurrent neural networks from the ODE's ' perspective, and Tao et al. (2018) uses non-local differential equations to model non-local neural networks.
2.2 TRANSFORMER
The Transformer architecture is usually developed by stacking Transformer layers (Vaswani et al., 2017; Devlin et al., 2018). A Transformer layer operates on a sequence of vectors and outputs new sequence of the same shape. The computation inside layer is decomposed into two steps: the vectors first pass through a (multi-head) self self-attention sub-layer and the output will be further put into a position-wis wise feed-for -erd-rd network sub-layer. Residual connection (He et al., 2016) and layer normalization (Lei Ba et al., 2016) are employed for both sub-layers. The visualization of a Transformer layer is shown in Figure 2(a) and the two sub-layers are defined as below.
Self-attention sub-layer The attention mechanism can be formulated as querying a dictionary with key-value pairs (Vaswani et al., 2017), e.g., Attention(4 Q, K, V) = softmax QKT) /Vdmodel) V. where dmodel is the dimensionality of the hidden representations and Q (Query), ) K (Key), V (Value) specified as the hidden representations of the previous layer in the so-called self-attention sub- layers in the Transformer architecture. The multi-head variant of attention allows the model to jointly attend to information from different representation subspaces, and is defined as
Multi-head (Q, K, V) = Concat(head,,, ... heaa W head2 Liteetiion@W<' KW VWY),
(3)
where WO E Remaics WK Rddmmtaxdy WY E Rd and WO E R Rdy xdmadet are project parameter matrices, H is the number of heads, and dx and dy are the dimensionalities of Key and Value.
Position-wise FFN sub-layer In addition to the self-attention sub-layer, each Transformer layer also contains a fully connected geed-forwart network. which is applied to each position separately and identically. This feed-forwart network consists of two linear transformations with activation function a in between. Specially, given vectors hi...., hn, a position- wise FFN sub-layer transforms each h; as FFN(h,) a (h, W b, + by, where W,, W2, b, and by are parameters.
In this paper, we take the first attempt to provide an understanding of the feature extraction process in natural language processing from the ODE's viewpoint. As discussed in Section 2.1, several
3

---

Under review as a conference paper at ICLR 2020
works interpret the standard ResNet using the ODE theory. However, we found this interpretation cannot be directly applied to the Transformer architecture. First, different from vision applications whose size of the input (e.g., image) is usually predefined and fixed, the input (e.g., sentence) ODE
in natural language processing is always of variable length, which makes the single-parte formulation used in previous works not applicable. Second, the Transformer layer contains very distinct sub-layers. The self-attention sub-layer takes the information from all positions as input while the position-wis feed-forward layer is applied each position separately. How to interpret these heterogeneous components by ODE is also not covered by previous works (Tao et al., 2018; Chen et al., 2018a).
3 REFORMULATE TRANSFORMER LAYERS AS AN ODE SOLVER FOR MULTI--ARTICLA DYNAMIC SYSTEM
In this section, first introduce the general form of differential equations in MPDS and then reformulate the stacked Transformer layers to show they form a numerical ODE solver for a specific problem. After that, we use advanced methods in the ODE theory to design new architectures.
3.1 MULTI-PARTICLE ODE AND ITS NUMERICAL SOLVER
Understanding the dynamics of multiple particles' movements in space is one of the important problems in physics, especially in fluid mechanics and astrophysics (Moulton, 2012). The behavior of each particle usually modeled by two factors: The first factor concerns about the mechanism of its movement regardless of other particles, c.g., caused by an external force outside of the system, which usually referred to as the convection; The second factor concerns about the movement resulting from other particles, which is usually referred to as the diffusion. Mathematically, assume there are particles d-dimensional space. Denote zt()) E IRd as the location of i-th particle at time t. The dynamics of particle i can be formulated as
(t)
((x;(t), (1) In (0), G(x,(t),
dt
x. (1)) = W/: = I.....
(5)
Function (();()),, [ar (!),... 2., (()] 1) represents the diffusion term which characterizes the inter- action between the particles. G(x,, 1) is function which takes a location æ and time as input and represents the convection term.
Splitting schemes As we can see, there are two coupled terms in the right-hand side of Equ (5) describing different physical phenomena. Numerical methods of directly solving such ODEs can be complicated. The splitting method is prevailing way of solving such coupled differential equations that can be decomposed into a sum of differential operators (McLachlan & Quispel, 2002). Furthermore, splitting convection from diffusion is quite standard for many convection-diffusil equations (Glowinski et al., 2017; Geiser, 2009). The Lie-Trotter splitting scheme (Geiser, 2009) is the simplest splitting method. It splits the right-hand side of Eqn (5) into function F(.) and G(.) and solves the individual dynamics alternatively. More precisely, to compute zt(t + 7) from (t), the Lie-Trotter splitting scheme with the Euler's method reads
(1) x1(()) + F((a (m (1), (.)().)...).))))) (t + = (t) (t).
(6)
(7)
From time to time t + 7, the Lie- Tie-trotter splitting method first solves the ODE with respect to F(-) and acquire intermediate location 2 (t). Then, starting from (t). it solves the second ODE with respect to G(-) to obtain (t +
3.2 PHYSICAL INTERPRETATION OF THE TRANSFORMER
We reformulate the two sub-layers of the Transformer in order to match its form with the ODE described above. Denote o (..,,,,...;; (t,,) as the input the (-th Transformer layer, where is the sequence length and æ1,; real-valued vector in Rd for any
4

---

Under review as a conference paper at ICLR 2020
Reformulation of the self-attention sub-layer Denote Et, the output of the (multi-head) self- attention sub-layer at position with residual connections. The computation of Tt, can be written
Tt,i 21, + Concat (head y ... headæ) W
(8)
12
exp(eij (k) j=1
where headg Σ
(9) 2
[20, W
Σ exp(e
and (R) is computed as the dot product of input 21,1 and 21, with linear projection matrices k e K.
and W i.e., Ej = d
model ( (ar, W Considering (k)
aiy as a normalized value of
the pair-wise dot product eij over j, we can generally reformulate Eqn (8) as
1 T1,i MultiHeadAttwy. (1 [21,1, "1,2, ,2t,,))
(10)
where W denotes all trainable parameters in the I-th self-attention sub-layer.
Reformulation of the position-wise FFN sub-layer Next, Tt, is put into the position-wise FFN sub-layer with residual connections and output 21+1,1 The computation of 2141, can written as
11+ Ct, + WF (ir,s),
(11)
where W ff
denotes all trainable parameters in the 1-th position-wist FFN sub-layer.
Reformulation of Transformer layers former layers' as
Combining Equ (10) and (11), we reformulate the Trans-
Tt.i 21,i
MultiHeadAAtly, (21,5, [12,1, "1 (iin))
(12)
F1+1, 3t, WFN wt (it,)).
(13)
We can see that the Transformer layers (Eqn (12-13)) resemble the multi multi-particll ODE solver in Section 3.1 (Eqn (6-7)). Indeed, we can formally establish the link between the ODE solver with splitting scheme and stacked Transformer layers as below.
MulriHeadAAtwi, (I1,5, [EI,1. ,2I,)) and (x),, FFNws, (I1)). The Transformer can be viewed numerical ODE solver ing Lie-Trotter splitting scheme and the Fuler's method (with time step for Eqn (5) with F and G*
Claim 1. Define (I1,,, [II,1, ,t,m,,
The above observation grants a physical interpretation natural language processing and provides a new perspective on the Transformer architecture. First, this perspective provides unified view of the heterogeneous components in the Transformer. The self-attention sub-layer is viewed as a diffusion term which characterizes the particle interactions while the position-wel wise feed-forward network sub-layer is viewed as convection term. The two terms together naturally form the convection- diffusion equation physics. Second, this interpretation advances our understanding of the latent representations of language through the Transformer. Viewing the feature (a.k.a., embedding) of words sequence as the initial position of particles, we can interpret the latent representations of the sentence abstracted by the Transformer as particles moving in a high-dimensional space as demonstrated in Figure (Zhang, 2019).
3.3 IMPROVING TRANSFORMER VIA
SPLITTING SCHEME
In the previous subsection, we have successfully mapped the Transformer architecture a numerical ODE solver for MPDS. However, we would like to point out that one of the key components in this ODE solver, the Lie- fe-Trotter splitting scheme, the simplest one but has relatively high errors. this
Layer normalization is sometimes applied to the sub-layen but recent work (Zhang et al., 2019a) shows that the normalization trick is not essential and can be removed, One still readily check that the reformulation (Eqn (12) and (13)) still holds with layer normalization.
5

---

Under review as a conference paper at ICLR 2020
Contextual
Representation
Position-wise Feed Forward
Convection
Convection Step
Position-wise
Step
Multi-Head Sclf Attention
Diffusion Step
Diffusion Step
Multi Head Self Attention
XN
Positional Encoding
Position-wise Feed Forward
Convection
Stop
Embedding
Inputs
Original Transformer
(b) Layers
Figure 2: The Transformer and our Macaron architectures.
subsection, incorporate of the most popular and widely used splitting scheme (Geiser, 2009), the Strang-Marchuk splitting scheme, into the design of the neural networks.
The Lie-Trotte splitting scheme solves the dynamies of F(.) and G(.) alternatively and exclusively in that order. This inevitably brings bias and leads to higher local truncation errors (Geiser, 2009). To mitigate the bias, we use simple modification the Lie-Trotter splitting scheme by dividing the one-step numerical solver for G(-) into two half-steps: we put one half-step before solving F-)) and put the other half-step after solving F(-). This modified splitting scheme known as the Trran-MMarchu splitting scheme (Strang, 1968). Mathematically, to compute x; (l 7) from ((),
the Strang- Marchuk splitting scheme reads as
(t) =,(() (()(())
(14)
(1) -..(1) (=((())((E)()) ( (=))), (..()))))
(15)
(, + - 2.(()) (
(16)
The Strann-Marchuk splitting scheme enjoys higher-order accuracy than the Lie-Trotter splitting scheme (Bobylev & Ohwada, 2001) in terms of the local truncation error (Ascher & Petzold, 1998), which measures the per-step distance between the solution and the approximated solution using
numerical schemes. Mathematically, for a differential equation dzít) scheme A, the local truncation error of numerical scheme A is defined
dt = fxx t) and numerical xxt 7) A(x(t), 7).
For example, when A is the Euler's method, TEaler = xt + 7) x(t) - (xx(), t). The order of local truncation error of the two schemes has been studied in Bobylev & Ohwada (2001), as shown in the following theorem.
Theorem 1. (Bobylev Ohwada, 2001) The local truncation error of the Lie- Trotter splitting scheme is second-order (O(??)) and the local truncation error of the Strang- Marchuk splitting scheme is third-order (O(r²)).
For completeness, we provide the formal theorem with proof in the appendix. We can see from Equ (14-16) that the Strang- Marchuk splitting scheme uses a three-step process to solve the ODE. Mapped to neural network design, the Strang-Marchutt Marchuk splitting scheme (together with the Fuler's method) suggests there should also be three sub-layers instead of the two sub-layers in the Transformer. By replacing function F and 7G by MultHHeaAAtt and FFN, we have
Ft. 21,4
FFN w.l.don (I1,))
(17)
fr. ēo,4 MultiHeadAll (ār,s, [Tr, 21,2,-.. (.,n))
(18)
T1+1 $1,4 FFN w (f1,1).
(19)

---

Under review as a conference paper at ICLR 2020
Table 1: Translation performance (BLEU) on IWSLT14 De-En and WMT14 En-De testsets.
IWSLT14 De-En
WMT14 En-De
Method
small
base
big
Transformer (Vaswani et al., 2017)
34.4
27.3
28.4
Weighted Transformer (Ahmed et al., 2017)
28.4
28.9
Relative Transformer (Shaw et al., 2018)
26.8
29.2
Universal Transformer (Dehghani et al., 2018)
28.9
Scaling NMT (Ott et al., 2018)
/
/
29.3
Dynamic Conv (Wu al., 2019a)
35.2
29.7
Macaron Net
35.4
28.9
30.2
From Eqn (17-19), we can see that the new layer composes of three sub-layers. Each hidden vector at different positions will first pass through the first position-wist wise FFN sub-layer with half-step residual connection ("2 in Eqn (17)), and then the output vectors will be feed into a self-attention sub-layer. In the last step, the output vectors from the self-attention sub-layer will be put into the second position-wise FFN sub-layer with a half-step residual connection. Since the FFN-attention- FFN structure is "Macaron" -like, call the layer Macaron layer and call the network using Macaron layers as Macaron Net, as shown in Figure 2(b). Previous works (Lu et al., 2017; Zhu et al., 2018) have successfully demonstrated that the neural network architectures inspired by higher-order accurate numerical ODE solvers will lead to better results in deep learning and we believe the Macaron Net can achieve better performance on practical natural language processing applications than the Transformer.
4 EXPERIMENTS
We test our proposed Macaron architectures in both supervised and unsupervised learning setting. For the supervised learning setting, we IWLST14 and WMT14 machine translation datasets. For the unsupervised learning setting, we pretrain the model using the same method as in Devlin et al. (2018) and the learned model over set of downstream tasks. Extra descriptions about datasets, model specifications, and hyperparameter configurations can be found in the appendix.
4.1 EXPERIMENT SETTINGS
Machine Translation Machine translation is an important application for natural language process- ing (Vaswani et al., 2017). We evaluate our methods on two widely used public datasets: IWSLT14 German-to-Englinn (De-En) and WMT14 English-to-German (En-De) dataset.
For the WMT14 dataset, the basic configurations of the Transformer architecture are the base and big settings (Vaswani al., 2017). Both of them consist of 6-layer encoder and 6-layer decoder. The size of the hidden nodes and embeddings of heads are for base and 16 for big. Since the IWSLT14 dataset much smaller than the WMT14 dataset, the small setting is usually used, whose size of hidden states and embeddings is set to 512 and the number of heads set to 4. For all settings, the dimensionality of the inner-layer of the position-wisg FFN four times of the dimensionality of the hidden states.
set 512 for base and 1024 for big. The number
For each setting (base, big and small miaal.3 11, we replace all Transformer layers by the Macaron layers' and obtain the base, big and small 11 Macaron, each of which contains two position-wise feed-
"The half-step critical in the Mtreng- splitting scheme solve ODE problem, but it may be essential in training particular neural network. For example, the FFN sub-layer the Transformer designed as (F,( = o(h, W. + by W2 + b2. Placing 1/2 to rescale this FFN sub-layer is equivalent to rescale the parameter Wz and b2 from the beginning of the optimization.
'The translation model is based on the encoder-decoder framework. In the Transformer, the decoder layer has a third sub-layer which performs multi- head attention over the output of the encoder stack (encoder-decoder decoder- attention) and mask to prevent positions from attending to subsequent positions. In our implementation of Macaron decoder, we also use masks and split the FFN into two sub-layers and thus our decoder layer is (PFN, self-attention, mncderrddecoderrattennin and FFN).
7

---

Under review as a conference paper at ICLR 2020
Table 2: Test results on the GLUE benchmark (except WNLD.
Method
CoLAA
SST-2
MRPC
STS-B
QQP
MNLI-n/mm
QNLI
RTE
GLUE
Existing systems
ELMo (Peters et ul., 2018)
33.6
90.4
84,4/78.0
74.2/72.3
63 1/84.3
74.1/74.5
79.8
58.9
70.0
OpenAl GPT (Radford al.)
47.2
93.1
87.7/83.7
85.3/84.8
70.1/88.1
80.7/80.6
87.2
69.1
76.9
BERT base (Devlin et al., 2018)
52.1
93.5
88.9/84.8
87.1/85.8
71.2/89.2
84.6/83.4
90.5
66.4
78.3
Our systems
BERT base (ours)
52.8
92.8
87.3/83.0
81.2/80.0
70.2/88.4
84.4/83.7
90.4
64.9
77.4
Macaron Net base
57.6
94.0
88.4/84.4
87.5/86.3
70.8/89.0
85,4/84.5
91.6
70.5
79.7
forward sub-layers in a layer. To make a fair comparison, we set the dimensionality of the inner-layer of the two FFN sub-layers in the Macaron layers to two times of the dimensionality of the hidden states. By doing this, the base, big and small Macaron have the same number parameters as the base, big and small Transformer respectively.
Unsupervised Pretraining BERT (Devlin et al., 2018) the current state-of-the-art pre-trained contextual representation model based multi-layer Transformer encoder architecture and trained by masked language modeling and next-sentence prediction tasks. We compare our proposed Macaron Net with the base setting from the original paper (Devlin et al., 2018), which consists of 12 Transformer layers. The size of hidden states and embeddings are set to 768, and the number of attention heads is set 12. Similarly, we replace the Transformer layers in BERT ba by the Macaron layers and reduce the dimensionality of the inner-layer of the two FFN sub-layers by half, and thus we keep the number of parameters of our Macaron base same BERT base.
4.2 EXPERIMENT RESULTS
Machine Translation We use BLEU (Papineni et al., 2002) as the evaluation measure for machine translation. Following common practice, we use tokenized case-sensitive BLEU and case-insensitive BLEU for WMT14 En-De and IWSLT14 De-En respectively.
The results for machine translation are shown in Table 1. For the IWSLT14 dataset, our Macaron small 1m outperforms the Transformer small 1 by 1.0 terms BLEU. For the WMT14 dataset, Macaron base outperforms its Transformer counterpart by 1.6 BLEU points. Furthermore, the performance of our Macaron base model is even better than that of the Transformer big model reported in Vaswani al. (2017), but with much less number of parameters. Our Macaron bi g outperforms the Transformer big g by 1.8 BLEU points. Comparing with other concurrent works, the improvements in our proposed method are still significant.
Unsupervised Pretraining Following Devlin et al. (2018), we evaluate models by fine-tuning them on 8 downstream tasks in the General Language Understanding Evaluation (GLUU) benchmark (Wang et al., 2019), including CoLA (Warstadt et al., 2018), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017), QQP (Chen et al., 2018 2018b), MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and RTE (Bentivogli et al., 2009). More details about individual tasks and their evaluation metrics can be found in the appendix and Wang et al. (2019); Devlin et al. (2018). To mune the models, we follow the hyperparameter search space in Devlin al. (2018) for all downstream tasks, including different batch sizes, learning rates, and numbers of epochs.
The GLUE results are presented Table We present the results of two BERT base models. One from Devlin et al. (2018), and the other is trained using our own data. Due to that some pieces of data used Devlin et al. (2018) are no longer freely distributed, the two numbers are slightly different. We can see from the table, our proposed Macaron Net base outperforms all baselines in terms of the general GLUE score. Specifically, given the same dataset, our proposed model outperforms our trained BERT base model in all tasks. Even comparing with the BERT base model in Devlin et al. (2018), our model performs better in out of tasks and achieves close performance in the rest tasks.
8

---

Under review as a conference paper at ICLR 2020
Table 3: Additional Comparisions on WMT14 En-De translation tasks.
Method
#Params
BLEU
Att-FFN-Att No residual
213M
28.6
213M
28.3
8-Layer Transformer (big. g, Wu et al., 2019b)
284M
28.8
10-Layer Transformer (big, Wu et al., 2019b)
355M
28.6
Macaron Net (big)
213M
30.2
As summary, the improvement in both machine translation and GLUE tasks well aligns with the ODE theory and our proposed architecture performs better than the Transformer in real practice.
4.3 ADDITIONAL COMPARISONS
As see, the main difference between Macaron Net and the Transformer is that Macaron Net uses two FFN sub-layers in layer block while the Transformer just uses one. One may argue that the improvements of the new architecture may be simply due to adding nonlinearities but not from ODE theory. We point out here that adding nonlinearities does not always guarantee improvements in deep learning. For example, foed-forward neural networks contain much more nonlinearities than convolutional neural networks but the performance of feed-forward neural networks is superior than that of convolutional neural networks in image classification. Furthermore, Wu al. (2019h) shows that simply adding nonlincarities by stacking more Transformer layers does not work well in practice.
To evaluate the effectiveness of network, we further conducted experiments show the ODE- inspired way of adding nonlinearities is better than following heuristic methods:
At-FFFN-Arr In Macaron Net, we use two FFN sub-layers and one attention sub-layer in layer. Here we construct baseline that sub-layer layer. Note that the attention sub-layer also contains nonlinearities in the softmax and dot-product operations.
attention sub-layers and one FFN
No residual: In the Transformer, the FFN sub-layer has one hidden layer. We changed it to two hidden layers without residual connections, which increases the nonlincarities.
We compare these two models with Macaron Net on WMT14 En-De task and keep all the model parameters to be the same the Transformer big setting. We list the performance of different models in Table 3. We can see that the BLEU scores of two models are 28.6/28. respectively. As comparison, the BLEU score of Macaron Net is 30.2. We also cite the results from Wu et al. (2019b), which shows that simply stacking more Transformer layers cannot reach comparable performance. Therefore, we believe our ODE-inspired architecture design is principled and better than heuristics, which provides better understanding to the Transformer model.
Furthermore, we can also see that the empirical results are consistent with the ODE theories. The step size in ODE relates to #layers in deep learning. In detail, small step size maps to a large value of #layer: a neural network with more layers stacked corresponds to an ODE solver with smaller step size In Table 1 and 2, we better than the Transformer. Results in Wu et al. (2019b) show that even using a smaller 7 (8-layer or 10-layer Transformer), our Macaron Net (6-layer) is still better. These results are consistent with ODE theories: (1) A higher-order ODE solver is better given the same step size (6-layer Macaron Net 6-layer Transformer).
that that given the same step size (#layers), our Macaron Net is
(2) The order matters, higher-order ODE solver works well even using a relatively larger step size (6-layer Macaron Net v.s. 10-layer Transformer).
5 CONCLUSION AND FUTURE WORK
In this paper, we interpret the Transformer as a numerical ODE solver for convection-diffusii equation in a multi- dynamic system. Specifically, how words in sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted approximating multiple particles' movement in the space using the Lie- Trotter splitting scheme and the Euler's method. By replacing the Lie-Trotter splitting scheme with the more advanced Strang-Marchuk

---

Under review as a conference paper at ICLR 2020
splitting scheme, obtain new architecture. The improvements in real applications show the effectiveness of the new model and are consistent with the ODE theory. In the future, we will explore deeper connections between the ODE theory and the Transformer models and use the ODE theory to improve the individual components in the Transformer architecture such as attention modules.
REFERENCES
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. arXiv preprint 17. 2017.
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character- language modeling with deeper self-attention. arXiv preprint arXiv: 1808. 04444, 2018.
Uri M Ascher and Linda R Petzold. Computer methods for ordinary differential equations and anfferentiillalgebbaae equations, volume 61. Siam, 1998.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo. Magnini. The fifth PASCAL recognizing textual entailment challenge. In TAC, 2009.
Alexander V Bobylev and Taku Ohwada. The error of the splitting scheme for solving evolutionary equations. Applied Mathematics Letters, 14(1):45-48, 2001.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual momilarity-ullilinnuue and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00053 2017.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetrierm:: A dynamical system view on recurrent neural networks. arXiv preprint MXKiv:190.099000 2019.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems, pp. 6572-6583, 2018a.
Z. Chen, H. Zhang, X. Zhang, and L. Zhao. Quora question pairs. 2018b. URL httts:://///att quora. www./c///ciaeeeeeeee Quora Daw.aaatatettttee Meseet-Releasse Question-Paaateee -
Carmen Chicone. Ordinary differential equations by vladimir i. arnold. Siam Review, 49(2):335-336. 2007.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context, arXiv preprint (:XXi:1901.02860 2019.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Eukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03811 2018
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 170.v:1810.04805, 2018.
William Dolan and Chris. Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the International Workshop Paraphrasing., 2005.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back- tack-translation at scale. arXiv preprint
2018.
Jonas Gehring, Michael Auli, David Grangier. Denis Yarats, and Yann N Dauphin. Convolutional Sequence to Sequence Learning. In Proc. of ICML, 2017.
Juergen Geiser. Decomposition methods for differential equations: theory and applications. CRC Press, 2009.
Roland Glowinski, Stanley J Osher, and Wotao Yin. Splitting methods in communication, imaging, science, and engineering. Springer, 2017.
10

---

Under review as a conference paper at ICLR 2020
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (1):014004, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pp. 630-645. Springer, 2016.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer. Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. ACL, 2007.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064400 2016.
Hector J Levesque, Ernest Davis, and Leora. Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp. 47, 2011.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint 1rXiv:1604.0364)) 2016.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 2017.
Brian W Matthews. Comparison of the predicted and observed secondary structure of phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442 -511, 1975.
Robert McLachlan and G Reinout W Quispel. Splitting methods. Acta Numerica, 11:341-433 2002.
Forest Ray Moulton. An introduction celestial mechanics. Courier Corporation, 2012.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. arXiv preprint arXiv:1806.001 001180660017, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th amnual meeting association for computational linguistics, pp. 31 311-318. Association for Computational Linguistics, 2002.
Matthew E Peters, Mark Neumann, Mohit Iyyer. Matt Gardner, Christopher Clark, Kenton Lec, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1802.080 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy. Liang. SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of EMNLP., pp. 2383-2392. Association for Computational Linguistics, 2016.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACI., 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint S003:::88030215555 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP., pp. 1631-1642, 2013.

---

Under review as a conference paper at ICLR 2020
Sho Sonoda and Noboru Murata. Transport analysis of infinitely deep neural network. The Journal of Machine Learning Research, 20(1):31 2019.
Gilbert Strang. On the construction and comparison of difference schemes. SIAM Journal on Numerical Analysis, 5(3):506-517, 1968.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-28266 2016.
Yunzhe Tao, Qi Sun, Qiang Du, and Wei Liu. Nonlocal neural networks, nonlocal diffusion and nonlocal modeling. In Advances in Neural Information Processing Systems, pp. 496-506, 2018.
Matthew Thorpe and Yves van Gennip. Deep limits of residual neural networks. arXiv preprint arXiv: 117441000.1111,...1 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-tase tasl benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. arXiv preprint 1805. 12471, 2018.
E Weinan. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HITT, 2018.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.1045: 2019a.
Lijun Wu, Yiren Wang, Yingce Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Depth growing for neural machine translation. CoRR, abs/1197.01196 2019b. URL org htgg/bbs/990/ 00 968.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 1901:0990.09330 2019a.
Wenxin Zhang. The Bonfire of Time, Artwork, 2019.
Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, and Bin Dong. Dynamically unfolding recurrent restorer: moving endpoint control method for image restoration. In International Conference on Learning Representations, 2019b. URL https: / reviinerviee ew net./ fitt/formm? f=SJfzKKi C5FX.
Mai Zhu, Bo Chang, and Chong Fu. Convolutional neural networks combined with runge-kutta methods. arXiv preprint arXiv:1802.0880 2018.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In arXiv preprint 4rXiv:1506.0672..000 2015.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
A PROOF OF THE THEOREM
Theorem 2. (Bobylev & Ohwada, 2001) We denote the true solution of equation de = =(x). = dt yo² at time 1 as s=()= = (wo) Simliarily, we can define S. and 5/+ The local truncation error at = of the Lie- Trottrrtte splitting scheme is ( - (y0) = = (Fo)
G' (vo) Fyo)) + O(7²) which is second-order. The local trumcation error at = 0 of the Strang- Marchuk splitting scheme 2 (57152 (yo)l} is ( (S (vo)]} O(7²) which is third-order.
(10)))
Proof. Since
(16) = 30 + 7 F((o) + 7²
F' + 0(72),
2
(y0) = y0 + -G(o))
+ G ((oo) + +
2
we have
22
$ (31) =57())))))))))))))) + +
((5))))))(((((
At the same time, we have
G(yo + + G(yo) + ((x))) + O
G((S((0))GG((3(mm) ('(n)G((y)) + 0(7).
Combine the estimations we have
S3157(3601 yo + (F(()) + G(yo)]
2 ('(o)) + + 2G (()) + 0(7²).
As result, we can estimate the local truncation error of Lie-Trotter splitting scheme as
S7-0(00) - (30))
= yo + G(vo)) 2 (" G(yo)) 0(>3)
(yo + > G(vo)] + 72
G (''()) + (16) + 2G'())) + (
2
2 2
= (F((ro) - ((w 0173).
To estimate the Marchuk splitting scheme as
splitting scheme's local truncation error, we rewrite the Strang-
(S
(S (vo))))-
From the previous estimation of Lie vie-Trotter splitting scheme have
72 (F'(yo)G(yo) - ((1)))(()) + +0(7),
S72c(10) (3157
8
22 8 (G' - +0(7))
5 (1) - (7)5
"Since time-dependent ODE can be formulated as a time-independenn ODE by introducing an auxiliary variahle (Chicone, 2007), the theorem here developed for time-independent ODEs can also he applied to time-dependent ODEs without loss of generality.
13

---

Under review as a conference paper at ICLR 2020
Combine the two estimations, have
22 S6 (6 (wo)I 574c(00) [F'(wo)((w)) G((o)) + _ (C'(o))) - (o0 Fuc(U)) +()²))
B EXPERIMENT SETTINGS
B. MACHINE TRANSLATION
Dataset The training/validationnnsst sets of the IWSLT14 dataset contain about 153K/7K/7K sen- tence pairs, respectively. We use a vocabulary 10K tokens based on a joint source and target byte pair encoding (BPE) (Sennrich et al., 2016). For WMT14 dataset, replicate the setup of Vaswani et al. (2017), which contains 4.5M training parallel sentence pairs. Newstest2014 is used as the test set, and Newstest2013 is used the validation set. The 37K vocabulary for WMT14 is based joint source and target BPE factorization.
Model For the WMT14 dataset, the basic configurations of the Transformer architecture are the base and the big settings (Vaswani et al., 2017). Both of them consist of 6-layer encoder and 6-layer decoder. The size of the hidden nodes and embeddings are set to 512 for base and 1024 for big. The number of heads are 8 for base and 16 for big. Since the IWSLT14 dataset is much smaller than the WMT14 dataset, the small setting is usually used, whose size of hidden states and embeddings is set to 512 and the number of heads is set to 4. For all settings, the dimensionality of the inner-layer of the position-wise FFN is four times of the dimensionality of the hidden states.
For each setting (base, big and 1), we replace all Transformer layers by the Macaron layers and obtain the base, big and small Macaron, each of which contains two position-wist feed- forward sub-layers in a layer. The translation model is based on the encoder-decoder framework. In the Transformer, the decoder layer has third sub-layer which performs multi-head attention over the output of the encoder stack (encoder-decoder attennion))) and a mask to prevent positions from attending to subsequent positions. our implementation of Macaron decoder, we also use masks and split the FFN into two sub-layers and thus our decoder layer is (FFN, self-attention, ponoder-decoder-atttenit and FFN).
To make fair comparison, we set the dimensionality of the inner-layer of the two FFN sub-layers in Macaron layers to two times of the dimensionality the hidden states. By doing this, base, big and small 1. Macaron have the same number of parameters as the base, big and small 11 Transformer respectively.
Optimizer and training We use the Adam optimizer and follow the optimizer setting and learning rate schedule in Vaswani et al. (2017). For the big setting, we enlarge the batch size and learning rate as suggested in Ott et al. (2018) to accelcrate training. We employ label smoothing of value Et 0.1 (Szegedy et al., 2016) in all experiments. Models for NVIDIA P40 GPUs respectively. Our code is based on the open-sourced fa fairseq (Gehring et al., 2017) code base in Py PyTorch toolkit.
are trained on 4/1
Evaluation We use BLEUS (Papineni et al., 2002) as the evaluation measure for machine translation. Following common practice, use tokenized case-sensitive BLEU and case- iase-insensitive BLEU for WMT14 En-De and IWSLT14 De-Ei respectively. During inference, we use beam search with beam size 4 and length penalty 0.6 for WMT14, and beam size 5 and length penalty 1.0 for IWSLT14, following Vaswani et al. (2017).
multi-bleu. 1
14

---

Under review as a conference paper at ICLR 2020
B.2 UNSUPERVISED PRETRAINING
Pre-training dataset We follow Devlin et al. (2018) to use English Wikipedia corpus and Book- Corpus for pre-training. As the dataset BookCorpus (Zhu et al., 2015) no longer freely distributed. We follow the suggestions from Devlin et al. (2018) to crawl and collect BookCorpus on our own. The concatenation two datasets includes roughly 3.4B words in total, which is comparable with the data corpus used in Devlin et al. (2018). We first segment documents into sentences with Spacy:? Then, we normalize, lower-case, and tokenize texts using Moses (Koehn et al., 2007) and apply BPE (Sennrich et al., 2016). We randomly split documents into training and one validation set. The maining-validatinn ratio for gre-training is 199:1.
Model We compare our proposed Macaron Net with the base setting from the original paper (Devlin et al., 2018), which consists of 12 Transformer layers. The size of hidden states and embeddings are set to 768, and the number of attention heads is set 12. Similarly, we replace Transformer layers in BERT base by the Macaron layers and reduce the dimensionality of the inner-layer of the two FFN sub-layers by half, and thus we keep the number of parameters of our Macaron base as the same as BERT base.
Optimizer and training We follow Devlin et al. (2018) to use two tasks to pretrain our model. One task masked language modeling, which masks some percentage of the input tokens random, and then requires the model to predict those masked tokens. Another task is next sentence prediction, which requires the model to predict whether two sentences a given sentence pair are consecutive. We use the Adam optimizer and follow the optimizer setting and learning rate schedule in Devlin al. (2018) and trained the model on NVIDIA P40 GPUs.
B.3 GLUE DATASET
We provide brief description of the tasks in the GLUE benchmark (Wang et al., 2019) and our fine-tuning process on the GLUE datasets.
CoLA The Corpus of Linguistic Acceptability (Warstadt et al., 2018) consists of English accept- ability judgments drawn from books and journal articles linguistic theory. The task is to predict whether an example is a grammatical English sentence. The performance is evaluated by Matthews correlation coefficient (Matthews, 1975).
The Stanford Sentiment Treebank (Socher al., 2013) consists sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of given sentence (positive/negative).. The performance is evaluated by the test accuracy.
SST-2
The Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent, and the task to predict the equivalence. The performance evaluated by both the test accuracy and the test Fl.
MRPC
The Semantic Textual Similarity Benchmark (Cer al., 2017) is collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair human-annotated with a similarity score from to 5; the task to predict these scores. The performance is evaluated by Pearson and Spearman correlation coefficients.
STS-B
QQP
The Quora Question Pairs (Chen et al., 2018b) dataset is collection of question pairs from the community question-answeringg website Quora. The task is to determine whether pair of questions semantically equivalent. The performance is evaluated by both the test accuracy and the test F1.
'tttps:// http://wwwwwmmsworr...... www.ssaassswsscommmmm com/ "https://spp
"https://data" wwa.ta...uor.... com/ /
Release Quest on Pairs
15

---

Under review as a conference paper at ICLR 2020
The Multi-Gent Natural Language Inference Corpus (Williams et al., 2018) is crowd- sourced collection sentence pairs with textual entailment annotations. Given premise sentence and hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment ), contradicts the hypothesis (contradiction), or neither (neutral). The performance is evaluated by the test accuracy both matched (in-domain) and mismatched (cross-domain) sections of the test data.
MNLI
The Question-ansse answering NLI dataset is converted from the Stanford Question Answering Dataset (SQuAD) (Rajpurkar al., 2016) to classification task. The performance evaluated by the test accuracy.
QNLI
The Recognizing Textual Entailment (RTE) datasets come from a series annual textual entailment challenges (Bentivogli et al., 2009). The task is predict whether sentences in a sentence pair are entailment. The performance is evaluated by the test accuracy.
RTE
WNLI The Winograd Schema Challenge (Levesque et al., 201 is a reading comprehension task in which a system must read sentence with pronoun and select the referent of that pronoun from list of choices. We follow Devlin et al. (2018) to skip this task in our experiments because few previous works do better than predicting the majority class for this task.
Fine-tuning on GLUE tasks the optimization hyperparameters in search space including different batch sizes (16/32), learning rates (5e-3/3e-5), number of epochs (3/4/5), and a set of different random seeds. We select the model for testing according to their performance the development set.
fine-tune the models, following Devlin et al. (2018), we search
Test data Note that the GLUE dataset distribution does not include the Test labels, and we only made a single GLUE evaluation server submission" for each of models.
Phttps://g wabenchmark.com
16


Reviews:
[{"review_id": "SygNWhNWtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Contributions: This paper builds an ad-hoc connection between the Transformer and the numerical ODE solver (the Lie-Trotter splitting scheme and the Euler's method) for a convection-diffusion equation in a multi-particle dynamic system. Then, the author(s) developed an ad-hoc Strang-Marchuk splitting style architecture, named Macaron Net. Finally, this paper provides some experiments to verify the performance of the proposed architecture. However, the comparisons with the benchmark results are questionable. I have listed my concerns in the Experiment section.\n\n\nMotivation: This paper developed the Macaron Net based on a locally third-order operator splitting scheme for the convection-diffusion equation. However, there is no theoretical interpretation of why third-order splitting corresponding to better architecture. Theorem 1 in the paper is a known result, and it is irrelevant to the paper, I highly recommend the author to remove it from the main text. I also suggest the author explore more operator splitting schemes and do a systematic comparison between them. Moreover, I think it will be a real contribution if the author can analyze the error between the numerical scheme and architectures.\n\n\nReformulate Transformer Layers ans an ODE solver for Multi-Particle Dynamic System: There is a big gap between Eqns 3, 4 and 5. Why F represents a diffusion term, why G represents a convection term? From a statistical mechanics point of view, this comparison does not make sense. I do not buy this model.\n\n\nRelated Work: There is no related-work section that discusses the related work, and all the referenced papers are generic. For instance, the efforts in developing language models, the application of convection-diffusion equation, and String-Marchuck and other operator splitting schemes in machine learning. The author should better position the paper to exist work.\n\n\nExperiments: This section is extremely questionable. My initial thought after reading the reported results is that the architecture proposed in this paper easily outperforms the existing work. However, after I do a cross-check with the existing work, I found the author did not compare with the best results reported in the benchmark work and hide much information. After simply checking two existing papers, I found that the author ignored the comparison with BERT large. Also, the author ignored the most important result reported by Wu et al. 2019b.  To be fair, the author should perform an apple-to-apple comparison with the existing work and report the uncertainties in their results. Moreover, the author should report the parameters used in all their experiments.\n\n\nI think this heuristic study might be a contribution to ICLR if all my concerns are addressed, and I am willing to raise my rating to accept.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1eKfLh4KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors show that the sequence of self-attention and feed-forward layers within a Transformer can be interpreted as an approximate numerical solution to a set of coupled ODEs. Based on this insight, the authors propose to replace the first-order Lie-Trotter splitting scheme by the more accurate, second-order Strang splitting scheme. They then present experimental results that indicate an improved performance of their Macaron Net compared to the Transformer and argue that this is due to the former being a more accurate numerical solution to the underlying set of ODEs.\n\nThe authors highlight an interesting connection between the Transformer architecture and ODEs. In particular, they derive a set of ODEs that is solved numerically by the Transformer and borrow from the body of literature on numerical ODE solvers to improve the architecture. I find that this is a very elegant and promising approach for finding better architectures. \n\nHowever, I also identified two major and a couple of minor shortcomings of the paper that are explained in detail below. Based on these shortcomings, I recommend rejecting the paper but I would be willing to increase the score if these points were addressed in sufficient detail.\n\nMajor points:\n\n1) Replacing the first-order operator splitting scheme by a second-order scheme only guarantees a lower overall truncation error if the split ODEs are solved with sufficiently high accuracy. In particular, the overall accuracy of the numerical solution to the original ODE depends on the accuracy of the operator splitting and the accuracy of the integration scheme used to solve the split ODEs (e.g. Euler’s method). The authors improve the operator splitting, i.e. they make it second-order, but they keep Euler’s method to integrate the individual ODEs. Because of that, they actually do not get rid of the lowest-order error term of the overall scheme and therefore do not obtain a more accurate ODE solver. I think this is a crucial point that invalidates the authors’ claim that the Macaron Net employs a higher-order integration scheme. As far as I am aware, this is not commented on in the paper at all. To address this shortcoming, the authors could replace Euler’s method by a second-order integrator. \n\n2) The experiments considered in this paper are interesting and show competitive performance but, in my opinion, they do not sufficiently support the claim that the Macaron Net yields a more accurate solution to the underlying set of ODEs compared to a Transformer. For a more convincing support of this claim, the authors could consider a toy problem, i.e. a simple set of ODEs with known analytical solution, and actually show that the Macaron Net is more accurate. The accuracy of a numerical ODE solver is commonly assessed by plotting the absolute difference between the exact solution (or a high-resolution numerical approximation to it) and the numerical solution vs the timestep (here \\gamma). I suspect that such an analysis would support my previous comment and show that the proposed new architecture is not more accurate ODE solver than the original one.  \n\nMinor points and questions:\n\ni) Eqs. (17-19) suggest that you apply two different FFN layers (doubling the number of parameters) instead of applying the same FFN layer twice. You comment on this in Sec. 4.1 when you say ‘..., we set the dimensionality of the inner-layer of the two FFN sub-layers in the Macaron layers to two times of the dimensionality of the hidden states’. It is not clear to me why consistency with Strang splitting requires two different layers rather than applying the same FFN layer twice. Is the reason for having a separate, trainable layer to account for the explicit time dependence of G in Eq. (16)? I think that this is a very important point that should be clarified.\n\nii) I think that this type of system is usually referred to as ‘dynamical system’ and not ‘dynamic system’. Please check that and, if applicable, update the title.  \n\niii) The authors say that Eq. (5) is a 'convection-diffusion equation’. As far as I am aware, the diffusion equation is a partial differential equation (PDE). Perhaps there is a different notion 'diffusion equation’ in ODE theory. If that’s the case, could the authors please clarify this point to avoid confusion, e.g. by adding a suitable reference in which this type of ODE is classified as a convection-diffusion equation? \n\niv) In Sec. 2 (2nd paragraph),  you cite Vaswani et al. (2017) but in that work the quantity under the square-root in the denominator of Attention(Q, K, V) is actually d_k, the dimension of the key, and not d_model.\n\nv) Figure 1 is a very vague illustration of the connection to ODEs and provides almost no explanation in the caption. I don’t think there is much value in having this figure there.\n\nvi) There are a couple of mistakes in the paper (grammar and expressions) that should be fixed. For example, ‘the Euler’s method’ instead of ‘Euler’s method’, ‘movement in the space’ instead of ‘movement in space’, ‘dynamic system’ instead of ‘dynamical system’, ‘project parameter matrices’ instead of ‘parameter matrices’ or ‘projections’, ‘specially’ instead of ‘specifically’, etc. Please take a look at the relevant sections in the paper and revise them accordingly.\n\nvii) You explain multiple times why the proposed architecture is called a ‘Macaron Net’ (Abstract, Sec 1, Sec. 3). To avoid repetition, I would only explain it once.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HyexZfMiKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper points out a formal analogy between transformers and an ODE modelling multi-particle convection (the feed-forward network) and diffusion (the self-attention head). The paper then adapts the Strang-Marchuk splitting scheme for solving ODEs to construct a slightly different transformer architecture: “FFN of Attention of FFN”, instead of “FFN of Attention”. The new architecture, refered to as a Macaron-Net, yields better performance in a variety of experiments.\n\nPROs\n1. The proposed new architecture is fairly simple.\n2. The experimental results are fairly good.\n\nCONs\n1. Introducing two feedforward layers with *different* parameters W^{down} and W^{up} is a significant deviation from Strang-Marchuk splitting. I expected the two FFNs inside the Macaron to have the same weights. As I understand it, the motivation for the splitting is to improve the numerical performance of the update scheme for the ODE. In contrast, allowing different weights for the FFNs means the “physical process” is now a lot more “free”. Is there any “physical” motivation for the different parameters? (Beyond the fact that it improves performance). How much worse is empirical performance when the parameters are the same?\n\n2. Following on from the above point, the analogy between the multi-particle system and the transformer is quite weak. The equations look similar when you squint the right way. But that’s as far as it goes. Fig 1 is a nice visualization, but it doesn’t provide insight into the dynamics of transformers. What does “Particles move in the space along time (Semantics encoded in stacked neural network layers)” mean? How do the particles connect to the semantics? \n3. The proof of Bobylev & Ohwada’s theorem is included in the paper. Is there any connection between the theorem (or the techniques used in its proof) and transformers? I suspect the answer is no.\n\nSUMMARY\nIn short, the paper (i) proposes two FFN layers instead of one in each block of the transformer and (ii) shows it performs slightly better than before. This is decent, but in my opinion not enough the clear the bar for ICLR.\n\nThe connection to multi-particle ODEs is genuinely interesting. However, it is not sufficiently fleshed out to count as a contribution (yet). It’s possible the authors have discovered something deep. It’s also possible they got lucky with a physically motivated modification of transformers that actually has nothing to do with the dynamics of multi-particle systems. I’m not sure what further experiments would be needed to make the case. But I recommend the authors dig into the equations and the dynamics to see what it really going on under the hood. Just showing improved performance on a few benchmarks is not enough to convince the connection is solid. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work, the authors interpret the Transformer as a numerical ODE modelling multi-particle convection. Guided by this connection, the authors take the Transformer that uses a feed forward net over attentions, and create a variant of transformer which instead uses an FFN-attention-FFN layer, thus the name macaron net. The authors present experiments in the GLUE dataset and in two MT datasets, and they overall report improved performance using their variant of Transformer. Thus, the main selling point of the paper is how seeing Transformer under his new light can potentially improve results through the construction of better models. The main criticisms from the authors is that  this story is not entirely convincing because the proposed variant departs a bit from the theory (R1 and comment about the Strang-Marchuk splitting) and the papers does not consider an evaluation of accuracy of Macaron in solving the underlying set of ODEs (comment from R3). As such, I cannot recommend acceptance of this paper -- I believe another set of revisions would increase the impact of this paper.
Decision: Reject