Are Transformers universal approximators of sequence-to-sequence functions?
Authors: Chulhee Yun
Srinadh Bhojanapalli
Ankit Singh Rawat
Sashank Reddi
Sanjiv Kumar
Venue:  (2020)
Keywords: Transformer
universal approximation
contextual mapping
expressive power
permutation equivariance

Abstract: Despite the widespread adoption of Transformer models for NLP tasks, the expressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, which is quite surprising given the amount of shared parameters in these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equivariance, and show that Transformer models can universally approximate arbitrary continuous sequence-to-sequence functions on a compact domain. Interestingly, our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, we prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self-attention layers and empirically evaluate them.

Published as a conference paper at ICLR 2020
ARE TRANSFORMERS UNIVERSAL APPROXIMATORS OF SEQUENCE -E---TTO---EEEE SEQUENCE FUNCTIONS?
Chulhee Yun' MIT chulheeylmit.edi
Srinadh Bhojanapalli Google Research NY bsr bsrinadhegoogle.. com
Ankit Singh Rawat Google Research NY
ankitsrawat@@ooglee.
Sashank J. Reddi
Sanjiv Kumar
Google Research NY
Google Research NY
sashank@google.com
sanjávk@google.com
ABSTRACT
Despite the widespread adoption of Transformer models for NLP tasks, the ex- pressive power of these models is not well-understood. In this paper, we establish that Transformer models are universal approximators of continuous permutation equivariant
functions with compact support, which is quite surprising given the amount of shared parameters these models. Furthermore, using positional encodings, we circumvent the restriction of permutation equiv- ariance, and show that Transformer models can universally approximate arbitrary continuous sequence-tor -euuuueeeee---seueeeeee
sequence functions our proof techniques clearly highlight the different roles of the self-attention and the feed-forward layers in Transformers. In particular, prove that fixed width self-attention layers can compute contextual mappings of the input sequences, playing a key role in the universal approximation property of Transformers. Based on this insight from our analysis, we consider other simpler alternatives to self- attention layers and empirically evaluate them.
compact domain. Interestingly.
INTRODUCTION
Self-attention based Transformer networks (Vaswani et al., 2017) have been at the center of the recent progress various natural language processing (NLP) tasks, including machine translation (Vaswani et al., 2017), language modeling (Radford et al., 2018; 2019), and question answering (Devlin et al., 2018; Yang al., 2019; Liu al., 2019). All these tasks involve learning models that map an input sequence of tokens to an output sequence of tokens. Transformers make feasible to train large models approximate these ggquence-to-sequeee process the input tokens in a parallel way, as opposed to the sequential nature of RNNs and LSTMs.
functions due to their ability to
Transformer block consists of two kinds of layers: self-attention layer and a token-wise feed- forward layer, with skip connections present in both layers. The self-attention layer transforms each input token embedding using a weighted combination of the embeddings of all tokens in the input sequence, where weights are generated by pairwise dot-products among the input token embeddings. The wise feed-forward layer then independently processes each of these modified input token embeddings without any interaction among them. Notably. Transformers employ parameter reuse across tokens, as both layers use the same parameters to process each token. Moreover, Transformers have to rely solely the pairwise dot-products to capture interaction between the input tokens.
Given the parameter sharing and limited interactions between tokens, is natural to wonder: what functions can the Transformer networks represent? Also, what is the role of the two different kinds of layers? Are both layers needed to obtain the representation power of Transformers? In the existing literature, the advantage of Transformers has often been attributed to their capability of computing contextual embeddings/mapping of the input, as opposed to fixed word embeddings as in word2vec (Mikolov et al., 2013). Is it possible to formalize the notion of
class of sqquence-to-seqeeee
"Based on work performed Google Research New York

---

Published as a conference paper at ICLR 2020
contextual mappings? If yes, can Transformers actually compute such mappings? Such questions still remain elusive.
In this paper, provide mathematical definition of contextual mappings and show that multi-head self-attention layers can indeed compute contextual mappings of the input sequences. We further show that this ability to compute contextual mappings coupled with the value mapping ability of the feed-forward layers makes Transformers universal approximators of any permutation equivariant soquence-to-sqqeenee function. We also improve this result using positional encodings, and show that Transformers can represent any sequence-to-sequenee function; i.e., the restriction of permuta- tion equivariance can be removed by positional encodings.
These results on universal approximation of possible have more efficient architecture compute contextual mappings, consequently, preserving the ability to universally approximate seguence-to-sqqueee functions? Towards this, we explore other architectures that can implement contextual mappings (to some extent), and experi- mentally evaluate their performance. In our experiments, we notice that the models that combine these simpler architectures with Transformers have better performance, compared to the standalone Transformers. We conclude the paper by presenting more discussion and interesting future research directions along these lines.
functions raise a natural question:
1.1 SUMMARY OUR CONTRIBUTIONS
We prove that Transformers are universal approximators of continuous and permutation equiv- ariant eequence-to-sequennee functions with compact support (Theorem 2). We also show that, if Transformers have trainable positional encodings added to the input, then they are universal approximators of continuous sequence-to-sequence functions a compact domain (Theorem 3).
We formalize the notion of contextual mappings and show that the attention layers can compute contextual mappings, where each unique context is mapped to unique vector (Lemma 6).
We experimentally evaluate other simpler layers that can compute contextual mappings to some extent, such bi-linear projections and separable convolutions, and show that substituting some of the self- attention layers with these layers can result in better performance (Section 5).
1.2 RELATED WORKS & NOTATION
Analysis of attention based models. Given the popularity of Transformers, there have been numer- ous works trying to understand the role of attention layers in natural language processing models. One such line of work focuses probing the output of attention layers to understand the attention mechanism and internal language representation (Hewitt & Manning, 2019; Clark et al., 2019; Co- enen et al., 2019; Vig & Belinkov, 2019). Although these results give valuable insights, a consistent theoretical analysis corroborating these findings is missing.
Universal approximation theorems. Universal approximation theorems classical results in neu- ral network theory, dating back many decades (Cybenko, 1989; Hornik, 1991). These results show that given unbounded width, one-hidden-layer neural network can approximate arbitrary contin- uous function with compact support, up to any accuracy. Other results focusing on depth appeared more recently (Lu et al., 2017; Hanin & Sellke, 2017; Lin Jegelka, 2018). particular, Lu et al. (2017); Hanin & Sellke (2017) consider fully-connected ReLU networks whose input dimension is d, and show that networks with width d + and unbounded depth universal approximators of scalar-valued continuous functions. Lin & Jegelka (2018) show that residual network with one hidden neuron per residual block a universal approximator of scalar-valued functions, given un- bounded depth. Although Transformer networks do have residual connections, due to their heavy parameter sharing, the existing analyses for residual networks do not extend to Transformers. Sannai et al. (2019) consider universally approximating permutation invariant/equivaaiant functions using fully-connected ReLU networks.
Turing completeness results on Transformers. Recently, Pérez et al. (2019) have shown that Transformers with infinite precision are Turing complete, which is not the case in finite precision setting (Dehghani et al., 2018). We note that Turing completeness deals with computation formal languages (thus discrete objects), while universal approximation focuses on functions on a contin- uum. In other words, these two different concepts; and one does not imply another.
2

---

Published as a conference paper at ICLR 2020
Notation. We use the following notation in the paper. Given matrix A, let A..j. A... and A.j denote its (t,j)-th entry, i-th row, and j-th column, respectively. We use || to denote the entry- wise E norm of A. Let of-] be the softmax operator, which takes a matrix as input and applies softmax operation to each column of the matrix, which results in column stochastic matrix, i.e., a matrix that has non-negative entries with each column summing to 1. We similarly define onl be the hardmax operator, which outputs the one-hot representation of the arg max entry for each column of the input matrix. If there are arg max entries, then the output is 1/k for such entries. We use 1, to denote a vector of length n whose entries are 1. We denote the 0-1 indicator function by (-). We use d and to denote the embedding dimension and the sequence length, respectively. We assume throughout that n 2 2, as the Transformers reduce to residual networks when n =
2 TRANSFORMER NETWORKS
A Transformer block ecquence-to-sequemee function mapping Rx [d It consists of two layers: a self-attention layer and a token-wise feed-forward layer, with both layers having a skip connection. More concretely, for an input E Rxd" consisting of d-dimensional embeddings of n tokens, a Transformer block with multiplicative or dot-product attention (Luong et al., 2015) consists of the following two layers':
Xtt = X + Σ
11 WOWiX [[W X) (
(1)
FF(X)= ALln( X) Wº ReLU(W; Attn (X + by + 1
(2)
where W E Rddm W W k. w Ruxd W2 RR Wi E Brxd b2 IR, b, R', and (F(X) is the output of the Transformer block. The number of heads h and the head size m are two main parameters of the attention layer; and denotes the hidden layer size of the feed-forward layer.
Here, we would like to point out that our definition of the self-attention layer (1) is an equivalent re- formulation of (Vaswani et al., 2017),, where they concatenate attention heads and multiply matrix Wo E Rd to the concatenation. One difference in our setup is the absence of layer normaliza- tion, which simplies our analysis while preserving the basic architecture of the Transformer.
We define the Transformer networks as the composition of Transformer blocks. The family of the sequence-to-sequenee functions corresponding to the Transformers can be defined as:
Th.m. = {g: Rdx=_ + Rdxn is composition of Transformer blocks th.m.ris..
where ph.m.r, : Raxxn - Rdxa denotes a Transformer block defined by an attention layer with heads of size m each, and a feed-forward layer with hidden nodes.
We say that a function f TRdxn -> TRdxn permutation equivariant if for any permutation matrix P. we have f(XP) f(X) i..., if permute the columns of X, then the columns of f f(X) are permuted in the same way. A Transformer block is permutation equivariant, which we formally prove in Section A. This consequently establishes the permutation equivariance of the class Th.m..
Claim 1. A Transformer block thm defines permutation equivariant map from Rdxn to Rdxn.
As seen in above, both layers (cf. (1) and (2)) of a Transformer block employ parameter reuse/sharing, because each token/column undergoes the same transformations (e.g.. W W or W1) regardless of its position. Moreover, interactions between tokens can only be captured through pairwise dot-pros in the softmax operator of:] (ef. (I)). Given such limitations in single Trans- former block's 's representation power, it is not obvious what kinds of peruuecce-tossqqeeee functions Th.m.r can approximate; we provide the answer to this question in the next section.
3 TRANSFORMERS ARE UNIVERSAL APPROXIMATORS OF TO-EENCE--O-SEQQEENN FUNCTIONS
In this section, we present our theorems showing that the Transformer networks are universal ap- proximators of soquence-to-sequee functions. Let us start by defining the target function class FPg. which consists of all continuous permutation equivariant functions with compact support that
In our proof we use bias vectors big for query projections attention layers. We omit them here for brevity.
3

---

Published as a conference paper at ICLR 2020
map R to Rdx" Here, continuity is defined with respect to any entry-wisse & norm, S p Given two functions fI. f2 :RR
IRdx + Rdxn for S p < oo, we define a distance between them as
dp( = ( (2()
1/p
The following result shows that a Transformer network with constant number of heads h, head size m, and hidden layer of size r can approximate any function in Frg.
Theorem 2. Let 1 S p < oo and 0, then for any given f E FPE. there exists a Transformer network E T,1,,4 such that dp(f.g) g) S
Next, we present our theorem on Transformers with positional encodings. In order to endow the Transformer networks with the ability to capture the information about the position of tokens in the input sequence, it is common practice to add positional encodings E E Rdxx to the input sequence before feeding it to the Transformer network (Vaswani et al., 2017; Devlin et al., 2018). Consider the functions represented by Transformers with positional encodings:
TP T.m. = Lop g(x + B) 1 hmm. and E Rdxa).
Here show that if E is trainable, these positional encodings are sufficient remove the permu- lation equivariance restriction of the Transformers. Towards this, we define FCD to be the set of all continuous functions that map compact domain in Rdxx Rdd Note that FCD does not have in FPB, but any f FeD is defined on compact
the restriction of permutation equivariance domain instead of the whole Rdxx, The following result states that, equipped with the trainable positional encodings, Transformers can approximate any sequence-to-sequene function in FCD.
Theorem 3. Let 1 S network E Tp such that we have (f g) S e.
and > 0, then for any given f E FCD. there exists Transformer
Theorems 2 and 3 provide an interesting characterization of the representation power of fixed-width Transformer networks. Since the function classes Th.m.r and T.h...." become richer as increase the values of (h, m,,), r), our results establish that general Transformer networks are also universal approximators of sequence-to-sequene functions. Remarkably, none of the parameters (h, m, depend on the input sequence length n or embedding dimension d.
Here, we would like to again point out that Theorems and 3 appear quite surprising first glance, given the parameter sharing across all the tokens a sequence, e.g.. feed-forward layers are applied token-wise and the projection matrices in the self-attention layers are the same across different tokens. Furthermore, attention layers can only capture pairwise interaction between different tokens in the sequence. In the next subsection, we briefly describe one of our key steps in overcoming the aforementioned restrictions and proving universal approximation power of Transformers.
3.1 A KEY STEP: ATTENTION LAYERS CAN IMPLEMENT CONTEXTUAL MAPPINGS
Let us consider a setting where we are interested in embedding two sentences: 1) I am happy; and 2) am Bob. These sentences are fed to a sequence-tor to-eeettt---eeeeeeeeeeeee model as
X = xX X. X.3] = [VI, Vam, Phappy) and X [
X. [Vi, Tam. UBoh)
where or, Vam, Thappy; and Unoh denote d-dimensional embedding for the tokens 'T' 'am', 'happy', and 'Bob', respectively. Since the word 'I' occurs in different contexts in these sentences, in order to implement arbitrary sequencc-to-sequeeee the two occurrences of 'I' to different values. We formally define this requirement below.
functions, the sequence-to-sequenee to-sequence model should map
Definition 3.1 (Contextual mapping). Consider finite set IL Rdx=, A map q L IRI*
defines a contextual mapping if the map satisfies the following:
1. For any L the entries in L) are all distinct.
2. For any L, L' E L, with L + L', all entries of q(L) and q(L') are distinct.
In other words, a contextual mapping maps each token (column) of E to a unique value which depends the entire L; result, capturing the precise context of L. This allows the subsequent
4

---

Published as a conference paper at ICLR 2020
token-wise function (e.g., defined by the feed-forward layers in of Transformer networks) to realize the outputs of any arbitrary sequence-to-- to-sequence functions.
At the first thought, because this can capture the one-word difference (c.g., "happy" vs. "Bob") in two different contexts. However, if there are multiple words that different, it is not guaranteed that the average will be different. Indeed, requiring unique mappings for all the tokens for any change in any number of tokens, a steep requirement.
consider getting contextual mapping by simply averaging all the tokens,
While the self-attention layer does consider pair-wise interactions among different input tokens, not clear if this weak form of pair-wise interaction with shared projection weights is sufficient to extract the underlying context. The following result, which we sketch here, shows that self-attention layers can implement a permutation equivariant contextual mapping over almost all elements of grid in [0, 1jaxn, We defer the full statement to Section 4.2.
Lemma 6 (informal). Consider the grid G5 := 6..6,.... -8] 5]dxn. Then, there exist a function Ic Rd + Rdxn composed o - self-attention layers (h = 2, 1) and a vector R such that a ((L) u² (L) satisfies the following properties, for subset G3 G8 that contains almost all elements of Gg:
1. For any L E GG, the entries of ((L) are all distinct.
2. For any L, L' E EG such that L is not a permutation of L', all entries (D), ((T)) are distinct.
Lemma 6 shows that a series of self-attention layers can implement contextual mappings, despite the apparent restriction that each of them can only capture pair-wise interaction. However, the restriction of permutation equivarance still exists because attention layers are inherently permutation equivariant. Coupled with the ability of token-we -wise feed-forward layers to map different values in ((L) to arbitrary output values, we can prove universal approximation capability of Transformers.
3.2 PROOF OF THE UNIVERSAL APPROXIMATION THEOREM (THEOREM 2)
Next, we outline the proof of Theorem in greater detail. We refer the reader to Section C for the proof of Theorem 3, since it is modification of Theorem 2. Even though Theorems 2 and do not specifically mention the required depth for approximation, our proof techniques do characterize it, and we show that our construction is tight in the number of parameters. We defer the discussion of depth to Section 4.4.
Recall that we want to show that given function FPE. we can find a Transformer network E 2,,1,4 such that (f S Without loss of generality, we can assume that the compact support of is contained in [0, 1]axn, We achieve our desired objective in three key steps:
Step 1. Approximate FPE with piece-wise constant functions. We first use (a variant of) the classical result that any continuous function can be approximated up arbitrary accuracy by piece- wise constant functions. For > 0, we define the following class of piece-wise constant functions.
FPE(() = ( {s: Σ LEGs A. (X E Sz) ʃ is permutation equivariant, At E Rddx
where G6 = {0, 8, 6,....1 1-....,_8Jdxx 8jdxa and, for grid point L E Go St. == 11 II [Li Ljjt + 5) [0, 1]dx denotes the associated cube of width 5. Let J FPE be such that df S e/3.
Step 2. Approximate (0) with modified Transformers. We then consider a slightly modified architecture for Transformer networks, where the softmax operator o||] and ReLU(-) are replaced by the hardmax operator o [-] and an activation function & E Φ respectively. Here, the set of allowed activations & consists of all piece-wise linear functions with most three pieces, where at least one
piece is constant. Let h.m."" denote the function class corresponding to the
functions defined by the modified Transformer networks. The following result establishes that the modified Transformer networks in 2,1,1 can closely approximate functions in FPE(O).
Proposition 4. For each J E FP(()) and 1 S < oc, 3g E 2,1,, such that dJ, g) = ((8/p))
Step 3. Approximate modified Transformers with (original) Transformers. Finally, we show that g E T 2,1,1 be approximated by 22.1,1 Let g E 2.1144 be such that dp(g.9) e/3.
5

---

Published as a conference paper at ICLR 2020
Theorem 2 now follows from these three steps, because have
(p. s dp(s.)) + d,( dp(r.g) s 26/3 O(0")) Choosing å small enough ensures that d (f. e.
We refer the reader to Sections B. and B.2 in the supplementary material for the formal statements and proofs of Steps 1 and 3. respectively. As for Step 2, which is the most critical step in estab- lishing the universal approximation property of Transformers, we provide a sketch of the proof of Proposition 4 in the next section, and refer the reader to Section B.3 for the complete proof.
4 PROOF SKETCH OF PROPOSITION 4: DIFFERENT ROLES OF TWO LAYERS
As mentioned earlier, the heavy parameter sharing in Transformers makes the goal of universally approximating eqquence-to-seqenne functions seemingly difficult. Both the self-attention and the feed-forward layer weights inside Transformer block are fixed across n tokens. In this section, we show that Transformers are able to overcome this architectural constraint, and compute contextual mappings of the entire imput sequence just based on the pair-wise interactions. The token-wiss feedforward layers then transform these contextual mappings to the desired output sequence.
We highlight these inner workings of Transformers en route to proving Proposition 4. We want to show that given piece-wise constant function E FPE((), there exists modified Transformer network T that closely approximates f. We achieve this goal by establishing the following three claims, which correspond to Lemmas 5, 6, and
Given an input X E RRdxn series of feed-forward layers in the modified Transformer network nnd 0, 1 8]dxn.
can quantize X to an element L on the extended grid G := {-8-nd
2. Next, series of self-attention layers in the modified Transformer network can take the input L and implement a contextual mapping q such that, for L and L' that are not permutation of each other, all the elements in (L) and q( [L)) are distinct.
3. Finally, series of feed-forward layers in the modified Transformer network map elements of the contextual embedding (L) to the desired output value of f E FPg the input X.
Before discussing these three claims in detail, we note that even though a Transformer network stacks self-attention and feed-forward layers in an alternate manner, the skip connections enable these networks to employ composition of multiple self-attention feed-forward layers. Further- more, as alluded earlier, these three steps clearly highlight the different roles that self-attention and forwaro layers play in realizing the ability universally approximate sequence- ceseoncetto-eqqeeee oo-sequence functions: 1) self-attention layers compute precise contextual maps; and 2) feed-forward layers then assign the results of these contextual maps to the desired output values.
4.1 QUANTIZATION BY FEED-PORWARD LAYERS
Since our objective in Proposition 4 is to approximate the function J E FPE(O), which takes a con- stant value on the cubes S, ss the (modified) Transformer network approximating f first quantizes the input X according to these cubes. In particular, want each input X Sr, be mapped to the point L. The following result shows that modified Transformer network can indeed implement this quantization map with a composition of multiple feed-forward layers.
Lemma 5. Consider a scalar quantization map ent. F + f-8 1----nd 0, 1- 8):
ifkō < < (k: + 1)5, k= 0,... 1,80..,1/--1,1
ent (f) = 1-8-nd
otherwise.
There exists function 9q Rdxa + G composed of & + token-wise feed-forward layers with
r = 1 and activations in 4, which employs the scalar quantization ginh to each entry of its input.
As desired, the function Bg maps any X E Sr. to L. Furthermore, any element of X not in [0, 1].
the element is mapped -8 a-dd indicating that X is outside the compact support of f = FP)(

---

Published as a conference paper at ICLR 2020
4.2 CONTEXTUAL MAPPING BY SELF- ALTENTION LAYERS
In this subsection, we show that the (modified) Transformer network can compute contextual map- pings (cf. Definition 3.1) from the output L E G3 of the map ge (cf. Section 4.1) by using a com- position of self-attention layers. The following lemma, sketched earlier in Section 3.1, shows that the (modified) Transformer networks can implement a permutation equivariant contextual mapping over almost all elements of Gs, while mapping the rest of elements in G3 to disjoint set.
Lemma 6. Consider the following subset of G {0, 60, 83áxn:
Gs :- (L E Go L.,i + L.j for all + j}.
Assume that 2 2 and 8-1 2 2. Then, there exist function ge: R - R composed of 6 + self- self-attention layers (h 2, m = 1) that employ the σμ operator, a vector u E R" constants t,,t, t, IR (0 < < t,). such that q == u" (LL) satisfies the following properties:
1. For any L Gs, the entries of ()) are all distinct.
2. For any L, L' E eG& such that not a permutation of L', all entries of (L), Q(D)) are distinct.
3. For any L E Go all the entries of (L)) are in tr].
4. For any L E G3 Go. all the entries of q( ()) are outside [ta, trl
At this point, few remarks about the result in Lemma 6 are in order. First, since the Transformer networks are bound to implement permutation invariant maps, we require the Property 6.2 to hold for the pair of sequences that cannot be mapped to each other via permutation of columns. Further- more, the self-attention layers implement the desirable contextual map for only Gg S Gs, where all columns of L are distinct. Note that for small 8, G \ Gs constitutes a negligible fraction of G, because |Gg G = ( GG))) The function in Lemma 6 maps the elements of G3 \ GG outside [tr,tp] the interval where the outputs of the contextual mapping for G3 reside.
4.2.1 PROOF SKETCH OF LEMMA 6
Since Lemma 6 is one of the major technical contributions of this paper, we provide short sketch of its proof. The complete proof is presented in Section B.5. For simplicity, we consider the case d = 1, so the input L E G3 is row vector of length n.
The key idea of the proof is that, using two attention heads of size 1, one can implement a self- attention layer that shifts up imput entries that are in specific interval, while leaving all other entries intact. We call this the selective shift operation. Since the entries in L are quantized, we apply the selective shift operation to 0, .... 1 5 using 1/5 attention layers. Interestingly, the value of the largest output entry after these operations is unique for each L E G up to permutations. Using the largest entry, one can add one last layer that shifts up the entire matrix and outputs (L)) that satisfies Properties 6. and 6.2 of the lemma.
More concretely, the following function V/ ; 1R_ - RIx", parametrized by b, D E R satisfying b < 8', can be implemented with two attention heads of size 1 with the hardmax (σH) operator:
V Z: b, b'), max ZI,x - ming Z1,6
if < Zz.5 < b, if 11 < or 21,j > b.
If we define attention layer of the form Z + 2 (Z,b, b)), then any entry Zı.; in (b, b') is shifted up by max; Z1.k minn Zı,t, while all the other entries stay untouched. We can choose b and b' to selectively shift certain entries, hence the name selective shift operation.
We stack 1/5 self-attention layers, with attention parts 8 (-; - + 6/2) for each 1 e 6..6...., 8). in increasing order of I. With these layers, we can apply the selective shift operations to input entries of values 0, .,...,, 8. To see how the shift operations modify the input, now consider n = for simplicity, and let = [41 le] E GG Without loss of gener- ality, we can assume < lz. The selective shift operation applied to 4 first, shifting it by 8-1 5-i( min L) = 8 8-1(12 11), resulting in I = I + 8-1(12 - 4)) > lz. After that, the operation on la shifts it by 8 (T) - 12). Thus, the first 1/5 layers map L = [4 1z] (h < 12) to
12 + (8-2 8-1) 4)) i 7 [z] [4 5-11(2 11)
7

---

Published as a conference paper at ICLR 2020
We can show that the map from [1 [2] {L Gs 1 < (2) to 21 is one-to-one, and that < / < l2 < 8-2 We then add one last layer that shifts all positive entries of by 8-3 max L = 5 5-32, whose output we denote by ((L) = [5-3]2 + 5 8-312 [2]. All entries of (L)) are in [8 16-% 8-01/ + d and this interval is disjoint for different L's because + 12 one-to-one. Thus, (L) satisfies Properties 6.1 and 6.2 of the lemma. The remaining details are in Section B.5.
4.3 FUNCTION VALUE MAPPING BY FERD-RORWARD LAYERS
This brings us to the final step, which demonstrates the key utility of the feed-forward layers. After the contextual mapping by self-attention layers, each token captures the entire context available in the imput sequence. The following result shows that token- -wiee application of composition of feed-forward layers can map these tokens to the desired output values required by the function J.
Lemma 7. Let 9c Rdxn + Rdxn be the function from Lemma 6. Then, there exists a function gv _ddx Rd composed of (n(()) dd /n!) token-wise feed-forward layers (r = 1) with activations in & such that gv is defined by a token-wise functtton gv gkkn. Rdd - IRd on each column,
(x)
[g (Z.1)
gr [Z.
where for all j E (I.....n..
(ALdis
if E Go,
ge ku (ge( (o)ss)
Od if G Gs.
4.4 TIGHITESSS OF CONSTRUCTIONS
We showed in this section that Theorem 2 requires O(n(1/8)dm/m) Transformer blocks for ap- proximation, where 8 is the width of the cubes. Each transformer block is of constant width, so it has ((d) parameters; this means that the total number of parameters is (dd)(1/))dm /n!). We note that this exponential dependence cannot be avoided in the worse case. If we assume continuity without any additional smoothness, quantizing the domain cubes and approximating the function with constants require memorizing (output dim) x (num cubes)/n! n!! real numbers, where the factor of 1/nl is due permutation equivariance. Thus, Theorem 2 is optimal in the order of parameters.
If we compare with the residual network result (Lin & Jegelka, 2018), we can consider "flattening" X into dn-dimensional vector and fitting the function. The proof technique in (Lin & Jegelka, 2018) requires ((1/5)da)) layers, where each layer has (dd)) parameters: the total parameter re- quirement is (dn(1) (1/8)ddd)) ddm). This shows that Transformers can approximate permutation equivariant functions in more efficient way than residual networks.
In Section C, our proof of Theorem shows that we require O(n(1/8) layers to approximate con- tinuous (not permutation equivariant) sequence-to seuuence-o-ssquence sequence functions. As seen from the argument above, this construction is also optimal the order parameters.
5 DISCUSSION AND EXPERIMENTS
As detailed in Section 4, the ability of the self-attention layers compute contextual mappings plays a crucial role in the universal approximation property. Interestingly, our analysis shows that replacing the dot-product attention in Transformers with any other component capable of computing contextual mappings should preserve this universal approximation property. This leads naturally to questions about the alternative architectures that realize certain kinds of contextual mappings at dif- ferent computational and memory costs. We explore and discuss some examples of such alternatives in this section. Our preliminary empirical study demonstrates their practical utility.
5.1 BI-LINEAR PROJECTION
Given token embeddings X as input, the bi-linear projection layer computes the following update.
BProj X) = X + Wo X Wp.
The bi-linear projection layer (Gong et al., 2013) is motivated from the ability of random (Gaussian) matrices map sparse differences to dense vectors (Ailon & Chazelle, 2009). If there two input
8

---

Published as a conference paper at ICLR 2020
Parameters in millions
Parameters in millions
110
109.5
208.3
108.3
107.8
107.1
110
109.5
108.9
108.3
207
107.1
89.5
84.0
84.5
MNLI Accuracy
SQUAD F1
02.5
83.5
02.0
83.0
67.5
SQUAD
80.5
82.0
87.0
00.0
Conv layers
# Conv layers
(a) SQuAD
(b) MNLI
Figure 1: Performance of hybrid models constructed by first taking BERT 12 layer Trans- former model, and replacing the self-attention layers with depth-wise separable convolution layers, varying number of the Transformer blocks closer the input. Surprisingly. replacing self-attention layers with convolutions improves the performance, while replacing more hurts the performance. This suggests both that Transformers have functionality beyond just computing con- textual mappings, and having simpler layers to realize contextual mapping can aid Transformers.
X2 is sparse; however, after random projection, the difference (X1 Wp will be dense, and the numbers are distinct with high probability, implementing form "pair-wise contextual mapping." although different from the contextual mapping in Definition 3.1.
contexts X1 and X2 that differ in one token, their difference X1
This layer advantageously incurs smaller number of matrix multiplications compared the dot- product attention. That said, the number of parameters in this layer depend on the sequence length, making it harder reuse the model across tasks with different input sequence lengths. Moreover, the weights used to compute the contextual embeddings (Wp) are independent of the inputs (X), whereas in self-attention the weights (o X (WXX), X]) be addressed by replacing the linear projection with depth-wise separable convolution layer, which discussed in the next subsection.
W
depend on X. The first drawback can
5.2 DEPTH-WISE SEPARABLE CONVOLUTIONS
A depth-wise convolution layer (Sifre & Mallat, 2014; Chollet, 2017: Kaiser et al., 2017) involves convolving each dimension of with a corresponding convolution filter of size
SepConv(2 (X) = + Wo (X Wc),
where We Rdxk and (X * Wc),. == X,, (Wo).:. Unlike bi-linear projection, this layer can be used across tasks with different input sequence lengths as the number of parameters are independent of the sequence length. While a single layer is unable to compute contextual mappings when the filter size is small, stacking multiple such layers can potentially provide a cheaper way compute contextual mappings. In fact, based on depth-wise separable convolutions, Wu et al. (2019) proposed light-weight dynamic convolution architecture that performs competitively with Transformers on machine translation.
5.3 EXPERIMENTS
We present experiments with these other architectures, with the goal of understanding the extent to which computing contextual mappings can capture the performance of Transformers. As discussed earlier, BProj and SepConv do not implement contextual mappings (cf. Definition 3.1), so we do not expect that either BProj or SepCony based models to have the same performance as the expensive Transformers. These models do not input dependent weights compute attention. and hence have weaker representation power. Instead, our goal is to see if we can use these cheaper layers to replace (some of) the expensive self-attention layers.
This guarantee only holds for finite (can be exponential in n) of fixed vectors in TR."

---

Published as a conference paper at ICLR 2020
We follow the experimental setting from Devlin et al. (2018) to train the Transformers, with the masked language model pre- training followed by a task specific fine- -hniit, and work with a 12 layer architecture based on BERTRASE: We present our results on question answering task (SQuAD) (Rajpurkar et al., 2016) and a sentence entailment task (MNLI) (Williams et al., 2018). In our first set of experiments attention layer in cl) We notice that, as expected, these simpler models have weaker performance than the self- self-attention layer. See Table in Section D for comparison of these models on MNLI.
train models that employ BProj and SepConv layers, instead of the self-
Next, we swap a varying number of the first few self-attention layers in BERTRASE with SepConv, implemented with filter reuse across dimensions (Wu et al., 2019)². Fig. 1 illustrates the perfor- mance of these hybrid models. Interestingly, models with 1 or 2 convolution layers and rest the self-attention layers, perform better than models with only the self-attention layers. Note that, re- placing self-attention layer with SepConv also reduces the computational cost and the number of parameters. One explanation we have that the first few attention layers tend to attend broadly to the whole sequence (as empirically observed in (Clark et al., 2019)), and the cheaper convolution layers can perform this job more efficiently. A detailed evaluation of such hybrid architectures will be interesting future research.
Our experiments also call for a deeper understanding of the exact nature of the embeddings com- puted by practical attention models. Since Transformers in practice have fixed depth, we believe that they might not be able to exactly implement contextual mappings as we defined in Defini- tion 3.1. However, there is some preliminary empirical evidence that Transformers do implement some sort of "contextual mappings." For example, Fig. 4 of Coenen et al. (2019) presents visualiza- tions of embeddings of a single word in different contexts (sentences). They experimentally notice that Transformers, in addition to computing contextual mappings, also map word into semantic clusters. Formalizing and evaluating this property of Transformers is an interesting direction for future work. We again note that Wu et al. (2019) have proposed an alternative way to compute such embeddings based on dynamic convolution layers. Evaluating the mappings computed by these models should shed more light on the workings of attention models and inspire efficient and better performing architectures.
REFERENCES
Nir Ailon and Bernard Chazelle. The fast Johnson -oidnsoon-inddenstaauuss transform and approximate nearest neighbors. SIAM Journal on computing. 39(1):302-333 -222, 2009.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does BERT look at? an analysis of BERT's attention. arXiv preprint arXiv: (906::1990.43333 2019.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wat- tenberg. Visualizing and measuring the geometry of BERT. arXiv preprint 1rXivv1906.027/55: 2019.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics control, signals and systems, 2(4):303-314 1989.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03811 2018
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 100004000 2018.
Yunchao Gong, Sanjiv Kumar, Henry A Rowley, and Svetlana Lazebnik. Learning binary codes for ligh-diimnsional data using bilinear projections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 484-491, 2013.
Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint 1110. 2017.
We refer Section D for complete description of the setup.
10

---

Published as a conference paper at ICLR 2020
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pa-
pers), pp. 4139-4138, 2019.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4 (2):251-257, 1991.
Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. Depthwise separable convolutions for neural machine translation. arXiv preprint arXiv: 170.003999 2017.
Hongzhou Lin and Stefanie Jegelka. ResNet with one-neuron hidden layers is universal approxi- mator. In Advances in Neural Information Processing Systems, pp. 6169-6178, 2018.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv: 1692, 2019.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. In Advances in neural information processing systems, pp. 6231-6239, 2017.
Minh- Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1412-1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119 2013.
Jorge Pérez, Javier Marinković, and Pablo Barceló. On the Turing completeness of modern neural network architectures. arXiv preprint arXiv: 19110049, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. Technical Report, OpenAl, 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical Report, OpenAI, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Akiyoshi Sannai, Yuuki Takai, and Matthieu Cordonnier. Universal approximations of permutation invariant/equivariant functions by deep neural networks. arXiv preprint arXiv: 2019.
Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for image classification. Ph. D. disser- tation, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. arXiv preprint arXiv:1906.04250 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, Association for Computational Linguistics, 2018. URL http:/ / / a el web. wwt.//orggaaoooo..o hes//nnhholggyy /N1 /N1 1101.
11

---

Published as a conference paper at ICLR 2020
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430, 2019.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V.
Le. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint
arXiv:1906.08237, 2019.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19-27, 2015.
12

---

Published as a conference paper at ICLR 2020
A
PROOF OF CLAIM 1
Suppose X P was given as input, where P is permutation matrix. First note that
XWXX) xP) PT (W (W
After the softmax operation, we get
a [P" wF
Then,
Attn(XP) XP
Σ
W (WW, ( Poo(( Attn(X),,
where we used PP" I. Permutation equivariance of the token-wise feed-forward layer can be shown similarly:
FF(XP) Attn P + W2 RcLU(W, Attn(X) b,,, b
Attn X + W2 ReLU(W; (ttn(X) + b,
where ReLU(XP) ReLU(X) was used. This analysis shows that the function class T T.mm.. is restricted to permutation equivariant functions.
B PROOF DETAILS OF THEOREM 2
We first define some additional notation. For a, b N where a S b, let [a] = (l...., a} and [a b] fa, + 1.....b 1, For a, b,b.c R where b is integer multiple of 0, we write (a c: b] = {a, a cc a+ bc...., b.
B.1 APPROXIMATING FPE WITH FrE(O)
Lemma 8. For any given FpE and 1 S < co, one can find a 8* > 0 such that 37 E FPE(8*) which satisfies dn f) I 5/3..
Proof Since f Rdxn_ -> Rdxn is a continuous function with compact support, the function is uniformly continuous. Since continuity is defined using entry-wise lp norm, and entry-wise lp norm is equivalent to entry-wise too norm when the number of entries are finite, uniform continuity implies that
Ve 0, 36 > 0 such that VX, Y, | 11x < 1 S(X)),, <e
This means that given any c/3 0, have such 5 0. Using this 5, we create grid Go and corresponding cubes Sz. as described in the main text. For any L E Gg. we define Cz E S be the center point of the cube Sr. Then, we can define piece-wise constant approximation f(X) Elec, f(Cz)1 (X E Sz). Note that, for any X SL. have || Czll.o < ö, so by uniform continuity, we have [|f(X) f(x) llf - f(CL)ll, < e/3. This proves that f) </3.
As for permutation equivariance, since í is permutation equivariant, we have for any permutation matrix P. For any X St, we have X E SLP, so
T(XP) ((CLp) J(C,P) ((c
Thus, the approximation 7 is also permutation equivariant. This proves the lemma.
13

---

Published as a conference paper at ICLR 2020
B.2 APPROXIMATING T 2:1,1
WITH T
Lemma 9. For each g T T and 1 1 oo, 3g T2.,, such that dp(3.9) 9) S e/3
Proof Recall that mh. refers to the class of functions representable with composition of Trans- former blocks with h heads of size m in self-attention layers and r hidden nodes in feed-forward layers. The same notation holds for the modified Transformers T ".."..
Note that the softmax operator matrix A can be made arbitrarily close to hardmax by scaling up A. That is,
dAAi + [ as X co.
This means that by scaling up parameters inside a, we can approximate on arbitrarily closely. Thus, the modified self-attention layers can be approximated with the original self-attention layers of the same number of heads h and head size m.
Also, any arbitrary (possibly discontinuous) piecewise linear function e E o can be approximated arbitrarily closely by four ReLU's. Note that ő E & as at most three pieces, and at least one of the pieces is constant. For example, consider the following function á 4:
(b a2t
if t < Ct
x(1)
a22++ + by ast + by
if c S t < C22 if t.
This function can be approximated by four ReLU's, claimed by the lemma:
a2 b2 bi
0201 b2 bi ReI.U(t-
(() bi
ReI.U(E- - c +c) + 02 c
0302 + by az(02 - -) b2 02)
-2 1120ttc220
c
0302 by u2(c2 -6) 02
03
ReI.U(L - 02)
b1
ift < -6
220 220 by (1 (1) + a2201 b2
if 5 < C1,
agt + b2
if < 02 6,
ha -gll - c2) + a302 + by
if - e t < c2.
0gt + by
if S.
Also, make 0, can approximate & as closely possible using क. The cases where the second or third piece is constant can be shown similarly. This means that the modified feed-forward layers (whose activation is & E ②) with single hidden node can be approximated with the original feed-forward layers (RcLU) with four hidden nodes.
Thus, given any ğ E 2.1.1 there exists function g E 2,,1,4 arbitrarily close to g. by appropriately choosing the parameters to be large enough. This finishes the proof.
B.3 FINISHING PROOF OF PROPOSITION 4
As we have already discussed in Section 4, we establish Proposition three steps:
1. Given an input X, a group of feed-forward layers in the modified Transformer network can quantize X to element L on the extended grid G {-6- .0, .,...,11 - 6)dxe,
2. Next, a group of self-attention layers in the modified Transformer network take the input L and produce desirable contextual mappings (LL) such that, for L and L, that not permutation of each other, all the elements in (L)) and q( L) are distinct.
14

---

Published as a conference paper at ICLR 2020
3. Finally, group of feed-forward layers in the modified Transformer network can map ele- ments of the contextual embedding q( L)) to the desirable values, i.e., the output of f E Fre on the input X.
These steps formally stated in Lemmas 5, 6, and 7 in the main text. We present the proofs of these lemmas in the subsequent sections.
With the results established in these lemmas, are now equipped with all the tools necessary to complete the proof of Proposition 4. Let us recall the functions Pn and from Lemma 5, 6, and respectively. We now show that the (modified) Transformer network g gv ge 9q approximates the underlying peicewise constant function E Fre over all points in its support except for set of of measure (68)).
Consider point X & Thus, follows from Lemmas 6 and 7 that
[D, 1jdxn, where L E Gs. By Lemma 5, we have that Dq (X) L.
gv go 94 (X) = gv o ge (L) [gt (gc()..., (1)))
tku (go (L) = AL. g
On the other hand, any point X E ULEGIVÖ Sz U (RRd 10, 1jdxn) is mapped by 9q to L E G G3 as a result, we get Bv Io 9q X 9v (L)
X) A = f(X) for X E ULEE, St. and 0 everywhere else. Recall that / has its compact support in [0, 1js, thus bounded; i.e., there exists B 2 0 such that [IF(x))p < B. The modified Transformer network g takes the same value as f on all points in [0, 11dh except for set ULEGAYä SL that has measure O(8)). This implies that (f.F) s (Br8d) = ((8/P))
Therefore, have (X) g. 9g
B.4 PROOF OF LEMMA 5
The proof strategy is simple; using token-wiss wise feed-forward layers, we implement the quan- tization funetion gem that works on the first row of the input. Then stack another + layers that quantizes the second row, and so
Given input X, we first start by clipping X1,; in the set (-oo, OJUL,, +00) and mapping the intervals to - 8-nd This can be done by the following layer:
[ - 5-t- 5-nd
ift < 0r 1, otherwise.
+ Z + e ( (((e())) 2), s())
0
Next, add 1/8 layers of the following form, for h: 0, 0, 1- 8.
0
8< 0r
Z Z e(4) el)) ((( k11 d(t) {-t 0 Ost<b
Each layer quantizes Xi,: in [k6, kō + 5) to A5, without modifying other intervals.
Note that both ø's used in this construction least one of them are constant. Thus, both ø's are in +. We can repeat the same thing for the other rows, and at the end we will get map from d to G..
piecewise linear functions with three pieces, and
B.5 PROOF LEMMA
Selective shift operation. Before starting the proof, we first describe the key component of our proof, which we refer to the selective shift operation. Consider the following function, which can be expressed with a multiplicative attention head, with head size = 1 and hardmax оH:
(Z:bq) (1) Zon Z) u T bb 1x]
where u e IR is a vector that we will choose later, and e!)) = (1, 0, ......0) E Rd is the standard basis vector.
15

---

Published as a conference paper at ICLR 2020
To see what this function computes, first consider the j-th column of the attention matrix: (u" z) (u" Z,, - -q).. Note that, if u" Z., > bq. оh will calculate arg max u Z, whereas a Z < bq. will calculate arg min. Therefore, the (1, entry of / Z bQ)
E Rdx= can be
written
maxx u Z ming u² Z
if u Z > bQ. if uu Z bQ.
1 (Z:bol,, = x² Zonl(u' Z) (u" Zu. bQ)l
for j E [n]. Note that due to e all rows v) Z (q) except the first row are zero. From this observation, one can define function parametrized by bi and l/2. where ba < <b/g which consists of two attention heads:
/ (Z 1/2) vb (Z:bq) (Z:))))
v (Z bQ, 1/21.. maxx u ==Z,t Z,,k ming u Z,k
if by < u² Z
j < Vgg
u Z < bo or Z. > 1/6-
What this means is that, if we define attention layer of the form Z Z v (Z; 1/2): then any column Z satisfying w" Z E (bQ. 1/2) shifted up in its first coordinate Zi.j by max; x² Z. ming u² Zi,k, while all the other coordinates stay untouched. We call this the selective shift operation, because we can choose by and 1/2 to selectively shift certain entries of the input.
Bijective column id mapping. Recall that the input to this step is from the range of gu (Lemma 5), which is G3
{-5-nd, 0, o,.,.... djdxn_ Now consider L E G3 and u =
11,6-,8-2 3=?.....8600 8..11).
For any j [m]. it is easy to check two following facts:
1. If Li # -8-m nd for all i [d]. i.c., L.j E 60,6...., 1 8)d, then u² L.g E [0 8-d+1 -0. and the map Lij + "² Lig from {0, o,.., .6.....--11.... 832 to 10 ô 8:000 d+1 8) is
bijection.
2. If there exists i [d] such that Lij -5-nee then u L. - 5- < 0.
Therefore, one can say that u² Lij gives the "column id" for each possible value of L (0, 00,,,...,11 1-
The rough idea of the construction is to apply the selective shift operation to each column id, by setting u in the definition of (-) be (1,0-1,0-2 1 8-.1) and choosing bg = 6/2 and 1/2 1 5/2 for each E [0: 8-d+1 8). More concretely, we stack (1/6)d attention layers, with attention parts 5- g-d <T; 5/2, + 8/2) for each / E [0: 5 : 5-d+1 87, in increasing order of 1. After that, we add an extra single-head attention layer with attention part 8 (-(++))d (x+1))((:;0)).
We now divide possible input values L E G into three disjoint categories, and show how these layers change the input values at the end of all the layers. Recall the hierarchy G C G6 G... The categories are defined as follows:
1. L E Gg. All entries are between and - 8, and all columns are unique.
2. L G G All entries are between and 6, but there duplicate columns.
3. L G
GG. The point has least one entry that equals -8 --
B.5.1 CATEGORY
In Category we have L Gs. Let u L. Due permutation equivariance, can assume without loss of generality that I,'s in increasing order: 4 l2 ... < In. The first (1/8)d layers sweep the set [0 8 5 8- - 8] and apply selective shift operation each element in the set. This means that selective shift operation will be applied to l first, then 22, and then Ig. and so on, regardless of the specific values of 1,'s.
16

---

Published as a conference paper at ICLR 2020
First shift operation. In the first selective shift operation, the (1, 1-th entry of L (L1,1) is shifted by the operation, while the other entries are left untouched. The updated value LT,
Σt 1, 5 (max; u L, ming u L.,k)= L1,1 + +8 11) Therefore, after the operation, the output of the layer is [I,1 L22 L,n] and the new value of the first column _ results
u T _. = -111 Σ
it L L1,1 + o- "(n - 11) +
Σ
i+1 11,1 =1 + "l 11)- 8-
1=2
i=2
Let us denote the updated "column id" u² L,,1 as IT. We can show that I 7, because
1==44 4 8- (1) 2 2 8 = 8- In
Therefore, after updating.
max T [ L L = max (11, 12 ...... 1n = =T,,
and the new minimum is l2.
Second shift operation. entry 2 will be shifted. The updated value I is
The second selective shift operation is applied to l2, by which only one
I. = L12 + 8- (T - 12) = L12 + 8 (4 - 12) + 6---20 g-2 11) After updating, the new inner product of and L. results in
To = u L. = 12 + 8 4 12) + 5 5-2d(h - 11)
We can show that 7 < T2, because
1 + d 11) < 12 + 01 122 + 8 2-2d((, 11)
- ()22 - 41) < 66 1) (4n - -1),
and the last inequality is true because 5-d > and In > 122 Since we have 7 < Tz. and the new maximum in u² [ L2
L.3
L.n is now 12, and the new minimum is 13.
Repeating the process. More generally, we repeat this process, and show that the j-th shift operation shifts I by 8- 14), and results in the new column id
1-1
T, == u L..j = + Σ
kd (11 - + 5 id( 11)
k=1
In the general case, 7 < holds j = [2 n]. because
Ī 1 1j-1
8 kd+d (ly-k (1-++1)++++++ (0-4))))----1)6- 11)
k=
< = 1 + Σ
kd (1, - ( + 8- -1)
j-1 8 kd 1111--11111----1--- <8-00-)) 1)(h- - -1),
k=1
and the last inequality holds because
8 8-
4 (1) (i-11 (0y-441- (1-x)
Σ kd+d (1y-b11 -
Therefore, after the j-th selective shift operation, T is the maximum among (T..... .T, l........ 3 and 1,+1 is the new minimum, which makes us possible to continue the process until the n-th operation.
17

---

Published as a conference paper at ICLR 2020
After shift operations. As result, after the whole sweep from 0 to 8-4+1 8 by the first (1/8)d layers, total of shift operations are applied, and the input L is mapped new point L, where w = 12
In] and < < < <....... <bn
We can now prove the following technical lemma, whose proof is deferred to Appendix B.5.4:
Lemma 10. After n shift operations, In u² L... satisfies the following bounds:
8-1n-1)dt118- - -1) S In s 8- nd+1 10-0-111111 - 5(0- 1)2.
Also, the map from [1 l2 one-1o-one.
Ln] E [0 [ 8- 6] (where I < l2 < (n) to to
Global shifting by the last layer. As mentioned earlier, after this sweep, there is another attention layer with attention part å (a+1)d (:; 0). Since l < Ins what it does to L is that it adds
5 5-(++1)o max; u" L = 5 (-(++ Dd to each entry in the first row of L. The output of this layer is defined to be the function ge (L).
Now, in summary, for any L E GG E [d], and j E [n]. we have
L + Σk=
Add 8 (==x+1) 8-15855a 1d 41) 5-(11)
if 1, if E [2, d.
go (D))) and for any L E Go and j E [n].
-1 Ti.s
nT go (L) =T, - 8-(n11)
Checking Properties 6.1 and 6.2. Given this result so far, it now left to check if the constructed network really permutation equivariant contextual mapping, i.e., if satisfies Properties 6. and 6.2 in Lemma 6.
First, for any L E G& Property 6.1 holds because we already know T1 < Tz < < In, so they are all distinct. As for Property 6.2, note that the upper bound on ln from Lemma also holds for other Ij's, so
u² ge (L)s (0 (o-(n+)) (1++1)47 + 8- (1(1))641)
for all j E [n]. Now, from Lemma 10, two L' E Go (that are not permutations of each other) map different in and I, and they differ least by 5. This means that two intervals [8 (6-(+1))ef 8 (-(n+1]dT, -8--------- (m+1)d+1] and [8 (m-(n11)d 8 -m++1)df + ( (a+1)d+1) are guaranteed to be disjoint, so the entries of uT gL( and u uT (LL)) are all distinct. This proves Property 6.2.
Therefore, we finished showing that the map g-(-) we constructed using (1/6)² attention layers implements a permutation equivariant contextual mapping on G3.
Checking Property 6.3. It is now left to check if the map ge satisfies the other properties. At this point, we can check Property 6.3. From u² ge(LL... E [8- (8-(n+1)df, 5-(11)) 8- (1+++))d-1 and Lemma 10, we can show that for any L E GG. we have
8 2nd 1 - 1) s u" go (L). < (8- ndd- 1)8--8 -88---1111)) -1 + 8-(m+1)dH
8 (2n+ 18-d
where we used 8-1 2. This proves that all u² go (L)., are between 1 = 8 3-2nd+1 1) and t, = 8 62(2n+1)d+1 (8- 1). For the remaining input points L G \ GG.. we will check that
u ge (L).,, is outside the interval [t, t (Property 6.4).
B.5.2 CATEGORY
In Category 2, we have E G Gg. Here, all entries are between 0 and 1-8, 5, but there are duplicate columns. Again, let 1j u" L.j, j, and assume without loss of generality that L < l2 S In-
18

---

Published as a conference paper at ICLR 2020
For the input L in Category 2, there exist some j.j' |n|, j * j'. such that 1 1.. This means that when the imput passes through the attention layer 8 (- L 6 6/2,4, + 6/2), the selective shift operation for l is applied to both j-th and j'-th columns; the two columns are coupled together. More generally, suppose have n < distinct columns.
If n' = 1. the extreme case of = 1, we have max, 4 = miny 1,, so the selective shift operation applied at l, does not shift the entry at all; therefore, at the end of the first (1/8)* attention layers. L= L.
If <nn' n- When 1 < n' S n nn-1, let the n' distinct values of 1, be 11..... K.... The shift operation is applied n' times, to ,'....,,'y.... and shifts one or more entries at time. After the first (1/8)² layers, the output L has n' distinct 1, u u² L... 0 4 S و values are the same as the numbers we get when we apply shift operations to a length-n' sequence [ [. Then, applying the same calculations from Category shows that
I hn whose distinct
Tn = u L. Cn +
(Co - (+1 + (( -G),
and it follows from the upper bound in Lemma 10 that
1, 5 1
5(8- - 1)2 5 (m- 1).
Note that the RHS matches the lower bound in Lemma 10. This implies that the value I, cal- culated from the input L E Gs \ Go (Category 2) is always strictly less (by at least 8) than that calculated from L Go (Category 1).
Checking Property 6.4. After the global shifting by the last layer with attention part 8- 8-(n+1)d ((;+)))(((;))) we get the output gc(L) which satisfies
u² ge (L).s = -T, + da < (8 (n+1dd + (1 - 1) 8(8-6 1)2)
8 2nd+1 (8-
t.
where the RHS is a lower bound on possible values of u² ge LD) for L E Go (Category 1). This means that the entries of u" ge (L) for Category are outside [4, 4x]. which satisfies Property 6.4.
B.5.3 CATEGORY
In Category 3, we have E G \ Gg the point L has at least one entry that equals to --- Let 1, n L. and recall that whenever column L:j has an entry that equals -8-nd we have Assume without loss of generality that S l2 S ... S ln-
1, = u L.. S -dd 66- -
Recall that the selective shift operation is applied to each element of 8 8-d+1 8). not to negative values. In case of Category 3, we have min; u L., = 4 < 0, and 4 never gets shifted upwards, so remains as the minimum for the whole time.
If all ly's are negative. In case where all 1 are negative, selective shift operation never changes the input L, so get L L. Since have u L 0% (entry-wise), the last layer with attention part 8- (x-+1)d 0) adds (-(n-1)d ming u" L. to each entry in the first row of L, further pushing to the negative side. Therefore, the final output (L) satisfies ut (L) < 0g < t,17.
If not all 1j's are negative. Now consider the case where at least one l; is positive. Let i be the index that satisfies 4-1 < 4. Then, selective shift operation does not affect 1 11....,4-11, and then it shifts 1, by
8 8 8- a L., min u² L.,e) 8 u 41) 8- - +1) 8 (1+1)d+1
where we used 8-1 2 at the last inequality. The next shift operations shift 4+1 ......... ..... Ln by even larger amount, so at the end of the first (1/8)a layers, we have 5 (n+1)d+1 < T < ... S In while 7, =1 1
for E i 1).
19

---

Published as a conference paper at ICLR 2020
Shifts by the last layer. Here, the last layer with attention part 8 (x+1))(((((000)00) 0) acts differently for negative and positive ij's. For negative 1j's, it adds 8 5-(+1))d ming T 8 (x+1)4 to 1 14.1. pushing them further to the negative side. For positive 1 the layer adds 5 8-(n+1)d max & 5 5~(x+1)d7, 2 82(2+22) to I,....,Th In so that they are all greater than or equal 8 +2n++))++1 Note that 8 6-(2++2) > tr.
Checking Property 6.4. Therefore, in both cases, we can see that the final output 9e 9-(L) ()) satisfies u" 9o (D). & [tr, t, for j E [n]. This completes the verification of Property 6.4.
B.5.4 PROOF OF LEMMA 10
Proof of lower and upper bounds on i are straightforward:
in = la + kd (la- 1-++1) + +6 8-nd (hn- - 11)
2 8 (n-dd
(h-k b--) + 8 nnn - 11) (0-nd (n(nd))d) 11)
28-(n-1)d+ (8- -111 1),
in v In +0 (0 - In) + 8 n-nd(I - 11 S 8- 8 (8-nd - (6 (ā- - -8)
8 1 (8- 1) 5(5 2d 28- d 1) 5- -nd (8-d d - 1) 5(5 dd - -1)2-
For one-to-one property of the map, consider 14 la ... n and [ 1/2 ing entries, which are mapped to L, and i,, respectively. Suppose T, ln By definition,
M with increas-
I P (In (,)) + 8 80 (1o-1 - In - 1/ (2)) + 8 2d (ln-2 1n-1 1n22 + 11-1)) +
+ (n- 1d (11 - 12 12 + (2) + + 1--nd((1 (hn - 1 - 12 + 11)) 0.
Now assume for contradiction that L, # ln Then, we have - d+1 + < L, - M, 8- 6. However, the remaining terms have "coarse resolution", and they can never cancel 4, - 1n and make the sum zero, because for example, 8 (1 ln 1--1 + (2)) can only have values 0, 0- - d+1 28-d+1 28--4+1 ..... Thus, In lt must hold and the first term must be zero.
Similarly, assume that 4n-1 4n-1- Then, the second term is in the interval [-8-26+1 2d+1 + 8-d+1 5 2d+1 5 d+11. Again, the remaining terms cannot cancel the second term, hence h- = ln-1 must hold. We can proceed this way, and show that lj = I2 must hold for all j E [n]] hence proving that the map is one-to-one.
B.6 PROOF OF LEMMA
Note that |G C} + 1)da, so the image of ge (Gt) (from Lemma 6) has finite number of distinct real numbers. Let M be the maximum over all these numbers. By construction Je. we know that M > 0.
To construct a function g that satisfies the statement of the lemma, we first implement the second part: 9thn (g. (L).g) = 02 if L E G \ &... Note from Lemma 6 that, for any L E GG we have uT ga (D).. [1,, t,] for all j. and for any L G \ G&, ut (D( & [ti, t,] for all j. Using this, we add the following feed-forward layer:
Z Z (M + 1)1mo(u Z), d(t) 0 if [4,4]]
if & [t, 4.].
Input to this layer is (L). If E Gs. then o(u² T ge (L)) OT, so the output stays the same as the input. L E G. \ GG, then olu" (L)) = 17, so all the entries of the input are shifted by - .M 1, and become strictly negative.
20

---

Published as a conference paper at ICLR 2020
Recall that by definition of Gs. all the entries of ge (L) for L Gg nonnegative. So the next thing to do is mapping all strictly negative entries to zero. This can be done in similar way as Lemma 5. For i E [d]. add the following layer:
Z z + e e (((e()) 6(1) = -
ift if 2 0.
00
After these layers, the output for L E G3 Go is zero matrix, while the output for L E G is (L).
AL. for L Gg. Up permutation equivariance, each different context L maps to n unique numbers u" gc(L), which are at least apart from each other. The idea of value mapping is to map each unique number the corresponding output column.
Now, it is left to map ge (L)
More precisely, choose any L E Go.. For each value of u" ge (e( j E [n], we add one feed- forward layer
1< < 8/2, 50
Z ((At) g0 (D))))(((u Z- u g. ().
2(t)
[
6/2.
the imput Z zero matrix, which is the case for E G G TT O Since t is much larger than 0, activation is all zero. Thus, zero input matrix remains the same at the output.
If the input Z is ge(L), where L Go is not permutation of L, then
olu² ((uFge(L) gc()) u² go ( 0
so (L( is left untouched.
If some other L is permutation of L, and L, L. then
o(u² (L) u² go (D).j = (e!0)²
so i-th column of ge(1) will turn to ge(L) ge (.) ((Ar).) 9e (D()) ge (D).: ((AL):,) (c()) (AL)
which is the desired output. conclusion, this layer maps the column Je (D).., (AT)... without affecting any other columns.
As seen above, need one layer per each unique value of u² (L) for each L E Gs. Note that there are ((n(1/8)d such numbers, so we can use (nn(1/)) /n!) layers to finish our construction.
C PROOF OF THEOREM 3
Proof of Theorem 3 can be done in similar way as Theorem As the proof of Theorem 2, there three parts: Lemma 8, Proposition 4, and Lemma 9. The statement and proof of Lemmas 8 and can be done in almost the same way, this time without permutation equivariance.
For the proof of the second part, which corresponds to Proposition 4, we construct the network in similar way. Recall that we can assume without loss of generality that [0, 1|dx Choose
0
2
0
1
2
E
2
Then, the first column of X + is in [0, 1]d. second in [1, 21 and so on; this means that for the same technique as the
all rows, the coordinates monotonically increasing. So proof of Proposition to divide the input values into cubes, quantize them to L, apply contextual mapping, and then value mapping. We describe each step in the following.
21

---

Published as a conference paper at ICLR 2020
C.I QUANTIZATION BY FEED-FORWARD LAYERS
In similar way as Lemma 5, the goal of this step is to quantize the input in [0, 1 x [1. x x...X
[n m] its discrete version:
0 8: 87 11:6: 57 x In 1::1::1 ::::n 57
This can be done by dn/ó feed- forward layers. We add dn/6 layers of the following form, for k = 0, d................. n - 5 and = ,,...,d:
0 << 000r226
Z z + e (((e Z
k817), 6(t)
s t 6.
After dn/ō layers, any input entry of X + E in [kō, kō 6) is quantized to kō.
C.2 CONTEXTUAL MAPPING BY ATTENTION LAYERS
By Step I, we quantized any input X + E to its quantized version. We call this quantized version L:
L E [0: 60:6::0 1 81 [1 8 87 x... [n 1:::: -76
As done in Lemma 6, we define u == 1......,,6411 and Lj = u L... for all j E [n]. Note
that, because L.g E jj 5: j:j- j- 8]6, we have
1- 0 + S
1)(++ 11 +8 + 8 8-
-
and 1, 1/2 ... ln- Notice that this corresponds to the Category in the proof of Lemma 6.
For simplicity of notation, let 8j = (j - -j--) 10=0 o We stack n(1/0)d attention layers, with attention parts (.; 6/2,1 + 6/2) for each E Uf=1lsy : 5 8j + 8-d+1 5), in increasing order
These n21/6)d attention layers perform selective shift operations on ls's, in increasing order of j. As seen in Appendix B.5.1, shift operations result in 7, < T2 < ... l, Also, the map from L to 7, is one-to-one, which can be shown in the same way as Appendix B.5.4. Since the range of 1, 's bit different, we have different upper bound on i,:
n
In = In +
kd (In- - (- nd - -11
k=1
< In + 8
(h1 - ln) + +-8nd(1 - 11) l 8n + 8 -+1 - 8 (8-nd_ (8-nd-8-d)e (sn + +8-4+1 -5)
8-d -1
(d-nd_ nd 8-d 1) (n - 1) 8-1 1 + 5-d+11 8
< (6 nd 8 + 1)(0- -- 1)(n - 1 + 8) < nd (n+1)d
Finally, we add an extra single-head attention layer with attention part n8- n8-(n+1)d-11 0). We define the output of this layer as ge (L). In similar way Appendix B.5.1, this layer shifts all the layers by n from each other. This ensures that different contexts are mapped to distinct numbers u² go (L), thus implementing a contextual mapping.
thus making the intervals corresponding different values of T, disjoint
C.3 FUNCTION VALUE MAPPING BY FEED-FORWARD LAYERS
Now, it is left to map ge (L) to the desired output. As seen in the last step, each different context L ge(L). (L). which are least 8 apart from each other. The value mapping
maps to unique numbers u ge step can be done in a similar way as Lemma 7. The construction now requires ((n(1/8)dn) layers because there is permutation equivariance.
22

---

Published as a conference paper at ICLR 2020
Architecture
Average Attention
BProj
SepConv
Transtormer
# params
88.3M
90M
102.5M
110M
Masked LM accuracy (%)
28
59
60
63
MNLI accuracy (%)
66
72.3
73
78.2
Table I: Performance of bi-linear projection and separable convolution layers on masked LM pre- training task and MNLI Note that we expect these computationally cheaper models to have lower performance than the expensive Transformers as they do not compute input dependent attention weights and have weaker representation power. Our goal in studying them is to see if they can substitute some of the expensive attention layers for computing the contextual mappings. These models trained in large batch setting, with batch size of 8192 for 60k steps, unlike the other set of experiments reported in Fig. 1. Note that average attention has clearly worse performance, showing that theses tasks indeed require advanced architecture.
D EXPERIMENTAL SETUP
For our experiments we follow the same setting as BERT (Devlin et al., 2018). We first pre-train the models the masked language modeling task and the next sentence prediction task. We use English Wikipedia corpus and BooksCorpus dataset (Zhu et al., 2015) for this pre-training. We use BERT BASE. 12 layer Transformer model as the baseline. This model uses an embedding size of 768 and has 12 head self-attention layers and 3072 wide feed forward layers. We train it with the Adam optimizer, with ,01 dropout and weight decay. We do pre-training for 250k steps with a batch size of 1024 and a max sequence length of 512. Pre-training takes around days on 16 TPUv3 chips. We take the pre-train models and finetune them on the MNLI and SQuAD datasets separately using the same pappee-prramtters as in Devlin et al. (2018). MNLI is sentence entailment task in which, given a premise sentence, requires us to classify a hypothesis sentence into neutral, contradiction or entailment classes. We report the classification accuracy on this task. SQuAD is question answering task, in which given a paragraph and a question, requires us to identify the answer as a span of the words in the paragraph. For this task report both the Fl score and the Exact Match (EM) percentage. The metrics are reported on the dev sets of these datasets.
For our experiments with the depth- wise separable convolution layers, follow the implementation in (Wu et al., 2019). We first use a GIJU layer followed by the convolution layer. We use 16 separable convolution filters, of filter length 128, and reuse them, with each filter operating dimensions of the input. This layer also has a skip connection and the output is normalized using layer normalization, similar the self-attention layer. In our experiments, we replace the self- attention layers of the Transformers, in the lower layers, with this convolution layer. We keep the feed forward layer of the Transformer block the same.
48 of the 768
For the experiments performed in this paper, one might consider an alternate explanation that the tasks considered maybe are easy, and do not require any advanced architecture solve them, and even simple architecture (bi-linear projection or separable convolution) might solve these tasks. To rule out this case we consider an even simpler architecture, namely average attention, as a baseline for our experiments.
Average attention. An average attention layer replaces the self-attention layer, and just computes the average of projections of all the other tokens. That is, we replace ((W W in (1) with matrix full of 1/n. The model still has the skip connections and the feed feed-forward layers like Transformer.
23


Reviews:
[{"review_id": "ryxi5EF5qB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to analyse the Transformer, widely applied building block of a neural network component, to improve understanding of the internals of the model. The analysis starts showing that the transformer blocks generate permutation equivalent maps and then shows that the transformer can approximate any permutation equivalent map in a compact domain with arbitrary precision. Three key steps are developed and used to prove the universal approximation of arbitrary permutation equivalent map: 1) quantization of input via feed-forward layers, 2) contextual mapping via attention layers, and 3) value mapping via feed-forward layers. By introducing positional embeddings, the paper relaxes the restriction on permutation equivalence and proves that the Transformer is a universal approximator of any sequence to sequence function.\n\nOverall, the paper presents an interesting analysis of the Transformer providing some practical implications, with a caveat for some clarifications on the experiments. The structure of the manuscript could be improved.\n\nOne of the natural question after reading this paper is whether the claims made in section 5 are what actually happens inside of the Transformer because we often observe the attention layers of the first few stacks of Transformer blocks do something. Since the claims lead us to have a universal approximator of seq-2-seq functions, it would be great if there is an experiment based on an alternative model structure based on the claims. For example, one can design a new Transformer architecture with stacks of feed-forward layers, followed by stacks of self-attention layer, followed by other stacks of feed-forward layers. Which may reduce the number of model parameters while preserving a similar level of performance.\n\nHere are a few minor comments on the structure of the manuscript. It seems a bit unnatural to have a section with a proof sketch (section 4). Will it be better if it follows theorem 2. Also, the title of section 5 seems not very informative (proof sketch of proposition 4). You may consider rewriting these sections to improve readability. The definition of contextual mapping and the following lemma seem also one of the major contribution of this paper since the other proof techniques are somewhat familiar with the existing method on universal approximation theory. It would be good to have these result in the early part of the manuscript instead of having it in the end.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rylrdpvAFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CONTRIBUTIONS:\n\nC1. Transformers (without positional encodings and without layer normalization), with 2 attention heads of dimension 1 and feed-forward layers (FFN) with 4 hidden nodes, are universal approximators of continuous permutation-equivariant functions f of compact support, relative to any Lp metric (1 <= p < \\infty). (Thm. 2, p. 3). (Without positional encodings, a function f computed by a Transformer is permutation equivariant: f(P(X)) = P(f(X)) for any permutation P of the columns of X, which are the vector encodings of the input tokens.)\n\nC2. For transformers with trainable positional encodings, the same result holds without the restriction to permutation equivariance (Thm. 3, p. 4)\n\nC3. The notion of ‘contextual mapping’ is introduced (Def. 5.1, p. 6); this notion is central to the proofs of the theorems in C1-C2. Such a mapping q takes any two vectors L, L’ in a finite subset of a real vector space and maps them to a vector space such that all elements (in R) of q(L) and q(L’) are distinct. (In the permutation-equivariant context, L’ must not be a permutation of L.)\n\nRATING: Weak accept\n\nREASONS FOR RATING (SUMMARY). ‘Accept’ because the results seem important. ‘Weak’ because the explanation for the crux of the key result is inaccessible. It relies crucially on the notion of C3, the utility of which for understanding how real Transformers work is questionable.\n\nREVIEW\n\nSec. 4 does a good job of sketching the key proof. But Sec. 5 falls down at a crucial point, Lemma 6.\n\n\nWhat I can glean from the paper (Sec 5.0) is that the idea of the proof is this: (1) use the continuity of f to approximate it with a piecewise constant function c, which takes a fixed value within each hypercube of a discretization of the compact support of f; (2) use a FFN g to map f’s input space to a discrete subset, a regular finite grid G defining the corners of the hypercube discretization; (3) use modified Transformer attention layers to create a contextual mapping k of G; (4) use a FFN h on R to map each (unique) real number in the vector outputs of k to the correct output value so that h([k(g(X))]_i) = [c(X)]_i; (5) approximate the modified Transformer attention layers of k with standard Transformer attention layers. (The modified attention layers replace softmax with argmax and replace ReLU with a (varying) piecewise-linear activation functions “with at most 3 pieces, at least one of which is constant” [Step 2, p. 4.])\n\nThe paper should focus on steps (3) and (5), the only parts of the proof that pertain to what is special about the Transformer: attention. These should be explained fully and clearly in the main text. To make room for this, the rest, concerning FFN approximation and discretization, can be reduced to a paragraph each in the main text, since they are standard, not particular to Transformers.\n\nMy attempts to understand the proofs of (3) and (5) from the Appendix were not successful. To illustrate the kind of difficulty I had in several places: in the proof of Lemma 9 on p. 12, the conclusion is “Thus, given any \\bar{g} \\in \\bar{\\cal{T}}^{2,1,1}, there exists a function g \\in \\cal{T}^{2,1,4} arbitrarily close to \\bar{g}, by appropriately choosing the parameters to be large enough.” (“2,1,4” means a Transformer attention layer with 2 heads of dimension 1 followed by a FFN with 4 hidden units) But how do the numbers {2,1,1} and {2,1,4} relate to the equations internal to this proof? I can see some possible connections, but the authors should spell this out very clearly rather than making the reader work to fill in the gap. It should be clear how the ATTENTION MECHANISM is crucial for the proof by understanding just how that mechanism does the work needed in steps (3) and (5) above.\n\nGiven the time-consuming but ultimately unsuccessful attempt to understand the first few pages of the 11-page Appendix, I did not attempt to absorb the remaining pages.\n\nIt is clear that the notion of contextual mapping plays an important role in their proof of the universal approximation theorem, but does it play any role in the operation of Transformers in practice? Spraying the points of a grid G in the Transformer’s input space into a collection of vectors in which the same real number never appears twice is a nice step in an approximation proof, because it reduces the problem to mapping a set of unique real numbers to another set of real numbers (the elements of the output vectors). But does an actual trained Transformer in practice do anything resembling this? It seems implausible on the face of it, but perhaps there is a theoretical argument, or empirical evidence, that makes it plausible. Thus, for understanding how real Transformers actually do their work, the value of the notion of contextual mapping, and therefore the discussion at the end of the paper of alternatives to attention for achieving it, appears questionable to me. The subtitle of Sec. 5, “Demystifying Transformers”, is not clearly justified.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlbyQ5qtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper discusses the universal approximation capability of the Transformer, under certain assumptions, analyze the role of different components of the Transformer (e.g., self-attention layer for contextual mapping), and propose the use of some other layers that can also provide contextual mapping.\n\nOverall speaking, the problem studied by this paper is very important. The transformer has been used extensively in many applications today, however, deep theoretical understanding of it is not sufficient. Universal approximation capability is a very important theoretical property of deep learning, and advances on the universal approximation of the Transformer is important for the deep learning community. Therefore, I think people will be willing to see the results in this paper. \n\nWhile saying so, this paper has some limitations, which could be further improved. \n\n1)\tThe paper studies a variant of the Transformer, where the layer norm is removed. However, according to practical experiences, the layer norm plays a critical role in the Transformer. As a result, there is gap between this paper and practical situations, and the value of the paper becomes not very clear.\n\n2)\tThe paper lacks experimental verifications. It would be better to design some toy experiments with different types of target functions to see whether the Transformer can well approximate them, and see the contribution of different components of the Transformer\n\n3)\tThe discussions on contextual mapping are not solid enough. First, it seems that contextual mapping is kind of sufficient condition for the proof, however, it is unclear whether it is a necessary condition. Consequently, things are not clear regarding”\n\na.\tIf all the structures (self-attention, bilinear projection, depth-wise separable convolutions) are all sufficient conditions, why self-attention is better than the other two, and why the mix of them can generate even better results?\n\nb.\tIf they are not necessary conditions, we cannot say one should choose them since other structures may be equally good even if they do not satisfy contextual mapping conditions.\n\n4)\tThe experimental study in the paper is not very comprehensive. Given that the Transform has been used in many NLP scenarios, experiments on more datasets and more tasks are expected.\n\n\n**I read the author rebuttal. Some of my concerns still remain, and I would like to keep the current rating (which is positive).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a proof that Transformer networks (a popular deep learning model) are universal approximators for sequence-to-sequence functions. The theorem relies on the idea of contextual mappings (Definition 3.1), which models the attention layers. The results provide an important starting point for understanding a very widely used architecture.

As with many theoretical papers, the reviewers provided several suggestions as to which are important parts to be presented in the main paper. The authors were very responsive during the discussion period, updating the structure of the paper significantly. This shows nice evidence supporting the need for a long discussion period for ICLR. One reviewer upgraded their score (to 8), which is not reflected in the system.

This is an excellent paper, providing much needed theoretical analysis of a popular neural architecture. Clear accept.


Decision: Accept (Poster)