On the Turing Completeness of Modern Neural Network Architectures

1 INTRODUCTION . There is an increasing interest in designing neural network architectures capable of learning algorithms from examples ( Graves et al. , 2014 ; Grefenstette et al. , 2015 ; Joulin & Mikolov , 2015 ; Kaiser & Sutskever , 2016 ; Kurach et al. , 2016 ; Dehghani et al. , 2018 ) . A key requirement for any such an architecture is thus to have the capacity of implementing arbitrary algorithms , that is , to be Turing complete . Turing completeness often follows for these networks as they can be seen as a control unit with access to an unbounded memory ; as such , they are capable of simulating any Turing machine . On the other hand , the work by Siegelmann & Sontag ( 1995 ) has established a different way of looking at the Turing completeness of neural networks . In particular , their work establishes that recurrent neural networks ( RNNs ) are Turing complete even if only a bounded number of resources ( i.e. , neurons and weights ) is allowed . This is based on two conditions : ( 1 ) the ability of RNNs to compute internal dense representations of the data , and ( 2 ) the mechanisms they use for accessing such representations . Hence , the view proposed by Siegelmann & Sontag shows that it is possible to release the full computational power of RNNs without arbitrarily increasing its model complexity . Most of the early neural architectures proposed for learning algorithms correspond to extensions of RNNs – e.g. , Neural Turing Machines ( Graves et al. , 2014 ) – , and hence they are Turing complete in the sense of Siegelmann & Sontag . However , a recent trend has shown the benefits of designing networks that manipulate sequences but do not directly apply a recurrence to sequentially process their input symbols . Architectures based on attention or convolutions are two prominent examples of this approach . In this work we look at the problem of Turing completeness à la Siegelmann & Sontag for two of the most paradigmatic models exemplifying these features : the Transformer ( Vaswani et al. , 2017 ) and the Neural GPU ( Kaiser & Sutskever , 2016 ) . The main contribution of our paper is to show that the Transformer and the Neural GPU are Turing complete based on their capacity to compute and access internal dense representations of the data . In particular , neither the Transformer nor the Neural GPU requires access to an external additional memory to become Turing complete . Thus the completeness holds for bounded architectures ( bounded number of neurons and parameters ) . To prove this we assume that internal activations are represented as rational numbers with arbitrary precision . For the case of the Transformer we provide a direct simulation of a Turing machine , while for the case of the Neural GPU our result follows by simulating standard sequence-to-sequence RNNs . Our study also reveals some minimal sets of elements needed to obtain these completeness results . The computational power of Transformers and of Neural GPUs has been compared in the current literature ( Dehghani et al. , 2018 ) , but both are only informally used . Our paper provides a formal way of approaching this comparison . For the sake of space , we only include sketch of some proofs in the body of the paper . The details for every proof can be found in the appendix . Background work The study of the computational power of neural networks can be traced back to McCulloch & Pitts ( 1943 ) which established an analogy between neurons with hard-threshold activations and first order logic sentences , and Kleene ( 1956 ) that draw a connection between neural networks and finite automata . As mentioned earlier , the first work showing the Turing completeness of finite neural networks with linear connections was carried out by Siegelmann & Sontag ( 1992 ; 1995 ) . Since being Turing complete does not ensure the ability to actually learn algorithms in practice , there has been an increasing interest in enhancing RNNs with mechanisms for supporting this task . One strategy has been the addition of inductive biases in the form of external memory , being the Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) a paradigmatic example . To ensure that NTMs are differentiable , their memory is accessed via a soft attention mechanism ( Bahdanau et al. , 2014 ) . Other examples of architectures that extend RNNs with memory are the Stack-RNN ( Joulin & Mikolov , 2015 ) , and the ( De ) Queue-RNNs ( Grefenstette et al. , 2015 ) . By Siegelmann & Sontag ’ s results , all these architectures are Turing complete . The Transformer architecture ( Vaswani et al. , 2017 ) is almost exclusively based on the attention mechanism , and it has achieved state of the art results on many language-processing tasks . While not initially designed to learn general algorithms , Dehghani et al . ( 2018 ) have advocated the need for enriching its architecture with several new features as a way to learn general procedures in practice . This enrichment is motivated by the empirical observation that the original Transformer architecture struggles to generalize to input of lengths not seen during training . We , in contrast , show that the original Transformer architecture is Turing complete , based on different considerations . These results do not contradict each other , but show the differences that may arise between theory and practice . For instance , Dehghani et al . ( 2018 ) assume fixed precision , while we allow arbitrary internal precision during computation . We think that both approaches can be complementary as our theoretical results can shed light on what are the intricacies of the original architecture , which aspects of it are candidates for change or improvement , and which others are strictly needed . For instance , our proof uses hard attention while the Transformer is often trained with soft attention ( Vaswani et al. , 2017 ) . See Section 3.3 for a discussion on these differences . The Neural GPU is an architecture that mixes convolutions and gated recurrences over tridimensional tensors . It has been shown that NeuralGPUs are powerful enough to learn decimal multiplication from examples ( Freivalds & Liepins , 2018 ) , being the first neural architecture capable of solving this problem end-to-end . The similarity of Neural GPUs and cellular automata has been used as an argument to state the Turing completeness of the architecture ( Kaiser & Sutskever , 2016 ; Price et al. , 2016 ) . Cellular automata are Turing complete ( Smith III , 1971 ; Ollinger , 2012 ) and their completeness is established assuming an unbounded number of cells . In the Neural GPU architecture , in contrast , the number of cells that can be used during a computation is proportional to the size of the input sequence ( Kaiser & Sutskever , 2016 ) . One can cope with the need for more cells by padding the Neural GPU input with additional ( dummy ) symbols , as much as needed for a particular computation . Nevertheless , this is only a partial solution , as for a Turing-complete model of computation , one can not decide a priori how much memory is needed to solve a particular problem . Our results in this paper are somehow orthogonal to the previous argument ; we show that one can leverage the dense representations of the Neural GPU cells to obtain Turing completeness without requiring to add cells beyond the ones used to store the input . 2 PRELIMINARIES . We assume all weights and activations to be rational numbers of arbitrary precision . Moreover , we only allow the use of rational functions with rational coefficients . Most of our positive results make use of the piecewise-linear sigmoidal activation function σ : Q→ Q , which is defined as σ ( x ) = { 0 x < 0 , x 0 ≤ x ≤ 1 , 1 x > 1 . ( 1 ) We are mostly interested in sequence-to-sequence ( seq-to-seq ) neural network architectures that we next formalize . A seq-to-seq network N receives as input a sequenceX = ( x1 , . . . , xn ) of vectors xi ∈ Qd , for some d > 0 , and produces as output a sequence Y = ( y1 , . . . , ym ) of vectors yi ∈ Qd . Most of these types of architectures require a seed vector s and some stopping criterion for determining the length of the output . The latter is usually based on the generation of a particular output vector called an end of sequence mark . In our formalization instead , we allow a network to produce a fixed number r ≥ 0 of output vectors . Thus , for convenience we see a general seq-toseq network as a function N such that the value N ( X , s , r ) corresponds to an output sequence of the form Y = ( y1 , y2 , . . . , yr ) . With this definition , we can view every seq-to-seq network as a language recognizer of strings as follows . Definition 2.1 . A seq-to-seq language recognizer is a tuple A = ( Σ , f , N , s , F ) , where Σ is a finite alphabet , f : Σ → Qd is an embedding function , N is a seq-to-seq network , s ∈ Qd is a seed vector , and F ⊆ Qd is a set of final vectors . We say that A accepts the string w ∈ Σ∗ , if there exists an integer r ∈ N such that N ( f ( w ) , s , r ) = ( y1 , . . . , yr ) and yr ∈ F. The language accepted by A , denoted by L ( A ) , is the set of all strings accepted by A . We impose two additional restrictions over recognizers . The embedding function f : Σ → Qd should be computed by a Turing machine in time linear w.r.t . the size of Σ . This covers the two most typical ways of computing input embeddings from symbols : the one-hot encoding , and embeddings computed by fixed feed-forward networks . Moreover , the set F should also be recognizable in lineartime ; given a vector f , the membership f ∈ F should be decided by a Turing machine working in linear time with respect to the size ( in bits ) of f . This covers the usual way of checking equality with a fixed end-of-sequence vector . We impose these restrictions to disallow the possibility of cheating by encoding arbitrary computations in the input embedding or the stop condition , while being permissive enough to construct meaningful embeddings and stoping criterions . Finally , a class N of seq-to-seq neural network architectures defines the class LN composed of all the languages accepted by language recognizers that use networks in N . From these notions , the formalization of Turing completeness of a class N naturally follows . Definition 2.2 . A class N of seq-to-seq neural network architectures is Turing Complete if LN is exactly the class of languages recognized by Turing machines . Given an input sequenceX = ( x1 , . . . , xn ) , a seed vector y0 , and r ∈ N , an encoder-decoder RNN is given by the following two recursions h0 = 0 , hi = f1 ( xiW + hi−1V + b1 ) ( with 1 ≤ i ≤ n ) ( 2 ) g0 = hn , gt = f2 ( gt−1U + yt−1R+ b2 ) , yt = O ( gt ) ( with 1 ≤ t ≤ r ) ( 3 ) where V , W , U , R are matrices , b1 and b2 are vectors , O ( · ) is an output function , and f1 and f2 are activations functions . Equation ( 2 ) is called the RNN-encoder and ( 3 ) the RNN-decoder . The next Theorem follows by inspection of the proof by Siegelmann & Sontag ( 1992 ; 1995 ) after adapting it to our formalization of encoder-decoder RNNs . Theorem 2.3 ( Siegelmann & Sontag ( 1992 ; 1995 ) ) . The class of encoder-decoder RNNs is Turing complete . Turing completeness holds even if we restrict to the class in which R is the zero matrix , b1 and b2 are the zero vector , O ( · ) is the identity function , and f1 and f2 are the piecewise-linear sigmoidal activation σ .

Summary: This paper presents interesting theoretical results on Turing completeness of the Transformer and Neural GPU architectures, as modern architectures based on attention and convolutions, under particular assumptions. The basis of proofs in the paper relies on Turing completeness of the seq2seq architecture, which is Turing complete since it contains Turing complete RNNs. Turing completeness of the Transformer and the Neural GPU is proven by showing they can simulate seq2seq architecture.