Turing Completeness of Bounded-Precision Recurrent Neural Networks

1 Introduction . Symbolic ( such as Turing Machines ) and sub-symbolic processing ( such as adaptive neural networks ) are two competing methods of representing and processing information , each with its own advantages . An ultimate way to combine symbolic and sub-symbolic capabilities is by enabling the running of algorithms on a neural substrate , which means a neural network that can simulate a Universal Turing Machine ( UTM ) . Previous works [ 1 , 2 , 3 ] have shown that this is possible – there exists a recurrent neural network ( RNN ) that can simulate a UTM . These proofs assumed a couple of neurons with unbounded precision that equals the number of symbols used in the Turing tape . Here we provide an alternative simulation of a UTM by RNNs with bounded-precision neurons only . The general idea works as follows . The Turing Machine ’ s tape is stored in a growing memory module , which is a stack of neurons with pushing and popping operations controlled by neurons in the RNN . The size of the growing memory module is determined by the usage of the Turing tape - it dynamically recruits new neurons when more memories are needed and releases them when memories become irrelevant . The neurons in the stack , except for the top neuron , are not regularly updated ( and hence can be referred to as passive ) , saving computational cost for memories that are not in the focus of the computing and do not require change . Using growing memory modules , a 54-neuron bounded-precision RNN is constructed that can simulate any Turing Machine . Our proposed growing memory modules are inspired by biological memory systems . The process of dynamically recruiting new neurons when more memories are necessary is also observed in biological memory systems . Neurogenesis is the process by which new neurons are produced in the central nervous system ; it is most active during early development , but continues through life . * Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In adult vertebrates , neurogenesis is known to occur in the dentate gyrus ( DG ) of the hippocampal formation [ 4 ] and the subventricular zone ( SVZ ) of the lateral ventricles [ 5 ] . Since DG is well-known in neuroscience for its role in pattern separation for memory encoding [ 6 , 7 ] , this suggests that biological memory systems also dynamically recruit new neurons . The rate of neurogenesis in adult mice has been shown to be higher if they are exposed to a wider variety of experiences [ 8 ] . This further suggests a role for self-regulated neurogenesis in scaling up the number of new memory that can be encoded and stored during one ’ s lifetime without catastrophic forgetting of previously consolidated memory . Besides the mechanism of recruiting new neurons , the process of storing neurons in growing memory modules also shares some similarities with biological memory consolidation , a process by which short-term memory is transformed into long-term memory [ 9 , 10 ] . Compared to short-term memory , long-term memory is more long-lasting and robust to interference . This is similar to the neurons stored in growing memory modules - the values of these neurons ( except the top neuron in the stack ) remain unchanged and can not be interfered by the RNN , providing a mechanism to store information stably . Growing memory modules share similarities with other stack-augmented RNNs [ 11 , 12 , 13 , 14 , 15 ] . In neural stacks [ 11 ] , the RNN outputs two continuous scalars that control the strength of pushing and popping operations , and the stack is made differentiable by adding a strength vector . In stack RNNs [ 12 ] and DiffStk-RNN [ 15 ] , the RNN outputs the probability vector corresponding to pushing , popping , and no operation . In NNPDA [ 13 ] , the RNN outputs a continuous scalar that controls the pushing and popping operations , with the minimum value corresponding to popping and the maximum value corresponding to pushing . NSPDA [ 14 ] uses a discrete-valued action neurons to control pushing and popping operations . In contrast to these models , growing memory modules have a simple design and do not need to be differentiable . However , it can be easily shown that growing memory modules can be simulated by these stack-augmented RNNs in linear time , and thus growing memory modules can be considered a generic type of stack-augmented RNNs . Therefore , our proof on the Turing completeness of an RNN with growing memory modules can be extended to stack-augmented RNNs in general , establishing their theoretical motivation . A Turing-complete RNN that is fully differentiable was introduced in 1996 [ 16 ] ; this feature is a prerequisite to have the network trainable by gradient descent . It was followed by the Neural Turing Machine ( NTM ) [ 17 ] and its improved version , the differentiable neural computer [ 18 ] , which are both differentiable and trainable RNNs equipped with memory banks . Though inspired by Turing Machines , bounded-precision NTMs and differentiable neural computers are not Turing-complete due to the fixed-sized memory bank , but they can simulate space-bounded Turing Machines ( see Section 5 ) . Simulation of a Turing Machine by an RNN with growing memory modules represents a practical and biologically inspired way to combine symbolic and sub-symbolic capabilities . All neurons in the RNN and growing memory modules have fixed precision . While the size of growing memory modules is linear in the number of symbols used in the Turing tape , the number of neurons in the RNN is still constant . Moreover , the neurons in growing memory modules ( except the top neuron in the stack ) are passive at most times . As a result , the time complexity of the simulation is linear in the simulated machine ’ s time and independent of the memory size . By showing how to simulate a Turing Machine with a bounded-precision RNN and thereby constructing a bounded-precision RNN that can run any algorithms , our paper proposes a practical method that combines symbolic and sub-symbolic capabilities . The remainder of the paper is structured as follows . Section 2 describes the preliminary of the paper , including the definition of Turing Machines and RNNs . Section 3 revisits and extends theories relating to simulating a Turing Machine with unbounded-precision RNNs and shows the existence of a 40-neuron unbounded-precision RNN that is Turing-complete . Section 4 presents the growing memory modules and proves the existence of a 54-neuron bounded-precision RNN with two growing memory modules that is Turing-complete . Section 5 relates the number of neurons and the precision of RNNs when simulating Turing Machines . Section 6 concludes the paper . 2 Background and Notation . A Turing Machine is a 7-tuple M = ( Q , Σ , Γ , δ , q0 , ♯ , F ) , where Q is a finite set of states , Σ is a finite set of input symbols , Γ is a finite set of tape symbols ( note that Σ ⊂ Γ ) , δ : Q×Γ → Q×Γ× { L , R } is the machine ’ s transition rule , q0 ∈ Q is the initial starting state , ♯ is the blank symbol ( note that ♯ ∈ Γ but ♯ /∈ Σ ) , and F ⊂ Q is the set of final state . We only consider deterministic Turing Machines in this paper . The instantaneous description ( or the configuration ) of a Turing Machine is typically defined as a tuple of state , tape , and the location of the read/write head . However , we use a slightly different definition in the paper . We define the left-tape symbols ( or the left tape in short ) , denoted as sL , to be the string of symbols starting at the symbol under the read/write head and extending to the left . We define the right-tape symbols ( or the right tape in short ) , denoted as sR , to be the string of symbols starting at the symbol to the right of the read/write head and extending to the right . The first symbol in both strings is the closest to the read/write head in both sL and sR ( that is , the left tape is reversed in the representation ) and the blank symbols at the two ends are omitted . Therefore , the length of sL and sR combined equals to the number of symbols used in the Turing tape in each step ( that is , unbounded but not infinite ) . Since sL and sR encode both the tape and the location of the read/write head , we define the instantaneous description of a Turing Machine as a 3-tuple ( q , sL , sR ) ∈ ( Q , Γ∗ , Γ∗ ) , where q denotes the state , sL denotes the left-tape symbols , and sR denotes the right-tape symbols . Though the two definitions are equivalent , this definition allows easy encoding of tape symbols into neurons ’ values . The set of all possible instantaneous description is denoted as X : = ( Q , Γ∗ , Γ∗ ) . In each step of a Turing Machine , the symbol under the read/write head is read and , together with the state , determines the symbol to be written under the read/write head , the direction of moving the tape , and the next state . To be precise , the complete dynamic map of M , denoted as PM : X → X , is defined as follows : 1 . Let x = ( q , sL , sR ) be the input configuration , sL , ( 1 ) denote the first symbol in sL and sR , ( 1 ) denote the first symbol in sR . The transition is defined by the 3-tuple ( q′ , y , d ) = δ ( q , sL , ( 1 ) ) ; 2 . Replace the state of the machine q with q′ and the first symbol in sL by y ; 3 . Move the symbol in sL , ( 1 ) to become the new sR , ( 1 ) if d = L , and move sR , ( 1 ) to become the new sL , ( 1 ) if d = R ( if there are no symbols left in sL or sR for moving , append a blank symbol ♯ to it before moving ) . Denote the left-tape symbols and the right-tape symbols after 2. and 3. by s′L and s ′ R respectively . Then PM ( x ) = ( q′ , s′L , s′R ) represents one transition of the Turing Machine M from one configuration to the next . The partial input-output function of M , denoted as P∗M : X → X , is defined by applying PM repeatedly until q ∈ F , and is undefined if it is not possible to have q ∈ F by applying PM repeatedly . A recurrent neural network ( RNN ) is a neural network consisting of n neurons . The value of neuron i at time t ∈ { 1 , 2 , ... } , denoted as xi ( t ) ∈ Q ( Q is the set of rational numbers ) , is computed by an affine transformation of the values of neurons in the previous state followed by an activation function σ , i.e . xi ( t ) = σ ( ∑n j=1 wijxj ( t− 1 ) + bi ) , where wij are the weights and bi is the bias ; or in vector form : x ( t ) = σ ( Wx ( t− 1 ) + b ) , ( 1 ) where x ( t ) ∈ Qn , W ∈ Rn×n and b ∈ Rn . This defines a mapping TW , b : Qn → Qn which characterizes an RNN . For simplicity , we consider the saturated-linear function in this paper ; that is : σ ( x ) : = 0 if x < 0 , x if 0 ≤ x ≤ 1 , 1 if x > 1 . ( 2 ) Thus , x ( t ) ∈ ( Q ∩ [ 0 , 1 ] ) n for all t > 0 . We say that a neuron xi ( t ) has precision p in base b if for all t > 0 , xi ( t ) can be expressed as∑p l=1 a ( l ) ( t ) ∏l j=1 c ( j ) ( t ) for some strings a ( t ) ∈ { 0 , 1 , ... , b } p and c ( t ) ∈ { 1 , ... , b } p. For a string a , we use a ( i ) to denote the ith symbol in a and a ( i : j ) to denote the string a ( i ) a ( i+1 ) ... a ( j ) . For a function f that maps from a set Y to a subset of Y , we use fn to denote the nth iterate of f where 1 ≤ n < ∞ . For any two vectors x ∈ Rm , y ∈ Rn , we use x ⊕ y ∈ Rm+n to denote the concatenation of the two vectors . We use A∗ to denote all possible strings formed by elements from set A . The notation is summarized in the table found in Appendix D .

Summary: This paper analyses the Turing completeness of recurrent neural networks, and proposes a memory-augmented RNN architecture reminiscent of existing stack RNNs.   Specifically:  First, a construction for simulating any Turing machine with an unbounded-precision RNN is presented, in which effectively the left and right sides of the Turing machine's tape (with respect to its head) are encoded as two 'stacks' in the RNN. This construction is analysed and requires less simulation time than that of Siegelmann and Sonntag [1] (3T instead of 4T+O(used tape length), where T is computation time of the simulated Turing machine). Additionally the authors analyse the exact precision of the neurons necessary (as function of the simulated Turing machine's computation length) for the simulation (as opposed to previous works, which simply note a requirement for infinite precision).  Second, noting that it is unavoidable that the precision must grow when simulating longer Turing machine computations (to maintain the contents of the tape), the authors propose augmenting the RNN with a dynamic memory module, such that the neurons themselves may have constant, bounded, precision. They theoretically analyse their proposed architecture and show how a Turing machine may be simulated in it.