Turing Completeness of Bounded-Precision Recurrent Neural Networks

1 Introduction . Symbolic ( such as Turing Machines ) and sub-symbolic processing ( such as adaptive neural networks ) are two competing methods of representing and processing information , each with its own advantages . An ultimate way to combine symbolic and sub-symbolic capabilities is by enabling the running of algorithms on a neural substrate , which means a neural network that can simulate a Universal Turing Machine ( UTM ) . Previous works [ 1 , 2 , 3 ] have shown that this is possible – there exists a recurrent neural network ( RNN ) that can simulate a UTM . These proofs assumed a couple of neurons with unbounded precision that equals the number of symbols used in the Turing tape . Here we provide an alternative simulation of a UTM by RNNs with bounded-precision neurons only . The general idea works as follows . The Turing Machine ’ s tape is stored in a growing memory module , which is a stack of neurons with pushing and popping operations controlled by neurons in the RNN . The size of the growing memory module is determined by the usage of the Turing tape - it dynamically recruits new neurons when more memories are needed and releases them when memories become irrelevant . The neurons in the stack , except for the top neuron , are not regularly updated ( and hence can be referred to as passive ) , saving computational cost for memories that are not in the focus of the computing and do not require change . Using growing memory modules , a 54-neuron bounded-precision RNN is constructed that can simulate any Turing Machine . Our proposed growing memory modules are inspired by biological memory systems . The process of dynamically recruiting new neurons when more memories are necessary is also observed in biological memory systems . Neurogenesis is the process by which new neurons are produced in the central nervous system ; it is most active during early development , but continues through life . * Both authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In adult vertebrates , neurogenesis is known to occur in the dentate gyrus ( DG ) of the hippocampal formation [ 4 ] and the subventricular zone ( SVZ ) of the lateral ventricles [ 5 ] . Since DG is well-known in neuroscience for its role in pattern separation for memory encoding [ 6 , 7 ] , this suggests that biological memory systems also dynamically recruit new neurons . The rate of neurogenesis in adult mice has been shown to be higher if they are exposed to a wider variety of experiences [ 8 ] . This further suggests a role for self-regulated neurogenesis in scaling up the number of new memory that can be encoded and stored during one ’ s lifetime without catastrophic forgetting of previously consolidated memory . Besides the mechanism of recruiting new neurons , the process of storing neurons in growing memory modules also shares some similarities with biological memory consolidation , a process by which short-term memory is transformed into long-term memory [ 9 , 10 ] . Compared to short-term memory , long-term memory is more long-lasting and robust to interference . This is similar to the neurons stored in growing memory modules - the values of these neurons ( except the top neuron in the stack ) remain unchanged and can not be interfered by the RNN , providing a mechanism to store information stably . Growing memory modules share similarities with other stack-augmented RNNs [ 11 , 12 , 13 , 14 , 15 ] . In neural stacks [ 11 ] , the RNN outputs two continuous scalars that control the strength of pushing and popping operations , and the stack is made differentiable by adding a strength vector . In stack RNNs [ 12 ] and DiffStk-RNN [ 15 ] , the RNN outputs the probability vector corresponding to pushing , popping , and no operation . In NNPDA [ 13 ] , the RNN outputs a continuous scalar that controls the pushing and popping operations , with the minimum value corresponding to popping and the maximum value corresponding to pushing . NSPDA [ 14 ] uses a discrete-valued action neurons to control pushing and popping operations . In contrast to these models , growing memory modules have a simple design and do not need to be differentiable . However , it can be easily shown that growing memory modules can be simulated by these stack-augmented RNNs in linear time , and thus growing memory modules can be considered a generic type of stack-augmented RNNs . Therefore , our proof on the Turing completeness of an RNN with growing memory modules can be extended to stack-augmented RNNs in general , establishing their theoretical motivation . A Turing-complete RNN that is fully differentiable was introduced in 1996 [ 16 ] ; this feature is a prerequisite to have the network trainable by gradient descent . It was followed by the Neural Turing Machine ( NTM ) [ 17 ] and its improved version , the differentiable neural computer [ 18 ] , which are both differentiable and trainable RNNs equipped with memory banks . Though inspired by Turing Machines , bounded-precision NTMs and differentiable neural computers are not Turing-complete due to the fixed-sized memory bank , but they can simulate space-bounded Turing Machines ( see Section 5 ) . Simulation of a Turing Machine by an RNN with growing memory modules represents a practical and biologically inspired way to combine symbolic and sub-symbolic capabilities . All neurons in the RNN and growing memory modules have fixed precision . While the size of growing memory modules is linear in the number of symbols used in the Turing tape , the number of neurons in the RNN is still constant . Moreover , the neurons in growing memory modules ( except the top neuron in the stack ) are passive at most times . As a result , the time complexity of the simulation is linear in the simulated machine ’ s time and independent of the memory size . By showing how to simulate a Turing Machine with a bounded-precision RNN and thereby constructing a bounded-precision RNN that can run any algorithms , our paper proposes a practical method that combines symbolic and sub-symbolic capabilities . The remainder of the paper is structured as follows . Section 2 describes the preliminary of the paper , including the definition of Turing Machines and RNNs . Section 3 revisits and extends theories relating to simulating a Turing Machine with unbounded-precision RNNs and shows the existence of a 40-neuron unbounded-precision RNN that is Turing-complete . Section 4 presents the growing memory modules and proves the existence of a 54-neuron bounded-precision RNN with two growing memory modules that is Turing-complete . Section 5 relates the number of neurons and the precision of RNNs when simulating Turing Machines . Section 6 concludes the paper . 2 Background and Notation . A Turing Machine is a 7-tuple M = ( Q , Σ , Γ , δ , q0 , ♯ , F ) , where Q is a finite set of states , Σ is a finite set of input symbols , Γ is a finite set of tape symbols ( note that Σ ⊂ Γ ) , δ : Q×Γ → Q×Γ× { L , R } is the machine ’ s transition rule , q0 ∈ Q is the initial starting state , ♯ is the blank symbol ( note that ♯ ∈ Γ but ♯ /∈ Σ ) , and F ⊂ Q is the set of final state . We only consider deterministic Turing Machines in this paper . The instantaneous description ( or the configuration ) of a Turing Machine is typically defined as a tuple of state , tape , and the location of the read/write head . However , we use a slightly different definition in the paper . We define the left-tape symbols ( or the left tape in short ) , denoted as sL , to be the string of symbols starting at the symbol under the read/write head and extending to the left . We define the right-tape symbols ( or the right tape in short ) , denoted as sR , to be the string of symbols starting at the symbol to the right of the read/write head and extending to the right . The first symbol in both strings is the closest to the read/write head in both sL and sR ( that is , the left tape is reversed in the representation ) and the blank symbols at the two ends are omitted . Therefore , the length of sL and sR combined equals to the number of symbols used in the Turing tape in each step ( that is , unbounded but not infinite ) . Since sL and sR encode both the tape and the location of the read/write head , we define the instantaneous description of a Turing Machine as a 3-tuple ( q , sL , sR ) ∈ ( Q , Γ∗ , Γ∗ ) , where q denotes the state , sL denotes the left-tape symbols , and sR denotes the right-tape symbols . Though the two definitions are equivalent , this definition allows easy encoding of tape symbols into neurons ’ values . The set of all possible instantaneous description is denoted as X : = ( Q , Γ∗ , Γ∗ ) . In each step of a Turing Machine , the symbol under the read/write head is read and , together with the state , determines the symbol to be written under the read/write head , the direction of moving the tape , and the next state . To be precise , the complete dynamic map of M , denoted as PM : X → X , is defined as follows : 1 . Let x = ( q , sL , sR ) be the input configuration , sL , ( 1 ) denote the first symbol in sL and sR , ( 1 ) denote the first symbol in sR . The transition is defined by the 3-tuple ( q′ , y , d ) = δ ( q , sL , ( 1 ) ) ; 2 . Replace the state of the machine q with q′ and the first symbol in sL by y ; 3 . Move the symbol in sL , ( 1 ) to become the new sR , ( 1 ) if d = L , and move sR , ( 1 ) to become the new sL , ( 1 ) if d = R ( if there are no symbols left in sL or sR for moving , append a blank symbol ♯ to it before moving ) . Denote the left-tape symbols and the right-tape symbols after 2. and 3. by s′L and s ′ R respectively . Then PM ( x ) = ( q′ , s′L , s′R ) represents one transition of the Turing Machine M from one configuration to the next . The partial input-output function of M , denoted as P∗M : X → X , is defined by applying PM repeatedly until q ∈ F , and is undefined if it is not possible to have q ∈ F by applying PM repeatedly . A recurrent neural network ( RNN ) is a neural network consisting of n neurons . The value of neuron i at time t ∈ { 1 , 2 , ... } , denoted as xi ( t ) ∈ Q ( Q is the set of rational numbers ) , is computed by an affine transformation of the values of neurons in the previous state followed by an activation function σ , i.e . xi ( t ) = σ ( ∑n j=1 wijxj ( t− 1 ) + bi ) , where wij are the weights and bi is the bias ; or in vector form : x ( t ) = σ ( Wx ( t− 1 ) + b ) , ( 1 ) where x ( t ) ∈ Qn , W ∈ Rn×n and b ∈ Rn . This defines a mapping TW , b : Qn → Qn which characterizes an RNN . For simplicity , we consider the saturated-linear function in this paper ; that is : σ ( x ) : = 0 if x < 0 , x if 0 ≤ x ≤ 1 , 1 if x > 1 . ( 2 ) Thus , x ( t ) ∈ ( Q ∩ [ 0 , 1 ] ) n for all t > 0 . We say that a neuron xi ( t ) has precision p in base b if for all t > 0 , xi ( t ) can be expressed as∑p l=1 a ( l ) ( t ) ∏l j=1 c ( j ) ( t ) for some strings a ( t ) ∈ { 0 , 1 , ... , b } p and c ( t ) ∈ { 1 , ... , b } p. For a string a , we use a ( i ) to denote the ith symbol in a and a ( i : j ) to denote the string a ( i ) a ( i+1 ) ... a ( j ) . For a function f that maps from a set Y to a subset of Y , we use fn to denote the nth iterate of f where 1 ≤ n < ∞ . For any two vectors x ∈ Rm , y ∈ Rn , we use x ⊕ y ∈ Rm+n to denote the concatenation of the two vectors . We use A∗ to denote all possible strings formed by elements from set A . The notation is summarized in the table found in Appendix D .

Summary: This work proposes a dynamic-growing memory module for RNNs, which serves to simulate Turing machines of bounded precision.  First, the authors prove how to encode a Turing machine to an unbounded precision RNN. The encoding uses fractal encoding for symbols, like previous work, and it replaces each step of the Turing machine as 3 steps in the RNN. Leaning in the work of [14], they prove that there is a 40-neuron unbounded precision RNN to simulate any Turing machine (i.e., showing Turing-completeness). Next, they consider the previous unbounded-precision definitions with the memory module to obtain a bounded-precision RNN.The memory module is a stack of neurons, that requires two neurons in the RNN to control it: push and pop. Then it is proven that the proposed bounded-precision RNN utilizes 2 of these stack to simulate a Turing machine. Further, each stack is divided in groups of $p$ neurons, such that the most active neurons are in the RNN (near to each side of the memory head in the Turing machine). Leaning again in [14], they can show that there is a 54-neuron with $p$-precision RNN with 2 growing memory module, that can simulate any Turing machine. This proves Turing-completeness of the precision-bounded RNN. Finally, they remove the growing memory module and simulate the memory within the neurons of an RNN, showing that an infinite number of bounded-precision RNN can simulate any Turing machine. the work finishes with an interesting discussion of the results and limitations.