Unification of  Recurrent   Neural Network Architectures and Quantum Inspired Stable Design 

Various architectural advancements in the design of recurrent neural networks ( RNN ) have been focusing on improving the empirical stability and representability by sacrificing the complexity of the architecture . However , more remains to be done to fully understand the fundamental trade-off between these conflicting requirements . Towards answering this question , we forsake the purely bottom-up approach of data-driven machine learning to understand , instead , the physical origin and dynamical properties of existing RNN architectures . This facilitates designing new RNNs with smaller complexity overhead and provable stability guarantee . First , we define a family of deep recurrent neural networks , n-t-ORNN , according to the order of nonlinearity n and the range of temporal memory scale t in their underlying dynamics embodied in the form of discretized ordinary differential equations . We show that most of the existing proposals of RNN architectures belong to different orders of n-t-ORNNs . We then propose a new RNN ansatz , namely the Quantum-inspired Universal computing Neural Network ( QUNN ) , to leverage the reversibility , stability , and universality of quantum computation for stable and universal RNN . QUNN provides a complexity reduction in the number of training parameters from being polynomial in both data and correlation time to only linear in correlation time . Compared to Long Short Term Memory ( LSTM ) , QUNN of the same number of hidden layers facilitates higher nonlinearity and longer memory span with provable stability . Our work opens new directions in designing minimal RNNs based on additional knowledge about the dynamical nature of both the data and different training architectures . INTRODUCTION The invention of neural networks opens astounding directions in bottom-up learning of any universal function f : xi → yi by optimizing neural network parameters according to existing data pairs { xi , yi } generated from the function with x ∈ X input space , y ∈ Y output distribution , i ∈ Z index of distribution sample instance . It differs from traditional digital logic circuit encoding of a universal function in two major ways : first , the neural network can be made fully differentiable and thus can morph from one function to the other seamlessly to allow gradient based training ; second , knowledge of the function or the model to be learned is not required beforehand , but instead the multitude of data that carries the information of the functional relation becomes necessary . Although such agnostic learning capability is highly desirable in cases when data are more accessible than the underlying underlying functional form of the model , it also comes with two major limitations . On one hand , to compensate the lack of knowledge about the incoming data , neural network incurs unwanted complexity overhead by requiring more hidden layers and nodes per layer for larger dimensional but not necessarily more complex data . On the other hand , training of deep neural networks with additional hidden layers is difficult due to the lack of stability guarantee . The commonly adopted gradient based optimization of training is proven to fail in most cases due to the observation that gradients of the neural network parameters vanish or explode exponentially quickly in the number of hidden layers ( Bengio et al. , 1994 ) . This renders the overall learning process to become unstable and volatile to both the training methods and format of the incoming data . To reduce the complexity overhead , one can relax the strictly agnostic nature of modern neural networks by leveraging the time-sequential nature of data . One outstanding example is in the design of recurrent neural network ( RNN ) . It is realized by looping the output of the last layer of the neural network back as input to the first layer together with additional exogenous variables . If we unroll this recurrent structure , the equivalent number of neural network layers is as large as the number of the recurrence performed to the data , which can be arbitrarily deep . The recurrent structure , however , does not eliminate the issue of instability of deep neural networks ( DNN ) but rather amplifies possible inherent instability ( Bengio et al. , 1994 ; Pascanu et al. , 2013 ) . To mitigate the major limitation of RNNs , various architectural redesigns are introduced to improve the stability . Two of the most successful designs , the Long Short-Term Memory ( LSTM ) ( Hochreiter and Schmidhuber , 1997 ) and Gated Recurrent Units ( GRU ) ( Cho et al. , 2014 ) , for example add gated feedback structure to RNN to store and retrieve long and short-term memory . This improves the reversibility of the architecture for boosted stability . Related work by Hochreiter and Schmidhuber ( 1997 ) ; Cho et al . ( 2014 ) ; Koutnik et al . ( 2014 ) ; Arjovsky et al . ( 2016a ) and Jing et al . ( 2016 ) improves the controllability over the self-organization of causal dynamics and offers profound insights into how the length of memory span of RNNs affects the efficiency and quality of the training such as more recently proposed Clockwork-RNN ( CW-RNN ) ( Koutnik et al. , 2014 ) and many others ( Wermter et al. , 1999 ; Jaeger et al. , 2007 ; Bengio et al. , 2013 ) . Nonetheless , the stability of most RNN architectures is limited to empirical evidence ( Cho et al. , 2014 ; Jozefowicz et al. , 2015 ; Karpathy et al. , 2015 ; Alpay et al. , 2016 ; Greff et al. , 2017 ) . Moreover , the improved performances are accompanied by the ever increasing complexity of the architecture . What is the fundamental tradeoff between the stability and complexity of RNN design ? Indeed , a large volume of studies have identified the inherent susceptibility of these architectures to instability , and the respective ramifications of such instabilities upon learning by harnessing possible structure of the underlying dynamics of RNN ( Hochreiter et al. , 2001 ; Pascanu et al. , 2013 ; Schmidhuber , 2015 ) . In the majority of studies , steady state solution Lyapunov stability analysis was considered , under structural conditions upon the evolution operator , activation function boundness , or bounded time delays settings ( Jin et al. , 1994 ; Arik , 2000 ; Mandic et al. , 2001 ; Cao and Wang , 2003 ; Zhang and Zeng , 2018 ) . The dynamical system metaphor was often included in the analysis of the RNN as a whole , yet , finer grained analysis of recurring neural network themselves , on a layer by layer basis , using artificial time stepping within the deep neural network itself , was only analyzed ( for low-order integration schemes ) recently , in the context of deep neural networks ( Haber and Ruthotto , 2017 ) . The proposal of unitary evolution RNN ( URNN ) ( Arjovsky et al. , 2016b ) utilizes a unitary weight matrix to preserve the state variable norm and thus the accumulated gradient . However , the total degrees of the freedom in a unitary matrix scales quadratically as the input data dimension , and incur large memory overhead . To reduce such undesirable complexity , they resort to special ansatz which could otherwise impair the overall performance and representability . In Kosmatopoulos et al . ( 1995 ) , high order recurrent neural networks have been proposed utilizing high-order products of the network input components from previous states . The framework , only accounted for shallow , non-compositional integration structure , and therefore did not address the possibility of high order integration over artificial ( layers ) time steps . The significance of depth and compositional structures has been provably realized in recent work ( Mhaskar and Poggio , 2016 ) . If RNN is used to learn temporal structures from data produced in the physical world , why not seek inspirations from physics for RNN designs ? One counterpart of a neural network in a physical system is outlined by Richard Feynman , who proposed the first use of a quantum computer to conduct any classical computations reversibly based on the laws of quantum mechanics described by linear algebra and linear ordinary differential equations ( ODE ) in ( Feynman , 1986 ) . Feynman provides an exact map between any universal function and the dynamics of a quantum mechanical system . Such a map , similar to that of a neural network , is also fully differentiable in its parameters . Moreover , it also linear and reversible due to the fundamental linear nature of quantum mechanics . Ever since Feynman ’ s historical paper , continuous efforts have resulted in surprising discoveries in quantum physics and a plethora of quantum algorithms , e.g . ( Shor , 1994 ; Grover , 1996 ) , that possess provable advantages over the classical counterparts . A crucial connection remains to be made , between quantum dynamics and the dynamics of propagation in a neural network before one can utilize fruitful results in the quantum realm to design better RNNs . RNN formulation has known several transformations , which facilitated deeper insights into regarding both the dynamical process of RNN and the underlying physics behind the data . The first thrust of RNN revolution is based on our knowledge of the time sequentiality of the data , which addresses the complexity overhead of conventional deep neural network , but worsened the stability problem . The second wave of RNN proposals addresses the stability by ad hoc architectural changes that use only heuristic and empirical knowledge of the underlying mechanics . This leads us to attempt at establishing theoretical foundations for RNN and neural network in general , initiated by recent works such as Arjovsky et al . ( 2016b ) ; Mhaskar and Poggio ( 2016 ) and Haber and Ruthotto ( 2017 ) , to apply physics with numerical stability analysis used in solving physical problems for understanding and designing new machine learning architectures . Towards establishing this third wave of RNN revolution , we formulate a framework that unifies different proposals of RNN architectures according to the order of nonlinearity and the order of temporal memory scale of the underlying dynamics embodied in the integration of ODEs , see Table . 1 . To establishe the exact connection between quantum dynamics and RNN dynamics , we show in Sec . 2 that the update rules of RNN can indeed describe the model proposed by Feynman for encoding universal functions in a reversible manner . Lastly for reducing complexity while providing stability guarantees , in Sec . 3 we propose an embedding , called Quantum inspired Universal computing recurrent Neural Netowrk ( QUNN ) , with a complexity of the optimization that grows linearly with the temporal correlation length between input data but independent of the dimension of data itself . 1 STABLE RECURRENT NEURAL NETWORK . The success of supervised machine learning techniques depends on the stability , the representability and the computational overhead associated with the proposed training architecture . A generic neural network without any structure , however , is susceptible to exploding or vanishing gradients and requires additional heuristic optimization techniques to suppress such instability . The stability of deep neural networks can be understood by a simple connection between the integration of discrete ODE and neural network forward and backward propagation ( Haber and Ruthotto , 2017 ) . Take a type of stable deep ResNet proposed by Haber et . al . for example : let lth layer of hidden variable be Yl ∈ Rs×p and bias be bl ∈ Rs×p , to ensure the stability of propagation , they introduce a conjugate variable Zl± 12 ∈ R s×p as a intermediate step such that the propagation of neural network is described by Zl+ 12 = Zl− 1 2 − hlσ ( WTl Yl + bl ) , Yl+1 = Yl + σ ( WlZl+ 12 + bl ) . ( 1 ) The dynamics of the above discrete ODE is stable regardless of the form of weight matrix Ŵl ( Haber and Ruthotto , 2017 ) . In this section , we will extend upon the approach first introduced by Haber and Ruthotto ( 2017 ) to include existing architectures of RNN , such as LSTM , GRU , UNN and CW-RNN , as special instances under the a unified framework . We first define an ODE recurrent neural network with nth order in nonlinearity and tth order in time-derivative ( n-t-ORNN ) according to its propagation rule : the update of n-t-ORNN can be mapped to a generalized nth order Runge–Kutta integration . The specific choice of Runge-Kutta method is not essential to such generalization , and can be replaced by other integration method as well . We then analyze the corresponding n-t-ORNN family different existing RNN architectures belongs . Lastly , we define the n-2-ORNN with anti-Hermitian weight matrices as n-ARNN and prove the stability of 1-ARNN and 2-ARNN . 1.1 nth ORDER ODE RECURRENT NEURAL NETWORK Definition 1 . An ODE recurrent neural network of nth order in nonlinearity tth order in gradient ( nt-ORNN ) , with integers n , t ≥ 1 and k ∈ [ n ] , is described by the update rule between input state value Yl ∈ Rs at time step l , the hidden variables of the kth layer as Klk ∈ Rp with 1 ≤ k ≤ n , and output state value Yl+1 ∈ Rs as Kl1 = σl1 ( Wl1Yl + bl1 ) , Klq = γlqKlq−1 + κlqσlq ( WlqYl+tq−1 + blq + h q−1∑ k=1 αk ◦Klk ) , ( 2 ) Yl+tn = γln+1Yl + κln+1σl ( WlnYl+tn−1 + bln+1 + h n∑ k=1 βkKlk ) ( 3 ) where 2 ≤ q ≤ n ; the time corresponding to each hidden layer obeys tk = bt knc with the overall time steps shared by the n hidden layers being t ; the pointwise activation function σ∗ ( ◦ ) : Rn → Rq at each layer is a nonlinear map that preserves the dimension of the input q ; the weight matrix at each layer is represented by Wl∗ ∈ Rq2×q1 where q1 is the dimension of the input variable and q2 is the dimension of the output variable ; βk , γlm , κlm and αjk ∈ Rp×p are matrices served to rescale and rotate the hidden variables ; γk is a square matrix that manifests the energy dissipation of the ODE dynamics . Below , we analyze some of the most widely used RNN architectures in regard to the nonlinearity and memory scale of their underlying dynamics . Claim 1 . Both LSTM and GRU belong to the 2-L-ORNN . Proof : For one layer RNN , we have the update rule for LSTM ( Hochreiter and Schmidhuber , 1997 ) as : Kt = ft ◦Kt−1 + it ◦ σ2 ( WcYt−1 + UcKt−1 + bc ) , Yt = ot ◦ σ1 ( Kt ) ( 4 ) with vector coefficient determined by ft = σ ( WfYt−1 + UfKt−1 + bf ) , ( 5 ) it = σ ( WiYt−1 + UiKt−1 + bi ) , ot = σ ( WoYt−1 + UoKt−1 + bo ) ( 6 ) which is equivalent to setting n = 2 , t = 1 , γlm2 = D [ ft ] , κm2 = D [ it ] , Wlm2 = Wc , blm2 = bc , hα21 = Uc and γlm = 0 , κlm = ot in n-t-ORNN . Notice that the weight matrix in ORNN can depend on time and is therefore able to include the memory dependency from Kt−1 . We use D [ a ] to represent a p× p diagonal matrix with each diagonal element equal to each element of the vector a of length p. This is because the Hadamard product between two vectors can be re-written as diagonal matrix matrix multiplication with the second vector : a ◦ b = D [ a ] b . For multi-layer LSTM with L hidden layers , the only change is that the diagonal matricesD [ ft ] , D [ it ] and D [ ot ] are generalized to D [ f lt ] , D [ i l t ] and D [ o l t ] , which not only depend on the hidden variable of the same layer from the previous time step , but also the hidden variable of the same time step from a previous layer : f lt = σ ( WfK l−1 t + UfKt−1 + b l f ) , i l t = σ ( WiK l−1 t + UiKt−1 + b l i ) , o l t = σ ( WoK l−1 t + UoKt−1 + b l o ) ( 7 ) where K0t = Yt−1 , and thus the nonlinearity of the ODE increases by one when the number of hidden layers increase by one , thus gives L-2-ORNN for a L layer architecture . For one layer GRU ( Cho et al. , 2014 ) , we have the update rule as : Yt = ( 1− z ) ◦ Yt−1 + z ◦ tanh ( WtYt−1 +Wgr ◦ Yt−1 ) , with z = σ ( W zl Yt−1 ) , r = σ ( W rl Yt−1 ) ( 8 ) we can rewrite r ◦ Yt−1 as σ′ ( W ql Yt−1 ) and thus simplify the update rule to Yt = ( 1− z ) ◦ Yt−1 + z ◦ tanh ( WtYt−1 +Wgσ′ ( W ql Yt−1 ) ) ( 9 ) which is equivalent to setting n = 1 , γlm1 = D [ ( 1−z ) ] , κm1 = D [ z ] , Wlm1 =Wt , blm2 = 0 , hβ1 = Wg in n-ORNN . This can be similarly generalized to multi-layer GRU with L total hidden layers by allowing the weight matrices D [ ( 1− z ) ] and D [ z ] to also depend on same layer hidden variable of previous step : z = σ ( W zl Y l t−1 +W z′ l Y l−1 t ) , r = σ ( W r l Yt−1 +W r′ l Y l−1 t ) ( 10 ) which for lth layer it corresponds to L-2-ORNN . Q.E.D . Claim 2 . Unitary evolution RNN ( Arjovsky et al. , 2016a ) with L hidden layers belongs to the 2-L-ORNN . Proof : The propagation rule of URNN between the input to the RNN at time step 1 ≤ t ≤ T : Yt and hidden variables at the same time step Kt and output to the RNN of the same time step t as Yt+1 is : Kt+1 = σ ( WlKt + VlYt ) , for 1 ≤ l ≤ t , Yt+1 =Wl+1Kt+1 + bl+1 , ( 11 ) which corresponds to setting γjj = 0 and choosing n = 2 and t = L in Eq . ( 2 ) – ( 3 ) . URNN thus belongs to 2-L-ORNN . Q.E.D . Claim 3 . Clockwork RNN ( Koutnik et al . ( 2014 ) ) with L clocks belongs to the L-L-ORNN . Proof : The propagation rule for CW-RNN between input Yt at time step t , hidden layers at the same time step Kt as well as from the previous time step Kt−1 and output Yt+1 is described by : Kt = σh ( WH ( t ) Kt−1 +WI ( t ) Yt ) , Yt+1 = σo ( WoKt ) ( 12 ) where the time-dependent weight matrices WH ( t ) and WI ( t ) are structured to store memory of previous time steps in into different blocks with increasing duration of time delays such that effectively one can rewriteWH ( t ) Kt−1 = ∑ j Wjσ t−j−1 j WH ( j ) Kj+WI ( j ) Yj contributions from all previous step iteratively , and so is the clock structure in WI ( t ) Yt which contributes to all hidden layers after t. This is equivalent to setting n = t = L and γjj = 0 and t = 1 in Eq . ( 2 ) – ( 3 ) . CW-RNN thus belongs to L-L-ORNN . Q.E.D . Definition 2 . The nth order ODE anti-Hermitian recurrent neural network ( n-ARNN ) corresponds to setting all weight matrices in n-2-ORNN with anti-Hermitian matrices . Theorem 1 . 1-ARNN with monotonic activation function σ∗ ( · ) : Rn → Rn and purely imaginary anti-Hermitian weight matrix is stable for small enough h such that |hmaxk λ [ Wlk ] | < 1 . Proof : This will be proven in Theorem 4 , where the original complex anti-Hermitian matrix is embedded into a Hilbert space twice as large such that a purely imaginary anti-Hermitian weight matrix guarantees the stability of the first order integration method . Q.E.D . Theorem 2 . Both 2-ARNN and 1-ARNN are reversible . Proof : Since 2-ARNN corresponds to the first order mid-point integration and 1-ARNN corresponds to the symplectic Euler intergration their reversibility is guaranteed by the reversibility of these two integration schemes inside the stble regime . Q.E.D . It is notable that the definition of n-t-ORNN does not restrict weight matrices to be time independent . This setup is less restrictive than conventional definition of RNN and is indispensable for generalizing various architectures of RNN under the same framework . Such generalization , however , is wellfounded in ODE framework : a generic ODE does not have to be time-independent . The unification of different RNN architectures through n-t-ORNN prompts a better usage and obtainment of the memory scale and degree of nonlinearity of the data to further reduce the necessary complexity in the learning architecture . We demonstrate such combination between bottom-up data centered and top-down model centered approach in finding more efficient RNN design in the next section .

Summary: The authors make connections between RNN dynamics and those of a class of ODEs similar to RNNs (ORNN) that has different orders of nonlinearity and order of gradients in time. They show that typical RNN architectures can be described as members of the ORNN family. They then make the connection that quantum mechanical systems can be described as following the Schrodinger equation which can be cast as a series of coupled 1st order ODEs of the evolution of wavefunctions under an influencing Hamiltonian. They then claim that these discretized equations can be represented by a RNN similar to a unitary RNN. They go on to outline a RNN structure inspired by this insight that has time-dependent activations to increase the scale of temporal dependence.