Year: 2018

Abstract:
Various architectural advancements in the design of recurrent neural networks (RNN) have been focusing on improving the empirical stability and representability by sacrificing the complexity of the architecture. However, more remains to be done to fully understand the fundamental trade-off between these conflicting requirements. Towards answering this question, we forsake the purely bottom-up approach of data-driven machine learning to understand, instead, the physical origin and dynamical properties of existing RNN architectures. This facilitates designing new RNNs with smaller complexity overhead and provable stability guarantee. First, we define a family of deep recurrent neural networks, n-t-ORNN, according to the order of nonlinearity n and the range of temporal memory scale t in their underlying dynamics embodied in the form of discretized ordinary differential equations. We show that most of the existing proposals of RNN architectures belong to different orders of n-t-ORNNs. We then propose a new RNN ansatz, namely the Quantum-inspired Universal computing Neural Network (QUNN), to leverage the reversibility, stability, and universality of quantum computation for stable and universal RNN. QUNN provides a complexity reduction in the number of training parameters from being polynomial in both data and correlation time to only linear in correlation time. Compared to Long Short Term Memory (LSTM), QUNN of the same number of hidden layers facilitates higher nonlinearity and longer memory span with provable stability. Our work opens new directions in designing minimal RNNs based on additional knowledge about the dynamical nature of both the data and different training architectures.

text: Various architectural advancements in the design of recurrent neural networks (RNN) have been focusing on improving the empirical stability and representability by sacrificing the complexity of the architecture. However, more remains to be done to fully understand the fundamental trade-off between these conflicting requirements. Towards answering this question, we forsake the purely bottom-up approach of data-driven machine learning to understand, instead, the physical origin and dynamical properties of existing RNN architectures. This facilitates designing new RNNs with smaller complexity overhead and provable stability guarantee. First, we define a family of deep recurrent neural networks, n-t-ORNN, according to the order of nonlinearity n and the range of temporal memory scale t in their underlying dynamics embodied in the form of discretized ordinary differential equations. We show that most of the existing proposals of RNN architectures belong to different orders of n-t-ORNNs. We then propose a new RNN ansatz, namely the Quantum-inspired Universal computing Neural Network (QUNN), to leverage the reversibility, stability, and universality of quantum computation for stable and universal RNN. QUNN provides a complexity reduction in the number of training parameters from being polynomial in both data and correlation time to only linear in correlation time. Compared to Long Short Term Memory (LSTM), QUNN of the same number of hidden layers facilitates higher nonlinearity and longer memory span with provable stability. Our work opens new directions in designing minimal RNNs based on additional knowledge about the dynamical nature of both the data and different training architectures.
INTRODUCTION
The invention of neural networks opens astounding directions in bottom-up learning of any universal function f : xi → yi by optimizing neural network parameters according to existing data pairs {xi, yi} generated from the function with x ∈ X input space, y ∈ Y output distribution, i ∈ Z index of distribution sample instance. It differs from traditional digital logic circuit encoding of a universal function in two major ways: first, the neural network can be made fully differentiable and thus can morph from one function to the other seamlessly to allow gradient based training; second, knowledge of the function or the model to be learned is not required beforehand, but instead the multitude of data that carries the information of the functional relation becomes necessary.
Although such agnostic learning capability is highly desirable in cases when data are more accessible than the underlying underlying functional form of the model, it also comes with two major limitations. On one hand, to compensate the lack of knowledge about the incoming data, neural network incurs unwanted complexity overhead by requiring more hidden layers and nodes per layer for larger dimensional but not necessarily more complex data. On the other hand, training of deep neural networks with additional hidden layers is difficult due to the lack of stability guarantee. The commonly adopted gradient based optimization of training is proven to fail in most cases due to the observation that gradients of the neural network parameters vanish or explode exponentially quickly in the number of hidden layers (Bengio et al., 1994). This renders the overall learning process to become unstable and volatile to both the training methods and format of the incoming data.
To reduce the complexity overhead, one can relax the strictly agnostic nature of modern neural networks by leveraging the time-sequential nature of data. One outstanding example is in the design of recurrent neural network (RNN). It is realized by looping the output of the last layer of the neural network back as input to the first layer together with additional exogenous variables. If we unroll this recurrent structure, the equivalent number of neural network layers is as large as the number of the recurrence performed to the data, which can be arbitrarily deep.
The recurrent structure, however, does not eliminate the issue of instability of deep neural networks (DNN) but rather amplifies possible inherent instability (Bengio et al., 1994; Pascanu et al., 2013). To mitigate the major limitation of RNNs, various architectural redesigns are introduced to improve the stability. Two of the most successful designs, the Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014), for example add gated feedback structure to RNN to store and retrieve long and short-term memory. This improves the reversibility of the architecture for boosted stability. Related work by Hochreiter and Schmidhuber (1997); Cho et al. (2014); Koutnik et al. (2014); Arjovsky et al. (2016a) and Jing et al. (2016) improves the controllability over the self-organization of causal dynamics and offers profound insights into how the length of memory span of RNNs affects the efficiency and quality of the training such as more recently proposed Clockwork-RNN (CW-RNN) (Koutnik et al., 2014) and many others (Wermter et al., 1999; Jaeger et al., 2007; Bengio et al., 2013). Nonetheless, the stability of most RNN architectures is limited to empirical evidence (Cho et al., 2014; Jozefowicz et al., 2015; Karpathy et al., 2015; Alpay et al., 2016; Greff et al., 2017). Moreover, the improved performances are accompanied by the ever increasing complexity of the architecture.
What is the fundamental tradeoff between the stability and complexity of RNN design? Indeed, a large volume of studies have identified the inherent susceptibility of these architectures to instability, and the respective ramifications of such instabilities upon learning by harnessing possible structure of the underlying dynamics of RNN (Hochreiter et al., 2001; Pascanu et al., 2013; Schmidhuber, 2015). In the majority of studies, steady state solution Lyapunov stability analysis was considered, under structural conditions upon the evolution operator, activation function boundness, or bounded time delays settings (Jin et al., 1994; Arik, 2000; Mandic et al., 2001; Cao and Wang, 2003; Zhang and Zeng, 2018). The dynamical system metaphor was often included in the analysis of the RNN as a whole, yet, finer grained analysis of recurring neural network themselves, on a layer by layer basis, using artificial time stepping within the deep neural network itself, was only analyzed (for low-order integration schemes) recently, in the context of deep neural networks (Haber and Ruthotto, 2017). The proposal of unitary evolution RNN (URNN) (Arjovsky et al., 2016b) utilizes a unitary weight matrix to preserve the state variable norm and thus the accumulated gradient. However, the total degrees of the freedom in a unitary matrix scales quadratically as the input data dimension, and incur large memory overhead. To reduce such undesirable complexity, they resort to special ansatz which could otherwise impair the overall performance and representability. In Kosmatopoulos et al. (1995), high order recurrent neural networks have been proposed utilizing high-order products of the network input components from previous states. The framework, only accounted for shallow, non-compositional integration structure, and therefore did not address the possibility of high order integration over artificial (layers) time steps. The significance of depth and compositional structures has been provably realized in recent work (Mhaskar and Poggio, 2016).
If RNN is used to learn temporal structures from data produced in the physical world, why not seek inspirations from physics for RNN designs? One counterpart of a neural network in a physical system is outlined by Richard Feynman, who proposed the first use of a quantum computer to conduct any classical computations reversibly based on the laws of quantum mechanics described by linear algebra and linear ordinary differential equations (ODE) in (Feynman, 1986). Feynman provides an exact map between any universal function and the dynamics of a quantum mechanical system. Such a map, similar to that of a neural network, is also fully differentiable in its parameters. Moreover, it also linear and reversible due to the fundamental linear nature of quantum mechanics. Ever since Feynman’s historical paper, continuous efforts have resulted in surprising discoveries in quantum physics and a plethora of quantum algorithms, e.g. (Shor, 1994; Grover, 1996), that possess provable advantages over the classical counterparts. A crucial connection remains to be made, between quantum dynamics and the dynamics of propagation in a neural network before one can utilize fruitful results in the quantum realm to design better RNNs.
RNN formulation has known several transformations, which facilitated deeper insights into regarding both the dynamical process of RNN and the underlying physics behind the data. The first thrust of
RNN revolution is based on our knowledge of the time sequentiality of the data, which addresses the complexity overhead of conventional deep neural network, but worsened the stability problem. The second wave of RNN proposals addresses the stability by ad hoc architectural changes that use only heuristic and empirical knowledge of the underlying mechanics. This leads us to attempt at establishing theoretical foundations for RNN and neural network in general, initiated by recent works such as Arjovsky et al. (2016b); Mhaskar and Poggio (2016) and Haber and Ruthotto (2017), to apply physics with numerical stability analysis used in solving physical problems for understanding and designing new machine learning architectures.
Towards establishing this third wave of RNN revolution, we formulate a framework that unifies different proposals of RNN architectures according to the order of nonlinearity and the order of temporal memory scale of the underlying dynamics embodied in the integration of ODEs, see Table. 1. To establishe the exact connection between quantum dynamics and RNN dynamics, we show in Sec. 2 that the update rules of RNN can indeed describe the model proposed by Feynman for encoding universal functions in a reversible manner. Lastly for reducing complexity while providing stability guarantees, in Sec. 3 we propose an embedding, called Quantum inspired Universal computing recurrent Neural Netowrk (QUNN), with a complexity of the optimization that grows linearly with the temporal correlation length between input data but independent of the dimension of data itself.

heading: 1 STABLE RECURRENT NEURAL NETWORK
text: The success of supervised machine learning techniques depends on the stability, the representability and the computational overhead associated with the proposed training architecture. A generic neural network without any structure, however, is susceptible to exploding or vanishing gradients and requires additional heuristic optimization techniques to suppress such instability.
The stability of deep neural networks can be understood by a simple connection between the integration of discrete ODE and neural network forward and backward propagation (Haber and Ruthotto, 2017). Take a type of stable deep ResNet proposed by Haber et. al. for example: let lth layer of hidden variable be Yl ∈ Rs×p and bias be bl ∈ Rs×p, to ensure the stability of propagation, they introduce a conjugate variable Zl± 12 ∈ R
s×p as a intermediate step such that the propagation of neural network is described by
Zl+ 12 = Zl− 1 2 − hlσ(WTl Yl + bl), Yl+1 = Yl + σ(WlZl+ 12 + bl). (1)
The dynamics of the above discrete ODE is stable regardless of the form of weight matrix Ŵl (Haber and Ruthotto, 2017).
In this section, we will extend upon the approach first introduced by Haber and Ruthotto (2017) to include existing architectures of RNN, such as LSTM, GRU, UNN and CW-RNN, as special instances under the a unified framework. We first define an ODE recurrent neural network with nth order in nonlinearity and tth order in time-derivative (n-t-ORNN) according to its propagation rule: the update of n-t-ORNN can be mapped to a generalized nth order Runge–Kutta integration. The specific choice of Runge-Kutta method is not essential to such generalization, and can be replaced by other integration method as well. We then analyze the corresponding n-t-ORNN family different existing RNN architectures belongs. Lastly, we define the n-2-ORNN with anti-Hermitian weight matrices as n-ARNN and prove the stability of 1-ARNN and 2-ARNN.
1.1 nth ORDER ODE RECURRENT NEURAL NETWORK
Definition 1. An ODE recurrent neural network of nth order in nonlinearity tth order in gradient (nt-ORNN), with integers n, t ≥ 1 and k ∈ [n], is described by the update rule between input state value Yl ∈ Rs at time step l, the hidden variables of the kth layer as Klk ∈ Rp with 1 ≤ k ≤ n, and output state value Yl+1 ∈ Rs as
Kl1 = σl1 (Wl1Yl + bl1) , Klq = γlqKlq−1 + κlqσlq ( WlqYl+tq−1 + blq + h q−1∑ k=1 αk ◦Klk ) ,
(2)
Yl+tn = γln+1Yl + κln+1σl ( WlnYl+tn−1 + bln+1 + h n∑ k=1 βkKlk ) (3)
where 2 ≤ q ≤ n; the time corresponding to each hidden layer obeys tk = bt knc with the overall time steps shared by the n hidden layers being t; the pointwise activation function σ∗(◦) : Rn → Rq at each layer is a nonlinear map that preserves the dimension of the input q; the weight matrix at each layer is represented by Wl∗ ∈ Rq2×q1 where q1 is the dimension of the input variable and q2 is the dimension of the output variable; βk, γlm , κlm and αjk ∈ Rp×p are matrices served to rescale and rotate the hidden variables; γk is a square matrix that manifests the energy dissipation of the ODE dynamics. Below, we analyze some of the most widely used RNN architectures in regard to the nonlinearity and memory scale of their underlying dynamics.
Claim 1. Both LSTM and GRU belong to the 2-L-ORNN.
Proof : For one layer RNN, we have the update rule for LSTM (Hochreiter and Schmidhuber, 1997) as:
Kt = ft ◦Kt−1 + it ◦ σ2(WcYt−1 + UcKt−1 + bc), Yt = ot ◦ σ1(Kt) (4)
with vector coefficient determined by
ft = σ(WfYt−1 + UfKt−1 + bf ), (5) it = σ(WiYt−1 + UiKt−1 + bi), ot = σ(WoYt−1 + UoKt−1 + bo) (6)
which is equivalent to setting n = 2, t = 1, γlm2 = D[ft], κm2 = D[it],Wlm2 = Wc, blm2 = bc, hα21 = Uc and γlm = 0, κlm = ot in n-t-ORNN. Notice that the weight matrix in ORNN can depend on time and is therefore able to include the memory dependency from Kt−1. We use D[a] to represent a p× p diagonal matrix with each diagonal element equal to each element of the vector a of length p. This is because the Hadamard product between two vectors can be re-written as diagonal matrix matrix multiplication with the second vector: a ◦ b = D[a]b. For multi-layer LSTM with L hidden layers, the only change is that the diagonal matricesD[ft], D[it] and D[ot] are generalized to D[f lt ], D[i l t] and D[o l t], which not only depend on the hidden variable of the same layer from the previous time step, but also the hidden variable of the same time step from a previous layer:
f lt = σ(WfK l−1 t + UfKt−1 + b l f ), i l t = σ(WiK l−1 t + UiKt−1 + b l i), o l t = σ(WoK l−1 t + UoKt−1 + b l o)
(7)
where K0t = Yt−1, and thus the nonlinearity of the ODE increases by one when the number of hidden layers increase by one, thus gives L-2-ORNN for a L layer architecture.
For one layer GRU (Cho et al., 2014), we have the update rule as:
Yt = (1− z) ◦ Yt−1 + z ◦ tanh (WtYt−1 +Wgr ◦ Yt−1) , with z = σ(W zl Yt−1), r = σ(W rl Yt−1) (8)
we can rewrite r ◦ Yt−1 as σ′(W ql Yt−1) and thus simplify the update rule to
Yt = (1− z) ◦ Yt−1 + z ◦ tanh (WtYt−1 +Wgσ′(W ql Yt−1)) (9)
which is equivalent to setting n = 1, γlm1 = D[(1−z)], κm1 = D[z],Wlm1 =Wt, blm2 = 0, hβ1 = Wg in n-ORNN. This can be similarly generalized to multi-layer GRU with L total hidden layers by allowing the weight matrices D[(1− z)] and D[z] to also depend on same layer hidden variable of previous step:
z = σ(W zl Y l t−1 +W
z′
l Y l−1 t ), r = σ(W r l Yt−1 +W
r′
l Y l−1 t ) (10)
which for lth layer it corresponds to L-2-ORNN. Q.E.D.
Claim 2. Unitary evolution RNN (Arjovsky et al., 2016a) with L hidden layers belongs to the 2-L-ORNN.
Proof : The propagation rule of URNN between the input to the RNN at time step 1 ≤ t ≤ T : Yt and hidden variables at the same time step Kt and output to the RNN of the same time step t as Yt+1 is:
Kt+1 = σ(WlKt + VlYt), for 1 ≤ l ≤ t, Yt+1 =Wl+1Kt+1 + bl+1, (11)
which corresponds to setting γjj = 0 and choosing n = 2 and t = L in Eq. (2)–(3). URNN thus belongs to 2-L-ORNN. Q.E.D.
Claim 3. Clockwork RNN (Koutnik et al. (2014)) with L clocks belongs to the L-L-ORNN.
Proof : The propagation rule for CW-RNN between input Yt at time step t, hidden layers at the same time step Kt as well as from the previous time step Kt−1 and output Yt+1 is described by:
Kt = σh (WH(t)Kt−1 +WI(t)Yt) , Yt+1 = σo (WoKt) (12)
where the time-dependent weight matrices WH(t) and WI(t) are structured to store memory of previous time steps in into different blocks with increasing duration of time delays such that effectively one can rewriteWH(t)Kt−1 = ∑ j Wjσ t−j−1 j WH(j)Kj+WI(j)Yj contributions from all previous step iteratively, and so is the clock structure in WI(t)Yt which contributes to all hidden layers after t. This is equivalent to setting n = t = L and γjj = 0 and t = 1 in Eq. (2)–(3). CW-RNN thus belongs to L-L-ORNN. Q.E.D.
Definition 2. The nth order ODE anti-Hermitian recurrent neural network (n-ARNN) corresponds to setting all weight matrices in n-2-ORNN with anti-Hermitian matrices.
Theorem 1. 1-ARNN with monotonic activation function σ∗(·) : Rn → Rn and purely imaginary anti-Hermitian weight matrix is stable for small enough h such that |hmaxk λ[Wlk ]| < 1. Proof : This will be proven in Theorem 4, where the original complex anti-Hermitian matrix is embedded into a Hilbert space twice as large such that a purely imaginary anti-Hermitian weight matrix guarantees the stability of the first order integration method. Q.E.D.
Theorem 2. Both 2-ARNN and 1-ARNN are reversible.
Proof : Since 2-ARNN corresponds to the first order mid-point integration and 1-ARNN corresponds to the symplectic Euler intergration their reversibility is guaranteed by the reversibility of these two integration schemes inside the stble regime. Q.E.D.
It is notable that the definition of n-t-ORNN does not restrict weight matrices to be time independent. This setup is less restrictive than conventional definition of RNN and is indispensable for generalizing various architectures of RNN under the same framework. Such generalization, however, is wellfounded in ODE framework: a generic ODE does not have to be time-independent.
The unification of different RNN architectures through n-t-ORNN prompts a better usage and obtainment of the memory scale and degree of nonlinearity of the data to further reduce the necessary complexity in the learning architecture. We demonstrate such combination between bottom-up data centered and top-down model centered approach in finding more efficient RNN design in the next section.

heading: 2 QUANTUM DYNAMICS AND PROPAGATION OF RECURRENT NEURAL NETWORK
text: In this section, we start by defining mathematical notations that make the connection between quantum and classical computation more self-evident. We also review previous work by Feynman in reversible classical computation through quantum dynamical evolution. Since the underlying mathematical structure of quantum mechanics is none other than linear algebra, Table. 3, represents the required mapping between quantum and classical audiences. For the sake of consistency, we will henceforth adopt the quantum mechanical notations of linear algebra. Subsequently, we will establish a key connection between RNN and quantum dynamics: the equivalence between the discretized evolution of a quantum system and the propagation of an RNN up to inversely polynomial errors in the total time steps and spectrum norm of the ODE characteristic function. Since any existing classical solver is discrete in nature but can solve time-dependent quantum dynamics to a given accuracy, our results is general. Such specific connection forms the basis of our stable and efficient RNN ansatz to be discussed in the following section.
The dynamics of a quantum system can be described by a first order ordinary differential equation, namely the Schrödinger equation, where the quantum state parameter represented by the complex vector |ψ(t)〉 at time t obeys: d
dt |ψ(t)〉 = −iĤ|ψ(t)〉, (13)
where Ĥ is called the quantum Hamiltonian that determines the dynamical evolution of the state parameters. Stepping back to the world of linear algebra, the Hamiltonian matrix is essentially the gradient with respect to time in the first order linear ODE. Despite such fundamental linearity, the emergent phenomena in a sub-region of the quantum system can be highly nonlinear and intriguing.
The power of quantum dynamics in computation was first demonstrated by Richard Feynman in his proposal of a quantum computer: any Boolean function can be encoded into reversible evolution of a quantum system under a carefully chosen system Hamiltonian with total number of Hamiltonian terms equal the number of logical gates that describe the given Boolean function.
Theorem 3. Any uniform family of Boolean functions f : {0, 1}n → {0, 1}n can be mapped to a unique fixed point of ODE evolution with its characteristic function containing polynomial in n many parameters.
The complete proof is given in references (Feynman, 1986; Kitaev et al., 2002; Aharonov et al., 2008) and is reviewed in supplementary material A. To utilize this result for RNN design, we establish the connection between RNN propagation and discretized quantum evolution in Theorem 4.
Theorem 4. Evolution of a closed quantum system of dimension 2n under the Hamiltonian Ĥ(t) for time T can be approximated by the stable propagation of an RNN with anti-Hermitian weight matrix of size 2n+1, using T ||Ĥ||∞ timestep and incurring errors of order O ( (1/T ||Ĥ||∞)2 ) .
Proof : It is shown by McKague et al. (2009) that any complex Hamiltonian can be mapped to a real Hamiltonian Ĥ(t) at any time t with constant overhead in computation basis. Using their results, we assume our quantum Hamiltonian is real in each element, and separate the real part |P 〉 and imaginary part |C〉of the quantum state as |ψ〉 = |P 〉 + i|C〉. After discretization, the ODE that corresponds to Eq. (13) can be solved by symplectic Euler integration that is represented by an RNN with anti-symmetric weight matrix and identity activation function (Arjovsky et al., 2016b):[
P (n+ 1) C(n+ 1)
] = [ I2n+1×2n+1 δtĤ(n)
−δtĤ(n) I2n+1×2n+1 − δt2Ĥ2(n)
] [ P (n) C(n) ] (14)
with stable regime bounded by |δt||Ĥ||∞| < 2 satisfiable by |δt||Ĥ||∞| < 1. This gives a minimum time step of T ||Ĥ||∞ and in turn an error of order O ( (1/T ||Ĥ||∞)2 ) . Q.E.D.

heading: 3 QUANTUM INSPIRED UNIVERSAL COMPUTING NEURAL NETWORK
text: To design stable RNN architectures without incurring unwanted complexity, we propose a new RNN ansatz that uses input data itself to construct weight matrices: QUNN based on our physical knowledge of quantum computation and the proof of equivalence between RNN and quantum evolution.
To facilitate a more efficient training process, we reduce the total degree of freedom in the training parameters of our quantum inspired RNN ansatz from polynomial in both input data size and timecorrelation length, i.e., the memory range, of the dynamics, to only linear in the memory range. We achieve this by constructing time-dependent weight matrices in hidden layers from the input data. Such construction makes the most out of the knowledge of manifested time-correlated structure of the data itself in place of a conventional weight matrix ansatz that is data independent. QUNN is thus able to speedup training processes where input data are exponentially larger than the correlation time of the data structure.
Definition 3. Quantum inspired universal computing neural network: a recurrent neural network architecture that adopts the update rule between an input state |Yl〉, which could be either a binary, real, or complex vector depending on the problem type, and output of the network Yl+1 at the integer time step l ∈ {1, 2, ...., N} according to three stages. In the first emedding stage the incoming data is transformed into itself tensor producted with a clock state that marks the relative time sequence through the embedding weight matrix Êl:
|Kl〉 = σ1 ( Êl|Yl〉 ) = σ1 (|Yl〉 ⊗ |l〉) , Êl = |l〉 ⊗ ∑ j |j〉〈j| (15)
where σi(·) represents the monotonic and continuously differentiable point-wise nonlinear function of the ith layer. In the second stage, the hidden layer |Kl〉 is updated according to |K ′l〉 = Ŝ1|Kl〉+ σ2 ( Ĥl|Kl〉 ) , (16)
Ĥl = D̂lŴl (17)
D̂l = (1− p1(l))I ⊗ Ic + p1(l) (I ⊗ |l + 1〉〈l|c − I ⊗ |l〉〈l + 1|c) (18) Ŵl = (1− p2(l)Ŵl−1 + p2(l) (|Yl+1〉〈Yl| ⊗ |l + 1〉〈l| − |Yl〉〈Yl+1| ⊗ |l〉〈l + 1|) (19)
where the Dl consists of identity matrix weighted by 1 − p1(l) and a time re-ordering operator weighted by p1. Notice that |1〉〈l − 1|c is the generator of the permutation group of clock states which adds noise as well as correction to possibly mislabeled time sequence of training data. Such a dispersion step is in product with Ŵl, which records the flexible range of memory important for the training: the dependence of p2(l) on time step l will determine how long the memory lasts.
In the last stage, the state is mapped back to the original dimension by projecting on to the corresponding clock state of the next time step:
|Yl+1〉 = Ŝ2|Yl〉+ σ3 ( Ûl|K ′l〉 ) = σ3 ∑ j |j〉〈j| ⊗ 〈l + 1|K ′l〉  . (20) Notice that our weight matrix ansatz depends on the time step l, which is kept from the first input until the last input of the same set of time-sequential data and is reset to 1 at the beginning of each time-sequential data set. This is different from conventional definition of RNN where the weight matrix does not explicitly depend on time, but such memory dependence is indirectly actuated through the gate construction such as the forgetting unit fi.
The multi-layer generalization of the weight matrix construction in Eq. (19) to facilitate longer and longer time step rotation as the hidden layer number ln increases takes the form:
D̂ln = (1− p1(ln))I ⊗ Ic + p1(ln) (I ⊗ |l〉〈l − n| c − I ⊗ |l − n〉〈l|c) (21)
Ŵln = (1− p2(ln)Ŵ(l−1)n + p2(l2) (|Yl〉〈Yl−n| ⊗ |l + n〉〈l| − |Yl−n〉〈Yl| ⊗ |l〉〈l + n|) , (22) which can be understood as effectively implementing a different order of integrating a time-dependent Hamiltonian evolution of discretized Eq. (13), as illustrated in Fig. 1.
The existence of QUNN is guaranteed by Theorem 1. Moreover, the Hermitian matrix as the weight matrix of a neural network also secures its stability: its eigenvalues are always purely imaginary and thus guarantee the stability of the QUNN when choosing a leapfrog integration method (Haber and Ruthotto, 2017).

heading: 3.1 QUNN VS LSTM
text: With deeper understandings of RNN architectures provided in Sec. 1 in their nonlinearity and the memory scale of the underlying dynamics, we now analyze the connection between the quantum inspired RNN ansatz, proposed in the previous section, and one of the most widely used LSTM.
It is straightforward to see from the update rule of Eq. (15)–(15) that a QUNN with L hidden layers corresponds to an Lth order of nonlinearity in the ODE integration method. Moreover, notice that our matrix weight of the nth layers defined in Eq. (21) is determined by input state in the previous n+ 1 step. Therefore, together QUNN with L+ 2 total layers correspond to a L-L-ORNN. This means the functionality of QUNN is never equivalent to LSTM for any value of L . QUNN take time-sequential data as input to formulate weight matrix trainable through gradient based method to predict output at the same time step. LSTM however does not depend on the data directly.
We are now well-positioned to compare different RNN architectures from a top-down angle based on physical properties of their underlying dynamics according to different orders of nonlinearity and time-dependency as shown in Table. 4. More particularly, QUNN have longer range of memory scale than both LSTM and URNN, i.e., the order of time derivatives in the corresponding ODE is higher in QUNN. This comes with a price of additional layers of embedding in RNN architecture seen in the depth difference. But QUNN possess stability by construction, which is not guaranteed in generic LSTM architecture. This show cases the distinction between an ad hoc heuristic approach and physical inspired approach to designing RNN. In the problems where long-term dependencies are important, such RNN stability becomes essential to effective learning without been hampered by the exploding or vanishing gradients.

heading: 4 CONCLUSION
text: We propose a generalized framework of ODE neural networks with nth order linearity tth order time derivatives which includes many existing RNN architectures, including LSTM, GRU, URNN, CW-RNN, and the quantum inspried ansatz QUNN, as special cases. Our proof of the equivalence of between quantum dynamics and RNN propagation forms the basis of an RNN design, the QUNN. We show that this architecture is provably stable and possesses less complexity overhead in the dimension of input data than generic LSTM architectures.
Our findings support the concept of harnessing physical knowledge of the data for constructing the corresponding machine learning tools. Additional knowledge about the time correlation length that manifests memory scale of the dynamics and the degree of nonlinearity of the dynamics behind the input data will help the design of appropriate RNN suitable for the given problem.

Review summaries:
The authors make connections between RNN dynamics and those of a class of ODEs similar to RNNs (ORNN) that has different orders of nonlinearity and order of gradients in time. They show that typical RNN architectures can be described as members of the ORNN family. They then make the connection that quantum mechanical systems can be described as following the Schrodinger equation which can be cast as a series of coupled 1st order ODEs of the evolution of wavefunctions under an influencing Hamiltonian. They then claim that these discretized equations can be represented by a RNN similar to a unitary RNN. They go on to outline a RNN structure inspired by this insight that has time-dependent activations to increase the scale of temporal dependence.