Recurrent Neural Networks are Universal Filters

∀ > 0 , ∃ RNN-based filter , s.t . lim sup k→∞ ∥∥x̂k|k − E [ xk|Yk ] ∥∥ < , where x̂k|k is RNN-based filter ’ s estimate of state xk at step k conditioned on the observation history and E [ xk|Yk ] is the conditional mean of xk , known as the optimal estimate of the state in minimum mean square error sense . As an interesting special case , the widely used Kalman filter ( KF ) can be synthesized by RNN . 1 INTRODUCTION . Recurrent neural network ( RNN ) is a certain type of neural networks characterized by hidden variables that memorize the history of input sequences , and it has been successfully applied and brought amazing results in many different disciplines including computer vision , natural language processing and optimal control , etc . ( Mikolov et al . ( 2010 ) , Graves et al . ( 2013 ) , Du et al . ( 2015 ) , Fei & Lu ( 2017 ) , Ma et al . ( 2015 ) ) . Its huge empirical success in different engineering disciplines is grounded on the expressive power of RNN . However , how to understand the expressive power of RNN in a quantitative way is not fully understood . Even what RNN expresses is not totally clear . Another type of neural network , deep feed forward neural network has been well characterized as a universal function approximator ( Hornik et al . ( 1989 ) , Park & Sandberg ( 1991 ) ) . However , a similar way to characterize the expressive power of RNN is not obvious . DNN is a mapping from a finite dimensional Euclidean space to another finite dimensional Euclidean space , that is to say it can be regarded as a vector-valued multivariate function . However , RNN is a mapping from a sequence space to another sequence space and the current output depends on both current input and the whole observation history . The input sequence , in principle , can be infinite . The function of RNN is capturing the relationship between input process and output process . For example , in machine translation , the input process ( or the observation process ) is sentence in one language and the output process ( or the state process ) is sentence in another language . And in many other RNN ’ s application scenarios such as traffic forecast and optimal control , the input is a noisy observation or measurement sequence and the output is an estimate sequence of a certain quantity , e.g. , the traffic speed .. We observe that the function of RNN is quite similar to a filter . In our paper , we propose to characterize the expressive power of RNN in a quantitative way from the perspective of filtering . We consider a discrete filtering system as 1 . { xk = f ( xk−1 ) + g ( xk−1 ) wk−1 , yk = h ( xk ) + vk , ( 1 ) where the state xk at time instant k is an n-dimensional vector , f is an n-dimensional vectorvalued function , g is an n × r matrix-valued function , { wk , k = 0 , 1 , · · · } is an r-dimensional white Gaussian process and wk ∼ N ( 0 , Q ) , where Q is the covariance matrix of wk . yk is the m-dimensional observation ( measurement ) process , h is an m-dimensional vector-valued function , { vk , k = 1 , · · · } is an m-dimensional white Gaussian process and vk ∼ N ( 0 , R ) , where R is the covariance matrix of vk . And we assume that { wk , k = 0 , 1 , · · · } , { vk , k = 1 , · · · } and the initial state x0 are jointly independent . We use Yk to denote the sequence of observations up to time instant k , i.e. , Yk : = { y1 , · · · , yk } . ( 2 ) Given the realization of the sequence of observations Yk , the aim of filtering problem is to compute the optimal estimate of xk conditioned on Yk . Not surprisingly , recurrent neural network has been proposed to do filtering . James Ting Ho Lo showed that Recurrent Multi-layer Perceptron can be used to synthesize optimal filter ( Lo ( 1994 ) ) . However , Lo ’ s approach is based on simply copying and storing the whole observation history in the hidden variables and thus require the time horizon to be finite , which is fundamentally limited . Besides Lo ’ s work , many efforts have been made to connect RNN and dynamical system . Wilson & Finkel ( 2009 ) implemented a neural network based Kalman Filter ( KF ) but did not provide theoretical analysis on the approximation error . Parlos et al . ( 2001 ) proposed an algorithmic approach to do nonlinear filtering using recurrent neural network architecture but did not provide theoretical gurantee . Schäfer & Zimmermann ( 2006 ) shows that recurrent neural network is a universal approximator of dynamical system . However , they only consider the deterministic system dynamics and do not analyze the filtering relationship between two stochastic processes . Around the same time our work was under review , a similar paper was also uploaded in arXiv by Lim et al . ( 2019 ) , The results in the arXiv paper are similar to the June , 2018 graduation thesis of the first author of the current paper in the use of sufficient statistics , and emphasize on the design and implementation of the algorithm . Our current paper has been dramatically improved by introducing the error analysis and asymptotic convergence of the algorithm . And our work is inspired by the finite dimensional filter such as Beneš filter Beneš ( 1981 ) . Compared to the existing work , we make the following specific contributions : • Motivated by the similarity between RNN and filter , we propose to use the ability to approximate optimal filter to characterize the expressive power of RNN . Unlike existing work on expressive power of RNN ( Schäfer & Zimmermann ( 2006 ) ) where only deterministic system is considered , we consider a stochastic dynamic system with noisy observations and analyze the capability of RNN to estimate unknown state . • We unify RNN-based filter into Bayesian filtering framework . In this framework , the hidden variables of RNN are interpreted as statistics of the observation history . And the evolution of hidden variables are interpreted as the evolution of statistics . • Based on the Bayesian filtering framework , we derive our main result : Recurrent Neural Networks ( RNN ) are universal approximators of a large class of optimal finite dimensional filters . That is to say , RNN estimator ’ s asymptotic estimation error can be as close to minimum mean square error as desired . As an interesting special case , the widely used Kalman Filter can be synthesized by RNN . The consideration of asymptotic error differentiates us from existing work on expressive power of RNN ( Schäfer & Zimmermann ( 2006 ) ) . 2 PRELIMINARY : THE BAYESIAN FRAMEWORK OF FILTERING . We first introduce the definition of minimum mean square error estimate . Definition 1 ( Minimum Mean Square Error ( MMSE ) Estimate ( Jazwinski ( 1970 ) ) ) . Let x̂ be an estimate of random variable x and LMSE : = ( x− x̂ ) T ( x− x̂ ) . The estimate that minimizes E [ LMSE ] is called the minimum mean square error estimate . Theorem 1 ( Theorem 5.3 in Jazwinski ( 1970 ) ) . Conditioned on the observation history Yk , the minimum mean square error estimate of state xk is the conditional mean E [ xk|Yk ] . Proof . Proof can be found in our appendix A.1 . The Bayesian filtering consists of recursive prediction and update procedures ( Jazwinski ( 1970 ) ) : Prediction Step p ( xk−1|Yk−1 ) → p ( xk|Yk−1 ) : Given the posterior distribution p ( xk−1|Yk−1 ) at instant k − 1 , the prior distribution p ( xk|Yk−1 ) of xk satisfies the Chapman-Kolmogorov equation : p ( xk|Yk−1 ) = ∫ p ( xk|xk−1 ) p ( xk−1|Yk−1 ) dxk−1 ; ( 3 ) Updating Step p ( xk|Yk−1 ) → p ( xk|Yk ) : Given prior distribution p ( xk|Yk−1 ) , when the observation yk at instant k arrives , the posterior distribution p ( xk|Yk ) at instant k is given by equation 4 , p ( xk|Yk ) = p ( yk|xk ) p ( xk|Yk−1 ) ∫ p ( yk|xk ) p ( xk|Yk−1 ) dxk . ( 4 ) Then we can get the MMSE estimate by simply doing integration : E [ xk|Yk ] = ∫ p ( xk|Yk ) xk dxk . ( 5 ) To facilitate subsequent discussion , we make the following definition . Definition 2 ( Sufficient Statistic ( Beneš ( 1981 ) ) ) . If the conditional distribution p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) can be fully determined by a vector-valued function sk|k ( or sk|k−1 ) of the observation sequence Yk ( or Yk−1 ) , then we say sk|k ( or sk|k−1 ) is a sufficient statistic for p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) .1 Because sufficient statistic sk|k fully determines the posterior distribution p ( xk|Yk ) and MMSE estimate E ( xk|Yk ) is a functional of p ( xk|Yk ) , there exists a function γ that maps sk|k to E [ xk|Yk ] ,2 i.e. , E [ xk|Yk ] = γ ( sk|k ) . ( 6 ) 3 RNN BASED FILTER ’ S ARCHITECTURE . Motivated by the Bayesian framework of filtering , we propose the RNN based filter ’ s architecture as shown in Fig . 1 . Our RNN based filter ’ s architecture , Bayesian Filter Net ( BFN ) , consists of three parts : prediction network , update network and estimation network . To mimic the prediction step in Bayesian filtering , we use the prediction network to map the posterior distribution representation vector to a prior distribution representation vector . To mimic the update step in Bayesian filtering , we then use an update network to update the prior state distribution representation vector and the observation to get the posterior state distribution representation vector . Finally , we use an estimation network to map the current posterior state distribution representation vector to current estimation x̂k . We will see in the subsequent discussions , the so-called representation vector or hidden variables indeed can be interpreted as statistics of the underlying conditional distribution . 1 The sufficient statistic sk|k ( or sk|k−1 ) can be any vector-valued function of Yk ( or Yk−1 ) as long as it can fully determine p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) . Taking linear system as an example , sk|k can be the vector composed of the conditional mean and covariance because these two quantities sufficiently determine the conditional density function of the state in linear filtering system . ( See more details in our Appendix A.3 . ) 2For instance , as for the Kalman Filter in linear system , the sufficient statistic sk|k can be chosen to be the conditional mean and conditional covariance , thus γ function can be the function that copy the conditional mean part of sk|k .

Summary: This paper shows that RNN (of infinite horizon) can be universal approximators  for any stochastic dynamics system. It does that by using the Bayesian framework of filtering, which shows that if sufficient statistics given the observations so far can be computed at each stage of filtering, then the expected hidden states at each stage can be estimated as well. The paper then follows this framework by constructing deep neural network mapping from “observations so-far” to “sufficient statistics” and by universal approximation theorem this is possible. Then as long as after each stage, the image of such a mapping is bounded into a certain region, and if the mapping is contractive (|C_\psi C_\phi| < 1), this can be applied to arbitrarily long sequence (and thus infinite horizon).