Recurrent Neural Networks are Universal Filters

∀ > 0 , ∃ RNN-based filter , s.t . lim sup k→∞ ∥∥x̂k|k − E [ xk|Yk ] ∥∥ < , where x̂k|k is RNN-based filter ’ s estimate of state xk at step k conditioned on the observation history and E [ xk|Yk ] is the conditional mean of xk , known as the optimal estimate of the state in minimum mean square error sense . As an interesting special case , the widely used Kalman filter ( KF ) can be synthesized by RNN . 1 INTRODUCTION . Recurrent neural network ( RNN ) is a certain type of neural networks characterized by hidden variables that memorize the history of input sequences , and it has been successfully applied and brought amazing results in many different disciplines including computer vision , natural language processing and optimal control , etc . ( Mikolov et al . ( 2010 ) , Graves et al . ( 2013 ) , Du et al . ( 2015 ) , Fei & Lu ( 2017 ) , Ma et al . ( 2015 ) ) . Its huge empirical success in different engineering disciplines is grounded on the expressive power of RNN . However , how to understand the expressive power of RNN in a quantitative way is not fully understood . Even what RNN expresses is not totally clear . Another type of neural network , deep feed forward neural network has been well characterized as a universal function approximator ( Hornik et al . ( 1989 ) , Park & Sandberg ( 1991 ) ) . However , a similar way to characterize the expressive power of RNN is not obvious . DNN is a mapping from a finite dimensional Euclidean space to another finite dimensional Euclidean space , that is to say it can be regarded as a vector-valued multivariate function . However , RNN is a mapping from a sequence space to another sequence space and the current output depends on both current input and the whole observation history . The input sequence , in principle , can be infinite . The function of RNN is capturing the relationship between input process and output process . For example , in machine translation , the input process ( or the observation process ) is sentence in one language and the output process ( or the state process ) is sentence in another language . And in many other RNN ’ s application scenarios such as traffic forecast and optimal control , the input is a noisy observation or measurement sequence and the output is an estimate sequence of a certain quantity , e.g. , the traffic speed .. We observe that the function of RNN is quite similar to a filter . In our paper , we propose to characterize the expressive power of RNN in a quantitative way from the perspective of filtering . We consider a discrete filtering system as 1 . { xk = f ( xk−1 ) + g ( xk−1 ) wk−1 , yk = h ( xk ) + vk , ( 1 ) where the state xk at time instant k is an n-dimensional vector , f is an n-dimensional vectorvalued function , g is an n × r matrix-valued function , { wk , k = 0 , 1 , · · · } is an r-dimensional white Gaussian process and wk ∼ N ( 0 , Q ) , where Q is the covariance matrix of wk . yk is the m-dimensional observation ( measurement ) process , h is an m-dimensional vector-valued function , { vk , k = 1 , · · · } is an m-dimensional white Gaussian process and vk ∼ N ( 0 , R ) , where R is the covariance matrix of vk . And we assume that { wk , k = 0 , 1 , · · · } , { vk , k = 1 , · · · } and the initial state x0 are jointly independent . We use Yk to denote the sequence of observations up to time instant k , i.e. , Yk : = { y1 , · · · , yk } . ( 2 ) Given the realization of the sequence of observations Yk , the aim of filtering problem is to compute the optimal estimate of xk conditioned on Yk . Not surprisingly , recurrent neural network has been proposed to do filtering . James Ting Ho Lo showed that Recurrent Multi-layer Perceptron can be used to synthesize optimal filter ( Lo ( 1994 ) ) . However , Lo ’ s approach is based on simply copying and storing the whole observation history in the hidden variables and thus require the time horizon to be finite , which is fundamentally limited . Besides Lo ’ s work , many efforts have been made to connect RNN and dynamical system . Wilson & Finkel ( 2009 ) implemented a neural network based Kalman Filter ( KF ) but did not provide theoretical analysis on the approximation error . Parlos et al . ( 2001 ) proposed an algorithmic approach to do nonlinear filtering using recurrent neural network architecture but did not provide theoretical gurantee . Schäfer & Zimmermann ( 2006 ) shows that recurrent neural network is a universal approximator of dynamical system . However , they only consider the deterministic system dynamics and do not analyze the filtering relationship between two stochastic processes . Around the same time our work was under review , a similar paper was also uploaded in arXiv by Lim et al . ( 2019 ) , The results in the arXiv paper are similar to the June , 2018 graduation thesis of the first author of the current paper in the use of sufficient statistics , and emphasize on the design and implementation of the algorithm . Our current paper has been dramatically improved by introducing the error analysis and asymptotic convergence of the algorithm . And our work is inspired by the finite dimensional filter such as Beneš filter Beneš ( 1981 ) . Compared to the existing work , we make the following specific contributions : • Motivated by the similarity between RNN and filter , we propose to use the ability to approximate optimal filter to characterize the expressive power of RNN . Unlike existing work on expressive power of RNN ( Schäfer & Zimmermann ( 2006 ) ) where only deterministic system is considered , we consider a stochastic dynamic system with noisy observations and analyze the capability of RNN to estimate unknown state . • We unify RNN-based filter into Bayesian filtering framework . In this framework , the hidden variables of RNN are interpreted as statistics of the observation history . And the evolution of hidden variables are interpreted as the evolution of statistics . • Based on the Bayesian filtering framework , we derive our main result : Recurrent Neural Networks ( RNN ) are universal approximators of a large class of optimal finite dimensional filters . That is to say , RNN estimator ’ s asymptotic estimation error can be as close to minimum mean square error as desired . As an interesting special case , the widely used Kalman Filter can be synthesized by RNN . The consideration of asymptotic error differentiates us from existing work on expressive power of RNN ( Schäfer & Zimmermann ( 2006 ) ) . 2 PRELIMINARY : THE BAYESIAN FRAMEWORK OF FILTERING . We first introduce the definition of minimum mean square error estimate . Definition 1 ( Minimum Mean Square Error ( MMSE ) Estimate ( Jazwinski ( 1970 ) ) ) . Let x̂ be an estimate of random variable x and LMSE : = ( x− x̂ ) T ( x− x̂ ) . The estimate that minimizes E [ LMSE ] is called the minimum mean square error estimate . Theorem 1 ( Theorem 5.3 in Jazwinski ( 1970 ) ) . Conditioned on the observation history Yk , the minimum mean square error estimate of state xk is the conditional mean E [ xk|Yk ] . Proof . Proof can be found in our appendix A.1 . The Bayesian filtering consists of recursive prediction and update procedures ( Jazwinski ( 1970 ) ) : Prediction Step p ( xk−1|Yk−1 ) → p ( xk|Yk−1 ) : Given the posterior distribution p ( xk−1|Yk−1 ) at instant k − 1 , the prior distribution p ( xk|Yk−1 ) of xk satisfies the Chapman-Kolmogorov equation : p ( xk|Yk−1 ) = ∫ p ( xk|xk−1 ) p ( xk−1|Yk−1 ) dxk−1 ; ( 3 ) Updating Step p ( xk|Yk−1 ) → p ( xk|Yk ) : Given prior distribution p ( xk|Yk−1 ) , when the observation yk at instant k arrives , the posterior distribution p ( xk|Yk ) at instant k is given by equation 4 , p ( xk|Yk ) = p ( yk|xk ) p ( xk|Yk−1 ) ∫ p ( yk|xk ) p ( xk|Yk−1 ) dxk . ( 4 ) Then we can get the MMSE estimate by simply doing integration : E [ xk|Yk ] = ∫ p ( xk|Yk ) xk dxk . ( 5 ) To facilitate subsequent discussion , we make the following definition . Definition 2 ( Sufficient Statistic ( Beneš ( 1981 ) ) ) . If the conditional distribution p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) can be fully determined by a vector-valued function sk|k ( or sk|k−1 ) of the observation sequence Yk ( or Yk−1 ) , then we say sk|k ( or sk|k−1 ) is a sufficient statistic for p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) .1 Because sufficient statistic sk|k fully determines the posterior distribution p ( xk|Yk ) and MMSE estimate E ( xk|Yk ) is a functional of p ( xk|Yk ) , there exists a function γ that maps sk|k to E [ xk|Yk ] ,2 i.e. , E [ xk|Yk ] = γ ( sk|k ) . ( 6 ) 3 RNN BASED FILTER ’ S ARCHITECTURE . Motivated by the Bayesian framework of filtering , we propose the RNN based filter ’ s architecture as shown in Fig . 1 . Our RNN based filter ’ s architecture , Bayesian Filter Net ( BFN ) , consists of three parts : prediction network , update network and estimation network . To mimic the prediction step in Bayesian filtering , we use the prediction network to map the posterior distribution representation vector to a prior distribution representation vector . To mimic the update step in Bayesian filtering , we then use an update network to update the prior state distribution representation vector and the observation to get the posterior state distribution representation vector . Finally , we use an estimation network to map the current posterior state distribution representation vector to current estimation x̂k . We will see in the subsequent discussions , the so-called representation vector or hidden variables indeed can be interpreted as statistics of the underlying conditional distribution . 1 The sufficient statistic sk|k ( or sk|k−1 ) can be any vector-valued function of Yk ( or Yk−1 ) as long as it can fully determine p ( xk|Yk ) ( or p ( xk|Yk−1 ) ) . Taking linear system as an example , sk|k can be the vector composed of the conditional mean and covariance because these two quantities sufficiently determine the conditional density function of the state in linear filtering system . ( See more details in our Appendix A.3 . ) 2For instance , as for the Kalman Filter in linear system , the sufficient statistic sk|k can be chosen to be the conditional mean and conditional covariance , thus γ function can be the function that copy the conditional mean part of sk|k .

Summary: The paper attempts to establish the asymptotic accuracy of the "RNN" (but not the RNN models that are well-known in the literature - see the below comments) as a universal functional approximator. It considers a general state space model and uses feedforward neural nets (FNN) to learn the filtering and forecast distributions. Based on the well-known universal approximation property of FNNs, the paper shows that their RNN-based filter can approximate arbitrarily well the optimal filter.