Recurrent Neural Networks are Universal Filters
Authors: Wenjie Xu
Xiuqiong Chen
Stephen S.-T. Yau
Venue:  (2020)
Keywords: Recurrent Neural Networks
Expressive Power
Deep Learning Theory

Abstract: Recurrent neural networks (RNN) are powerful time series modeling tools in ma-
chine learning. It has been successfully applied in a variety of fields such as natural
language processing (Mikolov et al. (2010), Graves et al. (2013), Du et al. (2015)),
control (Fei & Lu (2017)) and traffic forecasting (Ma et al. (2015)), etc. In those
application scenarios, RNN can be viewed as implicitly modelling a stochastic dy-
namic system. Another type of popular neural network, deep (feed-forward) neural
network has also been successfully applied in different engineering disciplines,
whose approximation capability has been well characterized by universal approxi-
mation theorem (Hornik et al. (1989), Park & Sandberg (1991), Lu et al. (2017)).
However, the underlying approximation capability of RNN has not been fully
understood in a quantitative way. In our paper, we consider a stochastic dynamic
system with noisy observations and analyze the approximation capability of RNN
in synthesizing the optimal state estimator, namely optimal filter. We unify the
recurrent neural network into Bayesian filtering framework and show that recurrent
neural network is a universal approximator of optimal finite dimensional filters
under some mild conditions. That is to say, for any stochastic dynamic systems
with noisy sequential observations that satisfy some mild conditions, we show that
(informal)
∀ > 0, ∃ RNN-based filter, s.t. lim sup x̂ k|k − E[x k |Y k ] < ,
k→∞
where x̂ k|k is RNN-based filter’s estimate of state x k at step k conditioned on
the observation history and E[x k |Y k ] is the conditional mean of x k , known as the
optimal estimate of the state in minimum mean square error sense. As an interesting
special case, the widely used Kalman filter (KF) can be synthesized by RNN.

Under review as a conference paper at ICLR 2020
RECURRENT NEURAL NETWORKS ARE UNIVERSAL FILTERS
Anonymous authors Paper under double-bling review
ABSTRACT
Recurrent neural networks (RNN) are powerful time series modeling tools in ma- chine learning. It has been successfully applied in variety of fields such as natural language processing (iikolov et al. (2010), Graves et al. (2013), ) Du et al. (2015)), control (Fei & Lu (2017)) and traffic forecasting (Ma al. (2015)), etc. In those application scenarios, RNN can be viewed as implicitly modelling a stochastic dy- namic system. Another type popular neural network, deep (feed-forward) neural network has also been successfully applied in different engineering disciplines, whose approximation capability has been well characterized by universal approxi- mation theorem (Hornik et al. (1989). Park & Sandberg 11911, Lu et al. (2017)) However, the underlying approximation capability of RNN has not been fully understood in quantitative way. In our paper, consider stochastic dynamic system with noisy observations and analyze the approximation capability of RNN in synthesizing the optimal state estimator, namely optimal filter. We unify the recurrent neural network into Bayesian filtering framework and show that recurrent neural network is universal approximator of optimal finite dimensional filters under some mild conditions. That is to say, for any stochastic dynamic systems with noisy sequential observations that satisfy some mild conditions, we show that (informal)
Ve RNN-based filter, s.t. lim sup | [((rx/Ya]]] <e,
k-+00
where faJA is RNN-based filter's estimate of state at step k conditioned on the observation history and E[xg [Y,] [ is the conditional mean of æx. known as the optimal estimate of the state in minimum mean square error sense. As interesting special case, the widely used Kalman filter (KF) can be synthesized by RNN.
INTRODUCTION
Recurrent neural network (RNN) is certain type of neural networks characterized by hidden variables that memorize the history of input sequences, and has been successfully applied and brought amazing results in many different disciplines including computer vision, natural language processing and optimal control, etc. (Mikolov et al. (2010), Graves et (2013). Du et al.) (2015). Fei & Lu (011)), Ma al. (2015)). Its huge empirical success in different engineering disciplines is grounded on the expressive power of RNN, However, how to understand the expressive power of RNN quantitative way is not fully understood. Even what RNN expresses not totally clear. Another type of neural network, deep feed forward neural network has been well characterized as universal
batrrsaafuuucco function approximator (Hornik al. (198)), Park Sandberg (199I)). However, a similar way to characterize the expressive power of RNN is not obvious.
DNN is a mapping from finite dimensional Fuclidean space to another finite dimensional Euclidean space, that is to say it can be regarded vector vector-valued multivariate function. However, RNN is mapping from a sequence space to another sequence space and the current output depends on both current input and the whole observation history. The input sequence, in principle, can be infinite. The function of RNN is capturing the relationship between input process and output process. For example, in machine translation, the input process (or the observation process) is sentence in one language and the output process (or the state process) is sentence in another language. And in many other RNN's application scenarios such traffic forecast and optimal control, the input is noisy

---

Under review as a conference paper at ICLR 2020
observation or measurement sequence and the output is an estimate sequence of certain quantity, e.g., the traffic speed.. We observe that the function of RNN is quite similar to a filter.
In our paper, we propose to characterize the expressive power of RNN in quantitative way from the perspective of filtering. We consider a discrete filtering system as T
Zk = [(xk-1) + Uv h((x))))))))))))()))))) + Uk:
(1)
where the state de at time instant k is an n-dimensional vector, ʃ is an n-dimensiioal vector- valued function, g is an n X r matrix-valued function, (w,, 0, 1,...) is an r-dimensional white Gaussian process and wkk ~ (0, Q), where Q is the covariance matrix of wg. yk is the m-dimensional observation (measurement) process, is an m-dimensional vector-valued function, {vk, h: 1, } is an m-dimensional white Gaussian process and U& ~ ~N(00, R),, where R is the covariance matrix of 0k. And we assume that {wx, k = 0, 1. ... }. (Uk, k 1,...) } and the initial state xu are jointly independent. We use Y2 to denote the sequence of observations up time instant k, i.e.,
Y == {y M.
Given the realization of the sequence of observations Ys. the aim of filtering problem is to compute the optimal estimate of conditioned on Yı.
Not surprisingly, recurrent neural network has been proposed to do filtering. James Ting Ho Lo showed that Recurrent Multi-layer Perceptron can be used to synthesize optimal filter (Lo (1994).
However, Lo's approach based on simply copying and storing the whole observation history in the hidden variables and thus require the time horizon to be finite, which is fundamentally limited.
Besides Lo's work, many efforts have been made to connect RNN and dynamical system. Wiilson & Finkel (2009) implemented neural network based Kalman Filter (KF) but did not provide theoretical analysis on the approximation error. Parlos et al. (200T) proposed an algorithmic approach to do nonlinear filtering using recurrent neural network architecture but did not provide theoretical gurantee.
Schäfer & Zimmermann (2006) shows that recurrent neural network is a universal approximator of dynamical system. However, they only consider the deterministic system dynamics and do not analyze the filtering relationship between two stochastic processes, Around the same time our work was under review, a similar paper was also uploaded in arXiv by [Lm al. (001)), The results in the arXiv paper are similar to the June, 2018 graduation thesis of the first author of the current paper in the use of sufficient statistics, and emphasize on the design and implementation of the algorithm. Our current paper has been dramatically improved by introducing the error analysis and asymptotic convergence of the algorithm. And our work is inspired by the finite dimensional filter such as Beneš filter Bene3 [181I).
Compared to the existing work, we make the following specific contributions:
Motivated by the similarity between RNN and filter, we propose to use the ability to approximate optimal filter to characterize the expressive power of RNN. Unlike existing work on expressive power of RNN (Schäfer & Zimmermann (2006) where only deterministic system is considered, we consider a stochastic dynamic system with noisy observations and analyze the capability of RNN to estimate unknown state.
We unify RNN-based filter into Bayesian filtering framework. In this framework, the hidden variables of RNN are interpreted as statistics of the observation history. And the evolution of hidden variables are interpreted as the evolution of statisties.
Based on the Bayesian filtering framework, we derive our main result: Recurrent Neural Networks (RNN) are universal approximators of a large class of optimal finite dimensional filters. That is to say, RNN estimator's s asymptotic estimation error can be as close to mini- mum mean square error as desired. As an interesting special case, the widely used Kalman Filter can be synthesized by RNN. The consideration of asymptotic error differentiates us from existing work on expressive power of RNN (Schäfer & Zimmermann (0066).
PRELIMINARY: THE BAYESIAN FRAMEWORK OF FILTERING
We first introduce the definition of minimum mean square error estimate.
2

---

Under review as a conference paper at ICLR 2020
Definition 1 (Minimum Mean Square Error (MMSE) Estimate (Jazwinkki (1970))). Let & be estimate of random variable z and LMSE == (x - (x- The estimate that minimizes E[LME is called the minimum mean square error estimate.
Theorem (Theorem 5.3 in Jazwinski (1970). Conditioned on the observation history Yk, the minimum mean square error estimate of state 2t is the conditional mean [Exx\(Y))
Proof. Proof can be found in our appendix A.1
The Bayesian filtering consists of recursive prediction and update procedures (Jazwinski 1970))
Prediction Step p(rx-11 1 - p( 11-1): Given the posterior distribution p( V 1) at instant & I, the prior distribution p( I V 11) of z satisfies the
equation:
p Y2 12-1) /
p(z ( - dra-1:
(3)
Given prior distribution p( (k-)), when the observation yb at instant 1: arrives, the posterior distribution p(ax (Yx) at instant & is given by equation
Updating Step p(a* (k-1) + p( Y&):
p(n las (p(.r//(=---.....
p(xx|Ve)= =
(4)
pluk 2E (z Y dr
Then we can get the MMSE estimate by simply doing integration:
E E[zx [Ya]
p( (Y dra
(5)
To facilitate subsequent discussion, make the following definition.
Definition 2 (Sufficient Statistic (Bene3) (1981)). If the conditional distribution p(xx(VE)) (or p(zy|Yx-1)) can be fally determined by vector-valued function k (or S1(k-1) of the obser- vation sequence Y& (or Yk-1), then we say Sype (or su(k-1) is a sufficient statistic for p( Y&) (or p(zx|Yr-1)
Because sufficient statistic Saja fully determines the posterior distribution p(. Yx) and MMSE estimate (Vx) is functional of p(ax Vy),, there exists function y that maps 8k1s F[ Y i.c.,
B|rx|Ya] = (sk()))
(6)
3 RNN BASED FILTER'S ' ARCHITECTURE
Motivated by the Bayesian framework filtering, we propose the RNN based filter's architecture as shown in Fig.
Our RNN based filter's ' architecture, Bayesian Filter Net (BFN), consists three parts: prediction network, update network and estimation network. To mimic the prediction step in Bayesian filtering, we use the prediction network to map the posterior distribution representation vector to a prior distribution representation vector. To mimic the update step Bayesian filtering, we then use an update network to update the prior state distribution representation vector and the observation to get the posterior state distribution representation vector. Finally, we use an estimation network map the current posterior state distribution representation vector to current estimation 2g. We will see in the subsequent discussions, the callel representation vector hidden variables indeed can be interpreted as statistics of the underlying conditional distribution.
The sufficient statistic Sajk (or 5kjk-1) can be any vector-valued valued function of Yx (or Yx-1) as long as it can fully determine p(zx|Yr) [&)) (or p(rx|Yx-11) Taking linear system as an example, $ajk can be the vector composed of the conditional mean and covariance because these two quantities sufficiently determine the conditional density function of the state linear filtering system. (See more details in our Appendix AA.3
For instance, as for the Kalman Filter in linear system, the sufficient statistic skjk can be chosen to be the conditional mean and conditional covariance, thus function can he the function that copy the conditional mean part of sajk-
3

---

Under review as a conference paper at ICLR 2020
Prediction Neural Network
Prior State
Posterlor State Representation
Representation
Estimation Neural Network
Noisy Observations
Estimation of Current State
Update Neural Network
Figure 1: RNN based filter's architecture: Bayesian Filter Net (BFN).
RNN BASED FILTER IS UNIVERSAL
We now show that proposed RNN based filter is universal in that it can approximate large class of optimal finite dimensional filters to any asymptotic accuracy we desire. We summarize our insight into the diagram
pla (
P (x,, 1g
p(.. (
ye updatess the distribuzion.
Prodict the next step distribution.
Tinite
Finite Dimensional
Finite
Dimensional
Dimensional
Posterior
Prior
Statistics
Statistics
Statistics
Statistics Update Mapping sp. Update networks..
Statistics Prediction Mapping
Prediction networkó.
Combine.
Statisties Moment Estimation network ft
Mappingy.
Combiiee.
Figure 2: Our approach's illustrative diagram
As shown in the diagram we model the conditional prior probability p(rx|Ve-1) (posterior probability p(æz| Yx) resp.) of the state step & by finite dimensional prior statistics Stjk- (posterior statistics $k/k resp.). finite dimensional filter case (Beneš) (198I 1985); ); Daum (1987) ) there exist finite dimensional statistics that are sufficient, that is to say, the evolution of conditional probability can be fully captured by the evolution of a finite dimensional vector. We denote the evolution function in updating step by ip and the evolution function in prediction step by And further, after modelling the the probability distribution as finite dimensional statistics, we use two neural networks approximate the evolution of them. And get the final estimate of the state, we use another neural network to approximate the map from the statistics to the optimal estimation.
For the RNN based filter, one can naturally ask:
1. Will the neural network approximation error accumulate and blow up when time goes to infinity and make RNN asymptotically not work at all?
2. How general is this approach? What filter RNN approximate?
We give the answers in this section.
4

---

Under review as a conference paper at ICLR 2020
4.1 KALMAN FILTER (KF) CAN SYNTHESIZED BY RNN
When system equation D is linear and satisfies Gaussian noise assumption as shown in equation] 7 is well known that the filtering problem can be optimally solved by Kalman Filter (KF) Kalman (960)). (See more details in our Appendix A.33)
2 Fx + GwR-1
1/ Hrr Uk.
where F. G. are constant matrices with proper dimensions, the initial state 2o is Gaussian, and {wx, k 0.1..... } and (vg, = }.. are two independent white Gaussian sequences that are also independent of the initial state zp-
It can be known that the prior and posterior distributions are Gaussian and fully characterized by the sufficient statistics s composed of mean and covariance matrix. (See more details our Appendix
A.3
8k/k-1 == [m vec" (PAR-1))",
(8)
Skilk := [ma]e vee" veceeeecP" (PR{a)))"
where majk-1 and PEIK-1 are the conditional mean and covariance in step k conditioned on YE-1 and majk and Pajk are the mean and covariance in step l conditioned on Ys, and vec("n, xny) vec()n)xxg) ) is the n2 1 column vector obtained by stacking the columns of the matrix o on top of one another. Skjk (8kjk-1 resp.) is the theoretical statistic that determines the conditional probability distribution of the state 2ę conditioned on the observation history Y2 (Y resp.) and evolves according to some function and @. (See more details in our Appendix A.3 We also know that there exists some
function y that maps Sk/a to MMSE estimate 22 Yx). Thus we have,
yp 1: 8k+1|A ((8RR) E[ YR) ($8jk))
(9)
We use function generated by deep (feedforward) neural network (i.e. update network) to approximate sp, and use / generated by another DNN (i.e. prediction network) to approximate And the numerical statistics computed by RNN are denoted as 8kjk and 5k+1/k, i.c.
k/ (5 yk), kk+1|k = Ō(5kks)
(10)
And we use function 7 generated by the third deep feedforward neural network (i.e. estimation network) to approximate y in equation [ i.e.,
szjk = (5k(k))
(11)
We consider the probability space (2, $ P) with inner product (x, v) = =E[=" y] and norm | ==
[1/2 z:), which is a Hilbert space and denoted as (22 F IP). We first state the universal approximation theorem of feedforward neural network before we proceed show our results.
Theorem 2 (Universal Approximation Theorem (Hornik et al. (1989))). For any given compact subset K C. R", any given continuous function f defined on K and any given accuracy degree there exists function 9 represented by a finll-hhddeen-lae neural network with non-constant and bounded activation function such that max fx) g(x)) <c
Proof. It is natural corollary of the Thm. 2.1 in Hornik al. (1989).
Define Egja := ||an 3aja Il. which represents the cumulative error caused by the approximation error of á and G.. Similarly, we define cajk-1 ||a S shall give the condition which ensures the cumulative error will not blow up as time k approaches co.
ill In the following theorem,
Before we proceed show our main result, we first establish a key lemma, Lem. 2 We make two
assumptions on the system we'll consider..
Assumption 1. We assume that the linear dynamic system of the state in equation 7 is stable mean square sense (Samuels ( 1959)), i.e.,
lim 400
| S M,
(12)
where M is a finite constant.
5

---

Under review as a conference paper at ICLR 2020
Assumption 2. The dynamical system equation 7 is uniformly completely observable and uniformly completely controllable.
The definitions of uniformly completely observable and uniformly completely controllable can be found in section 7.5 of Dazwinski 1970p. We then state a lemma on the boundedness of conditional covariance.
Lemma (Lemmaa 7.1 in Jazwinski 1970 If Assumptiio 2
satisfied and Pojo then PatAe is
uniformly bounded from above for all k 2 N,
1 [1
0. 1, 1,
PaIK
(13)
where is positive integer, I is the n X n identity matrix and a, are positive constants.
Based on Lemma [ Assumption I and Assumption 2 we give the key Lemma 2
Lemma 2. In the discrete linear system equation? 7 suppose the Assumption ! and Assumption Pare satisfied, then for any given e > there exists a compact subset K C [djimssaje) such that the statistics computed by KF Sklk-1, Skik and the statistics Skjk-1, Sajk computed by RNN based filter with non- constant and bounded activation function satisfy ||52//=1 beje-_gK] 6, ||staa I B < 6,
< e and | ix < 6, where A an indicator function.
Proof. Proof can be found in our appendix A.4
We then derive our main result.
Theorem 3. Assume Skik, 0 are the theoretical statistics evolving according to equation
and Sklle k O are the real statistics computed by our RNN-based filter evolving according to
equation 10 Suppose the Assumption Ianattiionnnat [ad Assumption 2 are satisfie. Furthermore, we need to
assume functions y, o. 7 are Lipschitz, for any $1, 82.
||ş((11. y) (p((8,))) I Cpplss $211.
[+(())) d(x2)] S C [8| 8211:
(14)
(h(ss) -s>2)l I &C,lss - -zll.
where Cop and Co are Lipschitz constants. IfC, and Co satisfy 1CypCol < 1, then for any > 0, there exists an RNN based filter (with non-constant and bounded activation function) such that
lim sup eajk = lim sup ||a
SkjØ|l <e
(15)
k0000
Furthermore, we have
lim sup |aaA E[Zx|Y2]] [Ya] <e
(16)
Proof. For any 8 > O, we have the following. By Lem. there exists a compact ball K = B(0, r) C
R dimu(su))) such that || --- < 8, || ||aR I saje gx < 8, || - < 8 and
|R ā [ || 8. By Theorem 2 given any small &,, dg RR+ and 8, RR,, there exist two
functions Φ. ō which are represented by the DNN, such that
1|p - - < 8,, |0 5
K < 8,, ll - 71 < 6,,
(17)
where |b]| 0g = max [b(z)]. And without loss of generality, we set x(0) 50), (0)) ((0) and (0) = (0). In the prediction step, based on the evolution equations equation 9 and equation 10
we have
||(sx|=-1 $aju )l = || [| 1|--) - 0( 1)ll
CkJR-1
- - + (1 ō6 ( 1|--
< |0(8 [0(*-111--1) - 06 (- + ( 1 - /i (5 118 11k-1)) sx_ (18)
+ 11(0($x-1)k-1) - 06 - -
$ CCC2 0g + (Co + 0,)
Heer, X> Y Y resp.) if and only if X - Y (Y X resp.) is positive semi-definite, where X and Y are symmetric matrices.

---

Under review as a conference paper at ICLR 2020
where the last inequality follows from equation 14 and equation 17 and C3 is the Lipschitz constant
of Φ. We let 6g := 6x + (C& + C3)ó.
Similarly, in the updating step, we have EMjR I Cpexje-1 + 66 ,here 60 == := -- (Cpp + C2)
Combining this and equation 18 we obtain
EbIk S CGENRK-I+ 6 S (C,Co)) (Cro's + +6z)
(19)
Using equation 19 repeatedly, it follows that !
enjk - (C,Co)* eojo + ( (C,Co)+ -1
(00)
C,Co-1
Thus limsup eajk t-+000 once the condition [CpColl < 1 holds. We choose small enough 6q dgg and such that 1
< (C& 06 + (Co + C2)d) + + + (Gp + Cz))
- Co as h 8
(Cp 66 + (Co C2))0 + 6p + (CGp + +Gp))) C.C
Then get 15 Now we prove
16
(aa - R[xx || = ((((k) - ((WWK)] $ 17((1)))) - (SMW))] + (1((()))) - (5 < (190)) - (5x(()) ) + ||||||+1111111 (7((k)))) - (3x(k) -an- + (1((()))))
- (5x(k)) L -
< ECGEER + t., + (C, + C2) ~ C2 (C) eojo + Cr (Cip's + 6%) (CoCo)*
C,Cs -1 + 5y + +C, + C4]8
(21)
where the inequality *1 follows equation 14 and equation 17 the inequality *2 follows equa-
tion 59 and Cg. is the Lipschitz constant of 7. Thus sup [xjR
R[<x/V<]]] s 5Cpp
k-++00
5 C. Co + +1) + + (C,, + Cz)5. Again we can choose small enough 6g. 6z. 6, and 5 such that (C,5& +
+ +1) + 8, + (C, + C4)8 < e. Then we obtain the desired 16
An example satisfying all the assumptions of Thm. 3 can be found in our Appendix A.5 and general can be found in Appendix A.6 We also remark that in our proof. we implicitly require that the Lipschitz constants of Ø, ē, f are uniformly upper bounded. (See more details in our Appendix (..)) Thm. highlights that the optimal filter in linear system with Gaussian noise, Kalman Filter, can be synthesized by RNN. And RNN based filter's s asymptotic error can be as small as wanted under some Lipschitz conditions. That is to say, RNN is an approximator of Kalman filter.
class of systems satisfying assumption [cl <
4.2 RNN BASED FILTER IS A UNIVERSAL APPROXIMATOR OF OPTIMAL FINITE DIMENSIONAL FILTER
Thm. B shows that Kalman Filter (KF) can be synthesized by RNN. In this section, we'll Ill try to answer the question "How general is the RNN based filter?" and extend the result into a more general case. We'll show that any optimal finite dimensional filter can be universally approximated by RNN under some mild conditions. For a general system with noisy observation as shown in equation I once the conditional distribution p(# (Yx)) is obtained, the filtering problem is solved. However,
usually need to solve an infinite number of ordinary differential equations (ODE) in order to solve p(æ;) Y2). If the distribution p (Y)) admits a finite dimensional sufficient statistics, then we only need to solve a finite number of ODE (Chen 2003)) and we call such filter finite dimensional filter. Finite dimensional filter has been an active research area after the seminal work (Bene3) [1981 1985) of Beneš. It's is large class of filters. Some nontrivial finite dimensional nonlinear filter examples can
be found in Daum (986); Ferrante & Runggaldier 1990); GÜnther 1981); Levine & Marino 1986).
Similarly, we use vector Syj to denote the finite dimensional sufficient statistics of the posterior distribution p(æ)) Yx) and Syjk-1 to denote the finite dimensional sufficient statistics of the prior
See more details in Appendix A.7
7

---

Under review as a conference paper at ICLR 2020
distribution p(r -1) The evolution functions of the statistics are denoted as & and v, and the map from Sajk to conditional mean [x Y&) is denoted as I, i..,
Suik-1 - (Sx=11)-11,
Saka= ((SH)K-1,Bk) (xxx(())) = ((S2(a)
(22)
Similarly, in our proposed neural networks, we use DNN generated function & (prediction network) approximate Φ use another DNN generated V (update network) to approximate V, and the third DNN generated function I' to approximate Г. And the numerical statistics computed by RNN denoted SajK and i...
SaJR-1 = ((E--1 Sajk = (S U.), ZxIE [(Sup))
(23)
We also need the following assumption. Assumption 3. We assume that for any given c there exists a compact subset K Rdim((uu)) such that the statistics computed by the optiaml finite dimensional filter SaIk-1 and Suje: satisfy
- EK < and [Sacks, u e.
We can see equation 7 is the special case of system equation 1 and it satisfies the Assumption according to Lemma We further have Lem. 3 and Thm. A
Lemma 3. In the discrete system equation
7 for any given O there exists a compact subset
K Rddm(Sxv) such that the statistics Sajk- Skys computed by RNN based filter with non-constant
and bounded activation function satisfy SMjk-1 S 1 is indicator function.
< c and |kan S
< G, where
Proof. The proof is similar to the step 2 of the proof of Lem.
Theorem 4. Consider a discrete filtering system equation with optimal finite dimensional filter and
suppose Sulkr k are the theoretical statistics evolving according to equation 22 and Sujk k are the statistics generated by our RNN based filter and evolving according to equation 23 Suppose the Assumption Bis satisfied. Furthermore, we need to assume that functions & and are Lipschitz, i.e., for any Si, 52.
(V1, v) VD(2,, v)1 < Cw S21l.
[[ ((S2) I | S2l1,
(24)
[[T(S))- - r( I CCrl[S] - S2ll
where Cy and Ca are Lipschitz constants. If Cy and Cs satisfy (Cy < 1, then for any >> 0, there exists RNN based filter with non-constant and bounded activation function such that
lim sup |Suik - -S11 <E k-+00
(25)
Furthermore, we have
lim sup |aaj E[ Y4] <c
(26)
Proof. The procedure is similar the proof of Theorem
Thm. [highlights that RNN based filter can not only approximate Kalman filter, but any optimal finite dimensional filter under some Lipschitz conditions. Therefore, RNN's expressive power is characterized as the universal filtering property.
5 CONCLUSION
In our paper, we try to characterize the expressive power of RNN from the filtering perspective. We unify the recurrent neural network into Bayesian filtering framework and show that recurrent neural network is a universal approximator of optimal finite dimensional filters under some Lipschitz conditions. As an interesting special case, the widely used Kalman filter can be synthesized by RNN. Understanding the expressive power of RNN based filter in more general nonlinear filtering cases (with no finite dimensional sufficient statistics) can be a very interesting future direction.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. arXiv preprint arXiv:/81 1.05381, 2018.
V. E. Beneš. Exact inite-dimensional filters for certain diffusions with nonlinear drift. Stochastics-an International Journal of Probability & Stochastic Processes, 5(1-2):65 92, 1981.
V. E. Beneš. New exact nonlinear filters with large lie algebras. Systems & Control Letters, 5(4): 217-221, 1985.
Zhe Chen. Bayesian filtering: from Kalman filters to particle filters, and beyond. Statistic:: A Journal of Theoretical and Applied Statistics, 182(1):1-69, 2003.
Daum. Solution of the zakai equation by separation of variables. IEEE Transactions on Automatic Control, 32(10):941
1987.
Frederick Daum. Exact finite dimeessoins nonlinear filters. IEEE Transactions on Automatic Control, 31(7):616-62) 1986.
Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1110-1118 118, 2015.
Juntao Fei and Cheng Lu. Adaptive sliding mode control of dynamic systems using double loop recurrent neural network structure. IEEE Transactions on Neural Networks and Learning Systems, 29(4):127511286, 2017.
Marco Ferrante and Wolfgang Runggaldier. On necessary conditions for the existence of finite- dimensional filters in discrete time. Systems & control letters, 14(1):63-69, 1990.
Alex Graves, Aldel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks, In 2013 IEEE international conference acoustics, speech and signal processing, pp. 6645-6649. IEEE, 2013.
Sawitzki GÜnther. Finite dimensional filter systems in discrete time. Stochastics: An International Journal of Probability and Stochastic Processes, 5(1-2):107-114, 1981.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366 -365, 1989.
Andrew H Jazwinski. Stochastic processes and filtering theory. Academic Press, New York and London, 1970.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of basic Engineering. 82(1):35-45, 1960.
J Levine and R Marino. Nonlinear system immersion, observers and finite-dimensional filters. Systems & Control Letters. 7(2):133-442 1986.
Bryan Lim, Stefan Zohren, and Stephen Roberts. Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction. arXiv preprint urXiv:1901.08096, 2019.
J Ting-Ho Ho Lo. Synthetic approach to optimal filtering. IEEE Transactions on Neural Networks, 5(5): 803-811, 1994.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems, pp. 6231-6239, 2017.
Xue Luo and Stephen S-T Yau. Hermite spectral method to 1-d forward kolmogorov equation and its application to nonlinear filtering problems. IEEE Transactions on Automatic Control, 58(10): 2495-2507. 2013.

---

Under review as a conference paper at ICLR 2020
Xiaolei Ma, Zhimin Tao, Yinhai Wang. Haiyang Yu, and Yunpeng Wang. Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Transportation Research Part C: Emerging Technologies, 54:187-197, 2015.
Tomáš Mikolov, Martin Karafíát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010.
Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-basis-function networks Neural computation, 3(2):246 257, 1991.
Alexander Parlos, Sunil Menon, and A Atiya. An algorithmic approach to adaptive state filtering using recurrent neural networks. IEEE Transactions on Neural Networks, 12()):41111442, 2001.
J Samuels. On the mean square stability of random linear systems. IRE Transactions on Circuit Theory, 6(5):248-2) 259. 1959.
Anton Maximilian Schäfer and Hans Georg Zimmermann. Recurrent neural networks are universal approximators. In International Conference on Artificial Neural Networks, pp. 632-640.. Springer, 2006.
Robert Wilson and Leif Finkel. neural implementation of the kalman filter. Advances in neural information processing systems, pp. 2062-2070, 2009.
A APPENDIX
A.1 PROOF OF THEOREM 1
Proof. It is known that
LSSEl
=E( fxx²
(x))a(ra (Ix xx,)l
(27)
=E [ [T xxx) (xx - xx)l Yx]]
and let
14 = E[mxs|Val.
Since
E (IT )m)) T =xx) Yk
(28)
(In m (me m16)) 0,
and
E [#x=-w) 1 mmx)
(29)
independent of it. equation
27
becomes
E[LMSE]
E (xx kx) ( (Ix -xx)l Yx
E (xx p4 + th - 2xx (xx
144 + p4 +xx) Ykl
E E [œ ) S "ax -4k) + (#k - xx0 (( .xx)
(30)
+2(x1 pa)² T (mu - (ax)) Yx]]
E [ark - )*x)T (æx - [1))] + [ - 2xx 3T kxa)
=const + E [(M 3x (Nx - xx)1
Therefore [[[.MS]] clearly minimized by setting fs te EE(xAA Vi).
10

---

Under review as a conference paper at ICLR 2020
A.2 THE EVOLUTION DISTRIBUTION IN UPDATING STEP OF BAYESIAN FILTER
Let p (b) denote the conditional distribution of state 2y conditioned on the observation history Ys. then using Bayes' rule, we have:
p(xx| Bayes' nole p(Yx(TR))((TH) Pluk, 1 (Ya))
plun Yk-1)
p(ux R- 1. 3s (Vp))(Vxi11 (
p(un Yk (k-1)p((Kk)
Bayes' nole p(un Y k- Dbp (2 x 1 (
(31)
plva R 1
plu (tumlxx)p(e) \k-1)
P(uA (Y
plu mo (x x
pluu ) 2h Yx-1 dr
It can be seen that the posterior distribution p(xx is determined by the prior distribution (xk)) and likelihood function p(u,(=x), which depends on the observation model and the known distribution of Uk-
A.3 THE EVOLUTION FUNCTION STATISTICS FOR KALMAN FILTER
The widely used Kalman Filter assumes that the posterior distribution every instant is Gaussian, and then it can be parameterized by the mean and covariance. We consider the following system:
{ Tk = Fx + Gwk-1
(32)
1/x H Uks
where F,G. H constant matrices with proper dimensions, the initial state 2o is Gaussian, and {wk, k = 1... } and {un, k = ... are two independent white Gaussian sequences that are also independent of the initial state zo jointly.
Then KF can be viewed as the following recursive relationship:
P(Ex-11YE-1) = 11 PA-
(x = -N(x;; my k- PAjK-1)
(33)
P(zx/Yk)) = NM(xi k. P.).) PRI.)
where in the prediction step,
mkck-1 = Fmm--1R-1
(34)
PAJK-1 GQG F -P--IIR-IPT
in the updating step.
majk MM(R-1 Paja- HH H (HPM)-- H + R)-1
(y Hmmqu-1)
(35)
Pajk PRIK-1 PRIK-1 HH' (HPM)--IH* +R)
-H
and (( m, P) is a Gaussian distribution with mean m, and covariance P. The evolution functions of mean and covariance the prediction step and updating step can be expressed in the following compact form:
mp]k-1 = 21 Pk- 76-11k-1)
(36)
PaIK-1 = -22(m=-1kk-1. Pa 1x-11k-1);
11

---

Under review as a conference paper at ICLR 2020
majk (1(m// PaIE- ykk)
(37)
PRIE 92(ma(k-1) P
where Φi, d2, 401 and fP2 are defined as
11 (m, P) Fm
(38)
@2 P) = GQG + +FFFT
and
(i p.y) = : PH (HPH R) 11)-((10 PHPP
Hm)
(39)
@2(mm, P.y) = P PHPP + R) -HP,
Here,
m (R-I),, maja = [aa]
(40)
are the conditional means of state conditioned on observation history.
PRIK-1 = E [ - majk mxjk-1) mAja-1 [Vx-1]
(41)
Paje = E [(ar= mM) (
are the corresponding conditional covariance. Then we know that the posterior distribution p(ax (&s) parameterized by the conditional mean maps and covariance P& which are functions of Yu. i.e., p(ax|Y/--) parameterized by (ma(k. } And the similar conclusion can be obtained for prior
distribution p(z (Yx-)).
Now we vectorize {majks Pajk as the statistics
Skjk [majk vee" (PNR)]]²
(42)
Vectorizing the second equation in equation 36 and equation 37 we have
(m, P.y) vec(p22(m, P.y))
&i (m, vee (eecc((ec(P))) v) vec((p2(m, vee vec (vec( P)), v))
(43)
((m, vec( ().))) (ps. v),
and
&1 (m, P) (ez(02(m, P))
A d6m, vec(P)) ((s),
(44)
where we omit subscript characters without causing confusion, and vec xn, is the inverse operator of vec such that vec Xng (vec( (nxxn)))) = on, xnz- Then we obtain the evolution functions of the statisties:
Skik = (( yk), 8841/K = ((8I))),
(45)
where yp and @ are defined in equation 43 and equation
A.4 PROOF TO LEM.
Proof. Step 1: It can be known from the KF that the evolution function for conditional covariance is as follows (see appendix):
P = =GQG" + + 11k-11k-11 -PP
PRJK Pajk-1 PAIK-1 HT (H + R)-1
(46)
-H PR
and the initial value is Poo- It follows that the conditional covariance P evolves in a deterministic manner and without any randomness according to the evolution equation equation 46 therefore P is independent of the observations. Since
Pajk = E (IT mpk (Ixx - mmkk) Yo
12

---

Under review as a conference paper at ICLR 2020
and PajA is independent of Yg. have
Paja = E ( ma(k) (xk mals) )
(47)
According to Lemma have
1 1+ ag
Palk $
(
I. 1
(48)
where N is positive integer and a, B are positive constants. It can be easily checked that, there exists positive constant agg such that
PAJE + anl, 2 0.
(49)
According to equation 47
we have
=E (Ex mark) (Ix mEts)
(50)
=tr (PH)k)
where tr(+) denotes the trace of matrix *. Combining equation 49 and equation 50
we know that
|| ma/ell S /nao, A
(51)
Then according equation
T2 and equation we have
| |ll S | maja + ||
S v /aa + M
(52)
Co, A k: 2 0.
where Co a positive constant and can be chosen to be /naxo + M.
Now we have the conclusion that m&jk and PEIE are bounded for all k 2 0. Similarly, know that malk and Paja-1 are bounded following the similar procedure as above. According to equation 8 we know that all sufficient statistics 8k]k-1, Skjk are bounded.
Step 2: We then prove that | 1 I and || 1 sMAEK|| Recall that
Fkik = (6 Uk) 1 = (((A)k),
(53)
where & and Ø are approximating functions generated by neural networks with non-constant and bounded activation functions. Let denote the activation functions used by the neural network and assume that (xx) < C,, Vo E RR. And without loss of generality, we assume (( Uk. = (io(WiP1 (5 yk) + b1) + c1, where P1 is the vector valued function that takes input to the last hidden layer's ' output, Vi and W1 matrices with suitable dimension, b vector with suitable dimension. Then we'll have,
(Wio(WiI((BE + bi) + +a| || [W b1)l| ||ll|
(54)
||V (dim(b,)C, ||
where da)(x) is the dimension of the vector æ. We choose the compact set K cover the compact ball fo E ddim(fe))) ||~|| < |[Vill dim(b,)CC + ||c,]]) where ||+|| denotes Euclidean norm of +. Then it can seen that [ujk 1 x < e. And k < can be shown in similar way.
13

---

Under review as a conference paper at ICLR 2020
A.5 AN EXAMPLE THAT SATISFIES ALL THE ASSUMPTIONS OF THM.
Here we verify that the system shown in equation 55 with state's dimension satisfies all assumptions in Theorem 3
25 = (1 - --aaa-111 + vawi-1
(55)
= our + vauk:
where a < 1 small positive parameter.
As for Assumption
1 from the state equation of equation 55 we have
=(1 0-) + Vowk-1
=(1 a)) (1 a) Vour-2 + Vawk-1
(56)
= a) xo
xo
(1 - a) --1 Vaw,
==0
then we have
k-1
E a [] =(1 ai)
2 B 1 1 + Σ (1 a) 22222 22
1=0
(57)
a
2k-2 - (1 ax) aQ
=(1 a.
2
+
(1 a)²
It can be easily checked that Assumption is satisfied.
As for Assumption 2 it can be easily checked that system shown in equation 55 satifies Assumption 2 using the definitions of uniformly completely observable and uniformly completely controllable in section 7.5 of Jawwinskk (1970).
A.6 A CLASS OF SYSTEM THAT SATISFIES |CpOol < 1 IN THEOREM
Actually this condition CCpCol < 1 can be satisfied by general class of systems. Let us consider the 1-dimensional linear systems with discretization in continuous observation equation, i.e.,
Fte a3 + fw St. bâtot, o
(58)
where a,g, b are constants, At 44 - f--11 << the step size sampling, {wk, k 1, ... is an 1-dimensional white Gaussian process and wP (N10, Q). and (u is 1 -dimensionnll white Gaussian process with distribution. (o0 R/ At). According to Jazwinski 19770, we know that the discrete observation system in equation 58 is the sampled system of continuous observations.
Following equation (34)-(35) in Appendix A.2, we have
d (m, P) = am, (m, P) = 9Qg + a²P,
and
40 (m, P) ((
K(y batm), P2 (m, P) P KTAIP,
where
PAt R) can be easily checked that
a
11-Kbtt
(y - Atm) 8K
Vo =
2 V
UP Kbbt &AtPaK JP
O
where
aK 0P (bAt) + R) -2 Then have
Co = max a² )²), C2 = V 1| where || || the Frobenius norm.
If a < 1, which is a very natural stable condition, then
lim A1-50
CCpCCol < lim (a O(At)) = =a =a 1. Ar-0
Apparently, we have CGpCcl < with small step size At.
14

---

Under review as a conference paper at ICLR 2020
A.7 CALCULATION DETAILS OF INEQUALITY 20
CNjR - (C-Co) + (c
< (C,Co)) (CpCpes-21k-2 (Coos + +b²)) + (CoOs + +0z)
+ CC,Cs + 1| (Cpo's + +6))
(59)
< (C,C,)* eujo + (605s05 Σ (c,Co)*
(CoCo)* -1
CC) Eojo + (CrO' 66) CgCo
A.8 APPROXIMATE 6 AND Yp USING LIPSCHITZ NEURAL NETWORK
In the proof of Thm.
implicitly require the Lipschitz constants of e and & are uniformly bounded by another constant C. Note that this assumption won't affect the approximation capability of Deep Neural Network in approximating Lipschitz continuous function e and (Thm. 3 in Anil et al. (2018). With this requirement, the internal state variables of our proposed RNN based filter always inside compact ball independent of the choice of neural
network. And we will have (C& (66- + (Cs + C2,, + +6p + (CGp + C2)6) 1 C.C
1
(Cip (6x (Co + +C))++ 6 (C + C)8) T- C,Csss
thus we can choose small enough
5, de and 6pp to make (ce (8 + (Co + Cz)6) + d, + (Cp + Cg)o) C.C
1
(Cpp (ōx + (Co + +)0) + (Cp + C)
1 Cs <e.
A.9 EXPERIMENTS
A.9.1 EXPERIMENT SET UP
We use PyTorch as the deep learning framework to implement our algorithm and run our simulation on CPU clusters with 5 (ES-2630//3/2440GHz CPUs, each is equipped with 30GB memory. We use the Mean Squared Error(MSE)) as the metric to measure the performance of the filter. For subsequent discussion, make the following definition.
Definition 3 (PMSE). For a set of sampled paths fui,w2,". ... .wN,,, }. we define the Path Mean Squared Error at time slot i PMSE& to be n E [[: - I (w:))1
Further we define the Temporal Path Mean Squared Error TPMSE.
Definition 4 (TPMSE). For set of sampled paths (wi, w2, wNn } and horizon of simulation 1: M, define the Temporal Path Mean Squared Error (TPMSE) to be 1 ,EM, PMSE.
A.9.2 TRAINING
Alg shows how we train the RNN based filter.
A. A.9.3 KALMAN FILTER CAN BE SYNTHESIZED BY RNN BASED FILTER EFFICIENTLY
We consider a stable linear filtering system as in equation 60
Ik (aAn + In In) + Vawk-1
(60)
yk = az + Vauk,
15

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Bayesian Filter Net (BFN) training algorithm
1: Setting the sampling path number N, the number of simulated steps M, the optimization horizon length and the learning rate
2: Sampling (ws) i.i.d from the initial distribution (n(r) and set yw()) 0
3: for 0 S m I M -1 do
4: Initialize loss () be
5: for ml S k < (m + 1)/ do
6: for 1 15 i S N do
((xx( + (( 8: yk+1 (wr) - (w;)) + (wi)
7: 2k+1 (w:)
9:
end for
10:
for
S N N do
Feed the sampled (wx) to the Bayesian-Filter-No(FFT and get the corresponding output (w:1(w;)
Add the estimation squared error the loss function loss(0)
loss(0)
( xx+1(w;)l1
13:
end for
14:
end for
15:
Doing back propagation and update the network parameters 0 <- 0 A VVlloss(O)
16: end for
where < 1 is a small positive parameter controlling the changing rate of the system state, In n x n identity matrix and A, matrix with elements azy satisfying ai i, ( jf = n and 0, otherwise.
RKN 30
RMN
10
KF
RKN 20
15000-
RKN 70
RKN 50
Cineergd TIRSE
RKN 50
32000
1030
30
Figure 3: (a) The decrease of TPMSE of RNN based filter with different hidden variable dimensions training. (b) Convergent TPMSE of RNN based filter with different dimensions of hidden variables.
From Fig. 3a we can see that as the number of training epochs increase, RNN-based filter's MSE decrease and finally converge. And from Fig. 33 the dimension of the hidden variable increases, the converged MSE decreases. The results in Fig. highhlghh that just with the number of 90 hidden variables, RNN based filter can achieve the performance Kalman filter. Considering that there are 65 independent statistics in this case, this is a very efficient approximation.
A.9.4 RNN BASED FILTER CAN STILL WORK FOR INFINITE DIMENSIONAL FILTER
For finite dimensional filter, our analysis has shown that RNN based filter is universal approximator of optimal filter. In this subsection, we show that in infinite dimensional filter case (where finite dimensional statistics are not admitted to fully determine the distribution), our RNN based method
achieves even better performance to -ate-off-the-at spectral method Luo Yau 2013). We consider special and well-known infinite dimensional case.
Tk- + H Vowk-1
(61)
y/
axi + Var,
16

---

Under review as a conference paper at ICLR 2020
where is positive parameter controlling the change rate of the system. We set the to be 0.01, the total sampling steps to be 5000, the number of sampled paths to be 1000, the number of training epochs to be 100. We only use two fully connected layers for the prediction network, update network and estimation network. Fig. showswws the convergence of PMSE of our Deep-Filter-Net with respect the time. And fig. shows that our proposed Deep-Filter-Ne Net output Rh tracks 2 well. We also
1.6
1.0
3.-
37
30
0.0
5/0
10.0
125
20.0
tmm
(a)
Figure 4: (a) The evolution of PMSEE. (b) Sy. tracks Ty well.
compare our algorithm's 's performance with existing taae-of-the-aae spectral method. Our proposed DFÑ's TPMSE is 0.18, which much smaller than spectral method's TPMSE 0.40. One potential reason for its RNN's applicability in infinite dimensional filter case is that RNN implicitly learns the most relevant finite dimensional statistics.
17


Reviews:
[{"review_id": "SyghxB-0YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper shows that RNN (of infinite horizon) can be universal approximators  for any stochastic dynamics system. It does that by using the Bayesian framework of filtering, which shows that if sufficient statistics given the observations so far can be computed at each stage of filtering, then the expected hidden states at each stage can be estimated as well. The paper then follows this framework by constructing deep neural network mapping from “observations so-far” to “sufficient statistics” and by universal approximation theorem this is possible. Then as long as after each stage, the image of such a mapping is bounded into a certain region, and if the mapping is contractive (|C_\\psi C_\\phi| < 1), this can be applied to arbitrarily long sequence (and thus infinite horizon). \n\nThe paper is well-written and easy to follow. However, I have concerns about its novelty. Overall the paper seems to be a straightforward application of universal approximation theorem of deep neural network. The authors don’t specify when the condition will hold (e.g., |C_\\psi C_\\phi| < 1), what kind of empirical RNNs fall into this category and whether the trained models are operating in the contractive region. \n\nAlso with the same technique the authors also reach the same conclusion for nonlinear dynamics system (Sec. 4.2). From the proof, there is no much difference between linear case (Kalman Filter) and nonlinear case, since apparently DNN can fit everything. Then what are the new discoveries and takeaway messages? It is not clear what can we learn from this analysis and its impact is likely to be limited. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylewpCeqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows that recurrent neural networks are universal approximators of optimal finite dimensional filters. More specifically, the paper provides a theoretical analysis on the approximation error for any stochastic dynamic system with noisy sequential observations.\n\nI find the work interesting. Yet, I don't understand how it substantially deviates from previous work. The cited work by Schafer&Zimmermann only considers deterministic systems -- but what is the distinguishing idea of your paper that makes your more general analysis possible? To me it seems utilizing sufficient statistics? But working with sufficient statistics resembles lossly again working with a deterministic-like abstraction? While I understand that this a purely theoretical work, it would be instructive to have practical demonstrations, showing what's happening when learning is actually done. Finally, interpreting RNNs as bayesian filters was utilized in recent works, e.g. Recurrent Neural Filters.\n\nAs a side note, isn't eq.3 called the Chapman-Kolmogorov equation. Also, the book by Jazwinski is often cited -- I think if the main findings utilized in your text, e.g. the proof of Thm1, were replicated in the Appendix, your paper would feel more complete.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJlBaEZg5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to establish the asymptotic accuracy of the \"RNN\" (but not the RNN models that are well-known in the literature - see the below comments) as a universal functional approximator. It considers a general state space model and uses feedforward neural nets (FNN) to learn the filtering and forecast distributions. Based on the well-known universal approximation property of FNNs, the paper shows that their RNN-based filter can approximate arbitrarily well the optimal filter. \n\nThe paper targets an important problem in statistical learning and offers some interesting insights. However, their so-called RNN-based filter formulation is not anywhere close to the usual RNN models, such as Elman's basic RNN model or LSTM, that are currently known in the literature. Hence the paper's title \"RNNs are universal filters\" and the way it is presenting are confusing. I think what exactly the paper is about is as follows. They consider a general state space model, then use FNNs to approximate the so-called transition equation and observation equation of that SS model. Then, the resulting RNN-based filter is shown to be able to approximate well the optimal filter of the original SS model. I didn't check the proof carefully but I guess it's intuitively straightforward given the available results from the approximation capacity of FNNs. \n\nTherefore, I suggest the authors re-write the paper carefully to reflect better the problem that it actually targets. The property of the RNN-based filter is interesting, but using it is, I believe, very difficult from a practical point of view. It's well-known that doing particle filters is computationally expensive, especially in high dimensions, and the RNN-based filter might have millions of parameters! Could the authors please give some comments/discussion about this issue?\n\nSome minor points:\n1)  what exactly does \"synthesized\" mean? \n2) Page 5: \"Note that the probability space ... with finite moment...\" doesn't read well. What is the moment of probability space?\n\n\n           ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the Bayesian approach to filtering problem, the paper proves that RNN are universal approximators for the filtering problem.  Two reviewers, however, have doubts about the novelty and difficulty to get the result. Although I do not fully agree that Reviewer3 that the proof is just "DNN can fit anything" - it is not this case, but the concerns of Reviewer2 are more strong, especially about the usage of the term "recurrent neural network". The paper is purely theoretical and does not have any numerical experiments, which probably makes it too weak for ICLR in this form. However, I encourage the authors to continue to work on the subject, since the approach looks very interesting but it still very far from practice.
Decision: Reject