Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
Authors: Carlos E. Luis
Alessandro Giacomo Bottero
Julia Vinogradska
Felix Berkenkamp
Jan Peters
Venue: Accepted by TMLR ()

Abstract: Optimal decision-making under partial observability requires reasoning about the uncertainty of the environment’s hidden state. However, most reinforcement learning architectures handle partial observability with sequence models that have no internal mechanism to incorporate uncertainty in their hidden state representation, such as recurrent neural networks, deterministic state-space models and transformers. Inspired by advances in probabilistic world models for reinforcement learning, we propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models and train it end-to-end within a model-free architecture to maximize returns. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan, which scales logarithmically with the sequence length. By design, Kalman filter layers are a drop-in replacement for other recurrent layers in standard model-free architectures, but importantly they include an explicit mechanism for probabilistic filtering of the latent state representation. Experiments in a wide variety of tasks with partial observability show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.

Published in Transactions on Machine Learning Research (02/2025)
Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
Carlos E. Luis
carlos@@rooot-eammmin...
Bosch Corporate Research
Intelligent Autonomous Systems Group, Technicol University Dormstad!
Alcssandro G. Bottero
alessandrogiaccomm.bteeee botterotttotooooooooo
Bosch Corporate Research
Intelligent Autonomous Systems Group, Technicol University Dormstadt
Julia Vinogradska
julia, haaik..eiinoroddkkuaassccccccc
Bosch Corporate Research
Felix Berkenkamp Bosch Corporate Research
www.bbrrkknkampp@Baaccccccom
Jan Peters
jow.peters@tu-eeee weeters@tuddrrmelddd. de
Intelligent Autonomous Systems Group, Technicol University Dormstadt
German Research Center for AI (DFKT)
Al
Centre for Cognitive Science
Reviewed OpenReview: https: openreviee. pormff id=rfPnsOKdyg
Abstract
Optimal decision- decision-making under partial observability requires reasoning about the uncertainty of the environment's 's hidden state. However, most reinforcement learning architectures handle partial observability with sequence models that have no internal mechanism to incorporate uncertainty their hidden state representation, such as recurrent neural worksrks deterministic state-space models and transformers. Inspired by advances in probabilistic world models for reinforcement learning, we propose standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models and train it end-to-end within model-free architecture to maximize returns. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan, which scales logarithmically with the sequence length. By design, Kalman filter layers are a drop-in replacement. for other recurrent layers standard model-free architectures, but importantly they include an explicit mechanism for probabilistic filtering of the latent state representation. Experiments in wide variety of tasks with partial observability show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision making, outperforming other stateful models.
1 Introduction
The classical reinforcement lcarning (RL) formulation tackles optimal decision-making in fully observable Markov decision process (MDP) (Sutton and Barto, 2018). However, many real-workd problems are partially observable, since we only have access to observations that hide information about the state, e.g., due to noisy measurements. Learning partially obscrvable MDPs (POMDPs) is statistically and computationally intractable in general (Papadimitriou and Tsitsiklis, 1987), but in many practical scenarios is theoretically viable (Liu et al., 2022) and has lead to successful applications in complex domains like robotics (7hu et. al.,

---

Published in Transactions on Machine Learning Research (02/2025)
2017), poker (Brown and Sandholm, 2019), real-time strategy games (Vinyals et al., 2019) and recommendation systems (Ti et al., 2010).
Practical algorithms for RL in POMDPs employ sequence models that encode the history of observations and actions into a latent state representation amenable for policy optimization. Besides extracting task- relevant information from the history, probabilistic inference over the latent state is also crucial under partial observability (Kaelbling et al., 1998). As a motivating example, consider an Al chatbot that gives restaurant recommendations to users. Since the user's taste (i.e., the state) is unknown, the agent must ask questions before ultimately making its recommendation. Reasoning over the latent state uncertainty is crucial to decide whether to continue probing the user or end the interaction with a final recommendation. An optimal agent would gather enough information to recommend a restaurant with a high likelihood of user satisfaction. In Section 5.2, we evaluate performance of our proposed approach in a simplified version of this problem.
A standard recipe for model-free RL in POMDPs is to combinc a sequence model (c.g., LSTM (Hochreiter and Schmidluber, 1997), GRU (Cho et. al., 2014)) with a policy optimizer (e.g., PPO (Schulman et al., 2017) or SAC (Haarnoja et al., 2018)), which has shown strong performance in a wide variety of POMDPs (Ni et al., 2022). More recently, transformers (Vaswani et al., 2017) have also been adopted sequence models in RL showing improved memory capabilities (Ni et al., 2023). However, their inference runtime scales quadratically with the sequence length, which makes them unsuitable for online learning in physical systems (Parisotto and Salakhutdinov, 2020). Instead, recent deterministic state-space models (SSMs) (Gu et al., 2022a; Smith et al., 2023; Gu and Dao, 2023) maintain the constant-time inference of stateful models, while achieving logarithmic runtime during training thanks to efficient parallel scans (Smith et al., 2023). Moreover, SSMs have shown improved long-term memory, in-context learning and generalization in RI. (Lu et al., 2023). Yet, in problems where reasoning over latent state uncertainty is crucial, it remains unclear whether such methods can learn the required probabilistic inference mechanisms for decision making. The core objective of this work is to study the role of explicit probabilistic inference within a model-free RL architecture for POMDPs.
While model-free architectures focus on deterministic sequence models, in model-based RL probabilistic sequence models are a widespread tool to model uncertainty in environment dynamics (Watter et al., 2015; Hafner et al., 2019; 2020; Becker and Neumann, 2022). Considering these two sequence modelling approaches, concretely investigate the following questions:
Can we leverage the same inference methods developed for model-based RL as general-purpose sequence models in model-free architectures? If so, does it bring any benefits compared to deterministic models?
Our core hypothesis is that explicit probabilistic inference in sequence models may serve as an inductive bias to learn in tasks where uncertainty over the latent state is crucial for decision making, as our motivating example on the restaurant recommendation chatbot.
Our Contributions. Inspired by the simple inference schome in the Recurrent Kalman Network (RKN) (Becker et al., 2019) architecture for world models, we embed closed-form Gaussian inference in linear SSMs as a standalone recurrent layer denoted a Kalman filter (KF) layer - and train it end-to-end within a model-free architecture (Ni et al., 2022) to maximize returns. Since our KF layers are designed to be a drop-in replacement for standard recurrent layers, they can also be stacked together and combined with other components (e.g,, residual connections, normalization, etc.) to build more complex sequence models. Similar to Becker et al. (2024), we leverage the associative property of the Kalman filter operations for efficient training of KF layers via parallel scans, which scale logarithmically with the sequence length provided sufficient parallel GPU cores.
We systematically evaluate our research questions across a varicty of POMDPs that probe distinct capabilitics, such as uncertainty reasoning, adaplation, generalization and filtering of noisy observations. We benchmark the performance of KF layers against wide range of baselines from prior work, including GRUs, transformers, and deterministic SSMs, all embedded in the same model-free architecture. To ensure fairness in our comparisons, we meticulously control for confounding factors such as parameter count, training procedure and hyperparameters. By holding all aspects of the architecture and training constant, we isolate the impact of each sequence model in the overall performance, providing clear insights into their relative effectiveness. Through these experiments, we demonstrate that KF layers can be trained effectively end-to-end on model-free
2

---

Published in Transactions on Machine Learning Research (02/2025)
objectives, excelling in tasks where probabilistic inference is key for decision-making and showing significant improvements over deterministic stateful models.
2
Related Work
RL architectures for POMDPs. Partial observability in RL tasks requires agents to maintain memory of past interactions. Some approaches incorporate memory systems inspired by principles of human psychology, such as reward-based learning (Fortunato et al., 2019), or rely on mechanisms like context-dependent retrieval (Oh et al., 2016). A more widespread solution involves sequence models, also referred to as history encoders (Ni et al., 2024), which encode past observations and actions into a state representation useful for RI. These models have been used to augment policies (Wierstra et al., 2007), value functions (Schmidhuber, 1990; Bakker, 2001) and world models (Schmidhuber, 1991; Becker et al., 2019; Shaj et al., 2021a;b; 2023). This enables RL algorithms, such as DQN (Hausknecht and Stone, 2015), SAC (Ni et al., 2022), PPO (Kostrikov, 2018; Ni et al., 2023; Lu et al., 2023), DPG (Heess et al., 2015) and Dyna (Hafner et al., 2020; Beeker and Neumann, 2022) to handle partial observability, In this work, we adopt an off-policy model-free architecture similar to Ni et al. (2022), leveraging its strong performance in various POMDPs.
Sequence models in RL. Frame-stacking was one of the earliest methods used in RL to capture temporal context by concatenating consecutive observations (Lin and Mitchell, 1993). It remains a common tool for conveying velocity information from image-based observations, such in the Atari benchmark (Bellemarc et al., 2013; Mnih et al., 2013). However, frame-stacking fails to model long-range dependencies in more complex POMDPs due to its fixed and shallow representation of temporal relationships. To address this limitation, stateful recurrent models became the dominant approach for extracting relevant information from arbitrarily long contexts. Examples include RNNs (Lin and Mitchell, 1993; Schmidhuber, 1990), LSTMs (Bakker, 2001) and GRUs (Kostrikov, 2018). More recently, the transformer architecture (Vaswani et al., 2017) has shown promise in improving the long-term memory in RL agents (Ni al., 2023). However, while transformers excel at modcling long-range dependencies, their slow inference and large memory footprint reduce their practicality for real-time control tasks, where efficiency is critical (Parisotto and Salakhutdinov, 2020). These challenges emphasize the need for more efficient sequence models that balance representational power with computational feasibility.
Deterministic SSMs. State-space models are of particular interest to the RL community due to their computational efficiency compared traditional sequence models like RNNs and transformers. They maintain the fast inference of RNNs, but scale logarithmically (rather than linearly) with the sequence length during vanishing/expllodng training (Smith et al., 2023). Moreover, they also circumvent vanishing/ /exploding gradients with proper initialization (Gu et al., 2020) and match (or even exceed) the performance of transformers in long-range sequence modelling tasks (Lu et al., 2023). In particular, structured state space models such as S4 (Gu et al., 2022a), $5 (Smith et al., 2023) and S6 / Mamba (Gu and Dao, 2023) have emerged as a strong competilor to transformers in general sequence modelling problems like language (Fu et al., 2023), audio (Goel et al., 2022) and video (Nguyen et al., 2022). The adoption of these models in RL is still in its infancy, however. For instance, Morad et al. (2023) report bad performance of a variant of S4 (Gu et al., 2022b) in various POMDPs, while Lu et al. (2023) show that combining S5 with PPO yields strong results in long-term memory and in-context learning. These mixed findings suggest that the performance of deterministic SSMs in RL is sensitive to implementation details and possibly awvrontteentddpenddee how these models perform in tasks where uncertainty in the latent state is critical for decision makiing,,,iin as they lack explicit probabilistic inference mechanisms. We hypothesize that probabilistic inference is vital to handle such problems.
Furthermore, it remains unclear
Probabilistic SSMs. Probabilistic SSMs are common tool in model-based RL to train both discriminative (Haarnoja et al., 2016) and generative (Ha and Schmidhuber, 2018) models of the environment, often referred to as world models. These models are trained lo capture the enviroument's 's dynamics, which can then be used for: (i) planning (Hafner et al., 2019) or policy optimization (Becker and Neumann, 2022; Hafner et al., 2020) via latent imagination (i.e., generating imaginary policy rollouts auto-regressivelyy) or (ii) policy optimization on the learned latent representation (Becker et al., 2024). A prominent approach is the Recurrent State Space Model (RSSM) proposed by Hafner et al. (2019), which divides the latent state into deterministic and
3

---

Published in Transactions on Machine Learning Research (02/2025)
stochastic components and uses GRU to propagate the deterministic part forward. More recently, GRUs have been replaced by transformers (Chen et al., 2021) and S4 (Samsami et al., 2024) models, albeit in a simplified inference scheme that conditions only on the current observation rather than the history, possibly for computational efficiency. A common objective function for training these probabilistic SSMs is the evidence lower bound (ELBO), which provides a lower bound on the log-likelihood of the environment's 's data. This ensures that generative models produce plausible trajectories given an action sequence, typically optimized with variational autoencoders (Kingma and Welling, 2014). These autoencoders shape a low-dimensional latent representation to capture salient features of the environment's data generation process. The world model objective can be viewed as an auxiliary loss that helps the sequence model learn useful representation for control tasks, which has shown improved sample efficiency in problems with complex, high-dimensional observations like images (Hafner et al., 2023). Other approaches, such as rantvvstive learning (Laskin et. al., 2020), similarly propose a proxy objective that shapes the learned representation to maximize agreement between augmented views of the environment. These auxiliary losses has benefits and drawbacks: they provide a strong learning signal for representation learning, even in the absence of a reward signal, but they introduce complexity in training and can interfere with the RL objective, known as the objective mismatch problem (Lambert et al., 2020). While these tradeoffs warrant research on their own, the purpose of this work is to bring understanding in the role of probabilistic inference in model-free RL architectures without auxiliary objectives.
Kalman filters. A particular class of probabilistic SSMs of relevance to this work are Kalman filters (Kalman, 1960), which perform optimal inference in linear SSMs under a Gaussian noise assumption. Since then, Kalman filters have been theoretically extended to handle non-linear dynamics (Serra, 2018) and also widely adopted in range of science and engineering fields (Auger et al., 2013), including robotics (Urrea and Agramonte, 2021), vision (Chen, 2012), signal processing and sensor fusion (Khaleghi et al., 2013). They also have a rich history within the machine learning community, particularly in early applications for time-series forecasting (Shumway and Stoffer, 1982). While the linear-Gaussian assumption in standard Kalman filtering is restrictive for the high-dimensionall data often found in machine learning applications (Murphy, 2012; Bishop, 2006), several extensions have been proposed. One class of approaches circumvent. these limitations by modelling non-linear state transitions with neural networks (Krishnan et al., 2015; 2017) and performing approximate inference via stochastic gradient variational Bayes (Kingma and Welling, 2014). Alternatively, other approaches simplify the inference problem by embedding (locally) linear-Gaussian SSMs in learned latent spaces (Watter et al., 2015; Karl et al., 2017; Klushyn et al., 2021), enabling exact Kalman filtering and yielding better performance than methods with more complex dynamics but poor approximate inference (Fraccaro et al., 2017). However, the use of full transition and covariance matrices limits the practical dimensionalityy of the latent space and the expressivity of the models. In contrast, we adopt a simpler parameterization of the linear- Gauss-aa SSM, using diagonal matrices and covariances, which significantly reduces the computational burden of Kalman filtering. This approach scales to higher-dimensional latent spaces, enables logarithmic scaling (in the sequence length) of the Kalman filter equations (Sarkka and Garcia-Fernandez, 2021; Becker et al., 2024) and preserves the expressivity of the models by offloading representational power to other components of the architecture, such as encoders and decoders (Haarnoja et. al., 2016; Becker et al., 2019).
Kalman filters in RL. Kalman filters have been extensively used in RL as discriminative (Haarnoja et al., 2016; Becker el al., 2019; Shaj et al., 2021a;b; 2023) or generative (Watter et al., 2015; Becker and Neumann, 2022; Becker et al., 2024) world models. The former are trained on regression losses to obtain accurate predictions, while the latter are trained with variational inference for temporaaly-cconsiseent generation, In the model-free architecture considered in this work, only discriminative sequence models can be integrated without altering the training procedure; generative models would require auxiliary loss functions, which would modify the training process and introduce potential confounding factors that are not part of our experimental design. Closest to our approach is the Recurrent Kalman Network (RKN) (Becker et al., 2019), an encoder-decoder architecture that employs Kalman filtering using locally linear models and structured (non-diagonal) covariance matrices. Follow-up work has extended the RKN framework in various ways: Shaj et al. (2021a) include action conditioning, Shaj et al. (2021b) consider a multi-task setting with hidden task parameters and Shaj et al. (2023) propose a hierarchical, multi- timescale architecture. While these approaches train the latent representation to capture the environment's dynamics, our work instead focuses on training a
4

---

Published in Transactions on Machine Learning Research (02/2025)
similar Kalman filter end-to-end with the RL objective return maximization such that the latent space is shaped specifically for control rather than prediction.
3 Background
In this section, provide the relevant background and introduce core notation used throughout the paper. We use bold upper case letters (A) to denote matrices and calligraphic letters (A') to denote sets. The nolation diag(A) (A) refers lo a vector containing the diagonal elements for a square matrix A and P(X)) refers to the space of probability distributions over X.
3.1 Reinforcement Learning in Partially Observable Markov Decision Processes
We consider an agent that acts in a finite-horizon partially observable Markov decision process (POMDP) with state space S, action space A, obscrvation space O, horizon E N, transition function p:Sx + P(S) that maps states and actions to probability distribution over S, an emission function 0: - P(O) that maps states to a probability distribution over observations, a reward function r: I,, and a discount factor 7 E 1),
== O,
At time step of episode in M,, the agent observes or O--1 st) and selects action a A based on the observed history h. = (o., a:t-1) ) E Hr, then receives a reward r1 r(st, and the next observation O++1 ~ O 8141) with 11+1 ~ p(- 8t, a.).
We adopt the general setting by Ni et al. (2023; 2024), where the RL agent is equipped with: (t) a stochastic policy π : H4 - (AA) that maps from observed history to distribution over actions, and (it) a value function that maps from history and action to the expected return under the policy, defined
Q" Hi A
as (h., ax) E. > at) The objective of the agent is to find the optimal policy that
maximizes the value starting from some initial state so, nt argmax, Eg EnD xTre 50
3.2 History Representations
A weakness of the general formulation of RL in POMDPs is the dependence of both the policy and the value function on the ever-growt ever-growing history. Instead, practical algorithms fight this curse of dimensionality by compressing the history into a compact representation. Ni et al. (2024) propose to learn such representations via history encoders, defined by a mapping á n4 + Z from observed history to some latent representation 2; == (hh.) E With slight abuse of notation, we denote л(a; 24) and Q* (zt, at) as the policy and values under this latent representation, respectively.
3.3 Probabilistic Inference on Linear SSMs
We consider Line-varying discrete, Chear-Gaussian SSMs defined by
Et A, + B Es, ye = C, + Dru4-1 Vt.
(1)
where / > O, Ze R is the hidden or latent state, W, E R. is the input, E R M is the output, (A,, B, D.) are matrices of appropriate size, Et ~ (O, EP) and it N(O ΣP) are zero-men mean process and obsorvation noise variables with their covariance matrices ΣP and Eg, respectively. Without loss of generality and as it is common in linear SSMs, we set D. O. The latent state probabilistic model is then p(z; 2z-1, U(-1) (A B, Σ/) and the observation model is p(n. 21) WC Eg). Inference in such model has closed-form solution, which is equivalent to the well-studied Kalman filter (Kalman, 1960).
Predict. The first stage of the Kalman filter propagates forward the posterior belief of the latent state at step - 1, given by N( Σt1), ) to obtain a prior belief step i, N(xx 27), given by
x = A B Ex A.E. A. + E.
(2)
5

---

Published in Transactions on Machine Learning Research (02/2025)
Update. The second stage updates the prior belief at step given some observation wy, to obtain the posterior p(rz (t-1; M(x Eit) given by
य zt + Ku C,, = -1-
(3)
where K, = 2, (C,,E CT + - is known the Kalman gain. The predict and update steps are interleaved to process sequences of input and observations {u,, [ of length K, starting from some initial belief (( Σt,).
3.4 Simplifying Assumptions
The Kalman filter prodict and update equations from (2) and (3) involve expensive matrix multiplication and inversion, which scales poorly with the latent state dimension N. In this section we propose several simplifications, both for easier implementation but also for better scalability.
Time-invariance. Prior work proposed time-varying SSMs via state-dependent (Becker et al., 2019) or input-dependent (Gu and Dao, 2023) matrices. Instead, in this work we propose using simple time- invariant matrices, as similarly done in prior deterministic SSMs (Gu et al., 2022a; Smith et al., 2023). First, state- dependent matrices is well motivated by local linearization of dynamics, but they are incompatible with efficient parallel scan routines, as they break the associative property of the Kalman filter equations. Second, while input-dependent matrices excel in associative recall problems where input-selectivity is necessary, Kalman filters provide similar selection mochanisms via its postcrior update (sce Section 4.2), without the need for time-varying matrices. Concerning lime-invariant process noise, such simplification reduces the expressivity of the model. However, our initial experiments with input-dependent process noise showed worse performance in RL than its time-invariant alternative (see Section 4.2 and Appendix E).
Diagonal matrices. In order to scale to higher-dimennsooall latent spaces, prior work in both deterministic and probabilistic SSMs consider structured SSMs. This simply means special structure is imposed into the learnable matrices (A, B, C). In particular, we consider diagonal structure with the HiPPO initialization proposed in Gu et al. (2020), which induces stability in the recurrence for handling long sequences. In addition, we also consider: (i) diagonal process and observation noise covariances, (ii) N = M = P. which simplifies the implementation and (iii) identity emission matrices C = I, as proposed in (Becker and Neumann, 2022). Under this parameterization, the expensive Kalman filter equations reduce to element-wisc operations:
" = diag(A) A) ① x + diag(B) (B) @ u1 diag( Σ, = diag(A) 2 diag(E;t,) + diag(E!),
(4)
xt. xr + diag K. (w) x;) diag Σ (diag diag(K.)) diag E,);
(5)
diag K.) diag( Σ @ (diag( (diag(E,)) + diag(E?)),
(6)
where C denoted element-wise vector product and Ø denotes element-wise vector division.
3.5 Parallel Scans
Efficient implementation of state-space models and Kalman filters employ parallel scans to achieve logarithmic runtime scaling with the sequence length (Smith et al., 2023; Sarkka and Carcia-Fernandez, 2021). Given a sequence of elements (On, a1...., at-1) and an associative binary operator the parallel scan algorithm outputs all the prefix-suns (ag, ag a1...., ag az-1) in Olog K) runtime, given sufficient parallel processors.
4 Method: Off-Policy Recurrent Actor-Critic with Kalman filter Layers
In this section, we describe our method that implements Kalman filtering as a recurrent layer within a standard actor-critic architecture.
A binary operator
associative (a bb e c) for any triplet elements (o, b)

---

Published in Transactions on Machine Learning Research (02/2025)
Actor
Ot
0:t
h
History Encoder
zt
Embedder
MLP
at
a:
Ot
History Encoder
2
MLP
Q1,t
0:t
QQQ hig
Embedder
a:t-1
History Encoder
F1
MLP
Q2,t
Critic
Figure 1: General Recurrent Actor-Criti CrioicCriti (RAC) architecture. The components are trained end-to-end with the Soft Cettr-Critic (SAC) loss function (Haarnoja et al., 2018). To handle discrete action spaces, we use the discrete version SAC by Christodoulou (2019).
KF Layer
O
Uk
Predict
U
h
Linear
Wit
3 k Σ
Linear
Z,t
Σo :t
Wik
Update
Figure Our proposed Kalman filter layer build history encoders. The KF layer receives a history sequence ha and projects it into three separate signals in latent space: the input ua, the observation wa and the observation noise (diagonal) covariance Σg These sequences filtering equations, which scale logarithmically with the sequence length using parallel scans. Lastly, the posterior mean latent state x is projected from the latent space back into the history space to obtain the compressed representation Z.z-
processed using the standard Kalman
4.1 General Architecture
In Figure we present our Recurrent Actor-Crite (RAC) architecture inspired by Ni al. (2022), where we replace the RNN blocks with general history encoders. We will use this architecture in the following test the capabilities of different history encoders in various POMDPs.
For both actor and critic, we embed the sequence of observations and actions into a single representation h which is then passed into the history encoders. We use a single linear layer as embodder, which we found worked as reliably as more complex non-linear embedders used in similar RAC architectures by Morad et al. (2023); Ni et al. (2022). We also include the skip connections from current observations and actions into the actor-critic heads, as proposed in previous memory- architectures (Zintgraf et al., 2021; Ni al., 2022).
4.2 Kalman Filter Layers
Our main hypothesis that principled probabilistic filtering within history encoders boosts performance in POMDPs, especially those where reasoning about uncertainty key for decision-making, To test this hypothesis, introduce KF layers, as shown in Figure 2. The layer receives as input a history embedding

---

Published in Transactions on Machine Learning Research (02/2025)
sequence hu which is then projected into the input ua, observation w. and observation noise Σg sequences. These three signals serve as input to the standard KF predict-update equations (2) and (3), which output a posterior (filtered) latent state z.. Finally, the posterior sequence is projected back to the history embedding space to produce the compressed history representation 2,--
History encoders with KF layers. Similar to recent SSM layers such as $5 (Smith et al., 2023) and S6 (Gu and Dao, 2023), these KF layers can be stacked and combined with other operations such as residual connections, gating mechanisms, convolutions and normalization to compose history encoder block in the RAC architecture. In favor of simplicity, our history encoders are only composed of KF layers and (optionally) an RMS normalization (Zhang and Sennrich, 2019) output block for improved stability.
Filtering as a gating mechanism. We can draw interesting comparisons between KF layers and other recurrent layers from the perspective of gating mechanisms. It was shown in Theorem 1 of (Gu and Dao, 2023) that selective SSMs (S6) behave as generalized RNN gates through an input-dependent step size A. In this case, the gate depends on the SSM input and controls how much the input influences the next hidden state. Similarly, as hinted by Becker et al. (2019), during the update step the Kalman gain is effectively an uncertainty-controllee gate depending
the observation noise which regulates how much the observation influences the posterior belief over the latent state. Our experiments in Section 5 shed some light on the strengths and weaknesses of these approaches for RT, under partial observability.
SSM Parameterization. We follow the procedure in Gu and Dao (2023) and initialize the continuous- time system (A, B) with HiPPO matrices. The corresponding discrete-time system (A.B) is obtained via zero-order hold discretization with learnable scalar step size A > 0 (Smith al., 2023).
Design decisions. We want to highlight two considerations that went into the design of our KF layers. First, we could generalize the architecture to support time-varying process noise by including one extra output channel (alongside the input, observation and observation noise channels) in the history linear projection. Conceptually, such an input-dependent process noise adds more flcxibility to the gating mechanism implemented within the KF layer, which would be controlled both by the observation and the process noise signals. Second, we could include the posterior covariance Σ as an additional feature for the output linear projection, alongside the posterior mean 2;- We conduct an ablation study over those two choices in several continuous control tasks subject to observation noise and report the results in Appendix E. The best aggregated performance in this ablation was obtained with time-invariant, process noise and only using the posterior mean as feature for the output projection, which empirically justifies our final design.
4.3 Masked Associative Operators for Variable Sequence Lengths
In off-policy RAC architectures, the agent is typically trained with batches of (sub-)trajectorie) of possibly different length, sampled from an experience replay buffer. Thus, history encoders must be able to process batches of variable sequence length during training.
A common approach is to right-pad the batch of sequences up to a common length and ensure the model's output is independent of the padding values. For transformer models, this can be achieved by using the padding mask self-attention mask. For stateful models like RNNs and SSMs, it is imperative to also output the correct final latent state for each sequence in the batch. This typically requires a post-proccssing step that individually selects for each sequence in the batch the last state before padding. It turns out that for any recurrent model expressed with an associative operator (e.g., SSMs and KFs), we can obtain the correct final state from a batch of padded sequenees without additional post-processing by using a parallel scan routine with a Masked Associative Operator (MAO).
Definition 1 (Masked Associative Operator). Let. o be an associative operator acting on elements e E E, such that for any a, b, E E, it holds that (a b) c= a (b c). Then, the MAO associated with denoted ē, acts on elements ē E E (0, 1] = (c, m),, where m E f0. 1 is a binary mask. Then, for à (a, ma) and b- b (b, ms); we have:
(a b, ma)
if mg = 0
(7)
à
if m& =1 1
8

---

Published in Transactions on Machine Learning Research (02/2025)
In Appendix A, we show that any MAO is itself associative long as we apply a right-padding mask?, thus fulfilling the requirement for parallel scans. In practice, augmenting existing SSM and KF operators with their MAO counterpart is a minor code change. MAOs act as a pass-through of the hidden state when padding is appliod, thus yiclding the correct state at every time step of the padded sequence for each element of the batch without additional indexing or bookkeeping. Due to their pass-througl nature, MAOs require strictly equal or less evaluations of the underlying associative operator, which may yield faster runtimes if the operator is expensive to evaluate and/or many elements of the input sequence are masked.
MAOs for SSMs and KFs. As concrete example, the associative operators for SSMs and KF's involve matrix product and addition. A compute--fficient implementation of MAOs for such operators involves sparse matrix operations, where the sparsity is dictated by the padding mask. However, sparse matrix operations are only expected to yield better runtime than their dense counterparts for large matrices with sufficient levels of sparsity, which are not typical in our application. Thus, no speed-up expected from using MAOs in the context of this work.
MAOs are similar to the custom operator proposed by Lu et al. (2023), but their effect is fundamentally different: Lu et al. (2023) considers on-policy RL, where the goal is to handle multi-episode sequences, thus their custom operator resets the hidden state at episode boundaries. Instead, in our off-policy RAC architecture, MAOs act as pass-through of the hidden state for padded inputs.
5
Experiments
In this section, we evaluate the RAC architecture under different history encoders in various POMDPs. Implementation details and hyperparameters are included in Appendices and C, respoctively.
5.1 Baselines
We consider the following implementation of history encoders within the RAC architecture.
VSSM. Vanilla, real-valued SSM with diagonal matrices. It is equivalent to a KF layer with infinite observation noise, iee, the update step has no influence on the output. It can also be seen as a simplification of the S4D model (Gu et al., 2022b), where states are real-valued rather than complex (as in Mega (Ma al., 2023), such that the recurrence can be interpreted as an exponential moving average) and the recurrence is implemented with a parallel scan rather than convolutionn (as in (Smith et al., 2023)).
VSSM*KF. Probabilistic SSM via the KF layers described in Figure 2. While VSSM only predicts the next state (i.e., the prior in Kalman filtering), vSSM+KF additionally filters the predicted state conditioned on the latent observation. Therefore, VSSM+KF is equivalent to vSSM with the additional update step of the Kalman filtor. Similarly, vSSM is equivalent to vSSM+KF with an infinite observation noise variance.
VSSM+KF-u. Equivalent vSSM+KF without the input signal u.r. It maintains the uncertainty- based gating from the KF layer, but looses flexibility in the KF predict step to inffuence the prior belief via the input.
Mamba (Gu and Dao, 2023). Selective state-space model with input-dependent state transition matrices.
GRU (Cho et al., 2014). Stateful model with a gating mechanism and non-linear state transitions.
vTransformer (Vaswani et al., 2017). Vanilla encoder-only transformer model with sinusoidal positional encoding and causal self-attention.
All SSM-based approaches are implemented using MAOs and parallel scans. Besides these memory-based agents, include two additional memoryless agents that implement the same RAC architecure but without embedders or history encoders.
Oracle. It has access to the underlying state of the environment, effectively removing the partial observability aspect of the problem. This method should upper-bound the performance of history encoders.
AA right-padding mask is sequence (my m with m, E {0, 13 such that if m, = then m, for >
9

---

Published in Transactions on Machine Learning Research (02/2025)
2.5
2.5
0.0
0.0
0
20
40
20
40
Env. steps
Env. steps
Noisy obs. Bayesian state True state
Decision boundary
Two example episodes of the Best Arm Identification task of Section 5.2, with μt = 0.5 and two different noise scales. (Left) Narrow noise distribution with σg = 0.5. (Right) Wide noisc distribution with on = 1.0. In red, we visualize the Bayesian posterior mean and 3o confidence interval around μbb obtained via Bayesian linear regression using all prior observations in the episode.
Figure 3:
Memoryless. Unlike Oracle, it does not have access to the underlying state of the environment. This method should lower-bound the performance of history encoders.
All the baselines share a common codebase and hyperparameters. For all stateful models, we use the same latent state dimension N such that parameter count falls within 10% tolerance range except for GRU, which naturally has more parameters due to its gating mechanism (roughly 40% increase). For vTransformer we choose the dimension of the feed-forward blocks such that the total parameter count. is also within 10% of the SSM methods. With this controlled experimental setup, we aim evaluate strengths and weaknesses of the different mechanisms for sequence modelling (gating, input-selectivity, probabilistic filtering, self-attention) in wide variety of partially observable environments.
5.2 Probabilistic Reasoning Adaptation and Generalization
We evaluate probabilistic reasoning capabilities with a carefully designed POMDP that simplifies our motivating example from Section where Al chatbot probes a user in order to recommend restaurant. Given noisy scalar observations sampled from bandit with distribbtion ((us, σb), the task is to infer whether the mean μl lies above or below zero. At the start of each episode, H4 and σb (the latent parameters) are sampled from some given distribution. Then, at cach step of an episode, the RL agent has three choices: (1) request a new observation from the bandit, which incurs a cost. p, (2) decide the arm has mean above zero or (3) decide the arm has mean below zero, both of which immediately end the episode and provide a positive reward if the decision was correct, a negative reward if the decision was incorrect. We set a maximum episode length of 1000 steps; if the agent does not issue decision by then, it receives the negative reward. Example rollouts for this environment are provided in Figure 3. Given the Bayesian state from Figure 3, an optimal agent must strike a balance between requesting new information (which reduces uncertainty about the estimated mean) and minimizing costs. Effective history encoders for this problem should similarly produce a state representation that encodes uncertainty about the latent parameters.
We evaluate two core capabilities: adaptation and generalization. Intuitively, an optimal policy for this problem must be adaplive depending the latent parameters. For example, if ns close to zero the agent might need many observations to make an informed decision, whereas with large |up the correct decision can be made with few observations. Morcover, we can also evaluate generalization of the learned policy by testing on latent parameters not seen during training. Our hypothesis is that an agent that learns proper probabilistic reasoning (e.g,, Bayes' rule) should generalize reasonably well in this task.
We conduct experiments for all basclines under inercasing cost p. Instead of providing the latent parameters directly to the Oracle baseline, we provide the Bayesian posterior mean and standard deviation around the latent parameter (b; as shown in Figure The agents are trained under the latent parameter distribution given by 14 Unif(-0.5. 0.5) and σb Onif(0.0, 2.0). We additionally evaluate ditoo--distributinn generalization by using goD ~ Unif(2.0, 3.0), i.c., we test how the agent generalizes to bandits with higher variance. In Figure 4
10

---

Published in Transactions on Machine Learning Research (02/2025)
In-distribution
Out-of-distribution
Ep. length Norm. return
0.5
0.0
10a
10
10
0.0
0.01
0.05
0.0
0.01
0.05
Cost
Cost
VSSM+KF
VSSM
GRU
Memoryless
VSSM+KF-u
Mamba
vTransformer
Oracle
Figure Performance of sequence models in the Best Arm Identification problem after 500K environment steps. We conduct experiments for increasing cost of requesting new observations and evaluate performance both in and out of distribution, averaged over 100 episodes, and report the mean and standard error over 5 random seeds. (Top row) Normalized return, obtained by dividing returns by the reward given after winning (10 in our case). (Bottom row) Length of episodes.
we report the normalized return and average episode length for both the training and out-of-distributioo latent parameters. Full training curves are included in Appendix D. vSSM+KF achieves the highest return out of the memory-basod agents, both in and outtof-distribution, while matching the performance of Oracle in-distribution. The better performance of vSSM*KF correlates with longer episodes: compared to the other baselines, vSSM+KF learns to request more observations in order to issue more informed decision.
vSSM+KF improves adaptation and generalization. To gain further insights on the results, we do a post-training evaluation on a subset of the agents across the entire latent parameter space, shown in Figure vSSM+KF learns adaptation patterns similar to Oracle: the length of episodes increase as the noise scale ab increases and decrease as |ua] increases, as it intuitively expected. Such adaptation is less pronouneed in VSSM, vSSM+KF-u and Mamba, where episodes are shorter and ultimately results in lower win rates. While vSSM*KF does not match the generalization performance of Oracle, it remains the best amongst the history encoder basclines. Given controlled experimental sctup, attribute the enhanced adaptation and generalization of vSSM+KF to the internal probabilistic filtering implemented in the KF layer. Moreover, comparing vSSM+KF and vSSM+KF-u highlights that including the input signal in the KF layer leads to improved performance this task.
vSSM+KF can handle adversarial episodes. In Figure we compare latent space rollouts from vSSM+KF and vSSM in an adversarial episode: Ps is negative, but the first two observations are positive and of relatively large magnitude. After only four observations, vSSM is mislead by the positive obscrvations and issues the wrong decision, visualized in Figure 6 (middle) where we show the policy's output across latent space, overlaid with the rollout trajectory. Instead, vSSM+KF remains in the region where the policy requests more observations before it navigates to the correct region of latent space, as shown in Figure (right). While this example was hand-picked is consistent with the adaptation patterns from Figure 5.
5.3 Probabilistic Filtering Continuous Control under Observation Noise
In this experiment, we evaluate the ability to learn control policies subject to observation noise. Effective history encoders must learn to aggregate observations over multiple time steps to produce a filtered state
WW use latent state dimension worse performance than the results reported in Figure 4, where we use 128.
order to plot the policy decision boundary latent space, This results slightly
11

---

Published in Transactions on Machine Learning Research (02/2025)
Oracle
vSSM+KF
VSSM+KF-u
VSSM
Kamba
100
1%% anaa TEMM
0.0
50
-0.5
0.5
10a
[%6 tnmttl dg
0.0
10
-0.5
100
0.0
2.20 3.0 0.0
02.0 3.0 0.0
o2.0 3.0
0.0
o2.0 3.0 0.0
2. 3.0
Figure 5: Performance heatmap on Best Arm Tdentification problem (p= 0). We generate grid noise parameters (Hb; στ) for a total of 625 unique combinations. The red vertical line separates training (to the left) from out-of-distribution (to the right) latent parameters. For each pair of latent parameters, we evaluate performance on five independently trained agents over 100 episodes and report the average win rate and episode lengths.
Rollout with 1bb -0.2, 1.0
VSSM
VSSM+KF
New obs.
0
Mbb
5
< 0
0
10
20
-2
5
01
2
Env. steps
Noisy obs.
P4b
T1
Figure 6: Latent space rollouts in adversarial Best Arm Identification episode. (Left) Rollout in latent space (N = 2) for vSSM+KF and VSSM after training. (Widdle-Right Policy decision boundaries overlaid with the latent space trajectory. Circles and stars denote the beginning and end of trajectories, respectively.
representation amenable for control. Our hypothesis is that internal probabilistic filtering provides an inductive bias for learning such a filtered representation. To test our hypothesis, we conduct evaluations across nine environments from the DeepMind Control (DMC) suite (Tunyasuvunakool et al., 2020) with zero-mean Gaussian noise added to the observations, as done by Becker and Neumann (2022); Becker el al. (2024). We present aggregated performance in Figure 7 following the recommendations from (Agarwal et al., 2021). Detailed training curves are included in Appendix F. We discuss the main insights from this experiment.
vSSM+KF improves performance of stateful models. The KF layer is the only evaluated add-on for stateful models that significantly improves performance over the baseline model vSSM. This suggests that the uncertainty-based gating in Kalman filters more effective at handling noisy data compared to the gating mechanism implemented by GRU and Mamba. This observation matches the results in the Best Arm Identification problem from Section 5.2. Comparing vSSM+KF and vSSM+KF-u, there slight improvement in performance from using input signal in the KF layer, but it is not statistically significant.
vSSM*KF learns consistently across environments. From the detailed results in Figure 14, we observe that vSSM+KF consistently improves performance over the Memoryless lower-bound and achieves the best or
12

---

Published in Transactions on Machine Learning Research (02/2025)
1.00
1.00
Norm. IQM Return
Fraction of runs
0.75
0.75
with score >
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Env. steps (millions)
Normalized score (r)
vSSM+KF
vSSM
GRU
Memoryless
vSSM+KF-u
Mamba
vTransformer
Oracle
Figure 7: Aggregated performance in noisy DMC benchmark (9 tasks) with 95% bootstrap confidence intervals over five random seeds. (Left) Inter-quartile mean returns normalized by the score of Oracle. (Right)
Performance profile after 1M environment steps. Higher curves correspond to better performance.
finger-spin
cheetah-run
walker-run
Norm. Return
pendulum
reacher
point-mass
Norm. Return
Ox
0.5x
1x
Ox
0.5x
1x
Ox
0.5x
1x
Noise
Noise
Noise
vSSM+KF
VSSM+KF
VSSM
Mamba
GRU
vTransformer
Figure 8: Final performance comparison of recurrent models in six tasks increasing noise levels. We
report the mean and standard error over five random seeds (ten for pendulum due to large variance) of the return after 1M environment steps, normalized by the score of Oracle.
comparable final performance five out of nine tasks. Instead, GRU, Mamba and vTransformer completely lail to learn some lasks, barely matching the performance of Memoryless.
We conduct an additional ablation over increasing noise levels in six representative tasks from the DMC suite, shown in Figure 8. Training curves are included in Appendix G
vSSM+KF performs close to Oracle under full observability. We observe vSSM+KF generally matches the performance of Oracle in the absence of noise (normalized score close to 1.0), whereas vSSM and VTransformer significantly underperform in some tasks. This suggests that the added probabilistic filtering in vSSM+KF is a general-pupose strategy even under full observability.
vSSM+KF's robustness to noise is environment dependent. Figure 8 suggests that robustness noise depends generally the environment, without any clear patterns related to task specifies. vSSM+KF is
13

---

Published in Transactions on Machine Learning Research (02/2025)
1.5
1.00
POPGyn Normalized IQM MMER
Fract ion of runs
1.0
with score >
0.75
0.5
0.50
0.0
0.25
0.00
-0.5
0.00
0.25
0.50
0.75
1.00
0.0
0.5
1.0
1.5
2.0
Env. steps (millions)
POPGym Normalized score (r)
vSSM+KF
vSSM
GRU
Memoryless
vSSM+KF-u
Mamba
vTransformer
Oracle
Figure 9: Aggregated performance in POPGym selected environments (12 lasks) with 95% bootstrap confidence intervals over five random seeds. We normalize the maximum-mees mean episodic return (MMER) by the best reported MMER in (Morad et al., 2023) (Left) Normalized IQM MMER (Right) Performance profile after 1M environment steps. Higher curves correspond to better performance and a score of 1.0 means equivalent performance as the best baseline (per environment) reported in POPGym.
robust in finger-spin, cheetah-run and pendulum', vSSM is more robust in walker-run but significantly underperforms in other environments, and vTransformer is more robuts in reacher and point but fails to learn in pendulum. Overall, vSSM+KF shows the most consistent performance across environments and noise levels.
We include additional experiments with noisy DMC tasks in Appendix H, where we compare performance of VSSM and vSSM+KF against state-of-the-art moddel-asee approaches. The main insight from this comparison is that our model-free approach mostly matches the performance of model-based methods without additional representation learning objectives.
5.4 General Memory Capabilities
far the evaluations were conducted in tasks where probabilistic filtering was intuitively expected excel. In this experiment, we evaluate performance in wider variety of POMDPs from the POPGym (Morad et al., 2023) benchmark. We select subset of 12 tasks that probe models for long-term memory, compression, recall, control under noise and reasoning. The aggregated results are shown in Figure and full training curves are also included in Appendix I. Below we discuss the main insights.
KF layers can be generally helpful in POMDPs. From the performance profile in Figure 9 we observe a statistically significant gap between VSSM and vSSM+KF. Interestingly, the largest improvements in samplo-cffiiencyy [RepeatPreviousEasy] and final performance (MineSveeperEasy) correspond tasks that probe for memory duration and recall, respectively. The parameter count difference between vSSM and vSSM+KF in these problems is less than 6%, we believe model capacity is unlikely the reason behind the large performance difference. We hypothesize that, while probabilistic filtering is not required to solve these tasks, the KF layer has extra flexibility via the latent observation and noise signals to accelerate the lcarning process. We also highlight that vSSM+KF and vSSM*KF-u show comparable performance in this benchmark, suggesting the input signal to be less critical in general memory tasks.
vSSM+KF is less sample-effccintt in pure-memory tasks. particular, we observe that Mamba's 's input- selectivity is the best-suited mechanism for SSM agents to solve long-term memory problems, matching the
"We found that Oracle underperforms in the noiseless pendulun-ewingup. similarly reported in (Luis al., 2023), which is why the normalized score in this task is larger than 1.0 in some cases. Moreover, performance does not strictly decrease under higher noise levels, perhaps because noise may actually help avoid early convergence under sparse rewards.
14

---

Published in Transactions on Machine Learning Research (02/2025)
MineSweeperBasy
BanditHard
NoieyPendulumHard
RepeatFinstMedim
N 64, L
N = 64, I,
N
L
W = 128, L,
N 128, L
N 128, L
3
N 256, L
N 256 I
N 256 L
MMER 1
MMER
0
MMER
MMER
Figure 10: POPGym ablation for vSSM+KF over the latent state size N and the number of layers L. We report the mean and standard error over five random seeds of the MMER score after 1M environment steps. The MMER score is shifted from [-1, to [0, 1] for easier visualization. The vertical line represents the best score reported by Morad et al. (2023).
performace of GRU and vTransformer. This is an expected result based on the associative recall performance of Mamba reported in its original paper (Gu and Dao, 2023).
Linear SSMs can have strong performance. Morad et al. (2023) report poor performance when combining PPO with the S4D (Gu et al., 2022b) model. While we do not evaluate the S4D model and use an off-policy algorithm in our RAC architecture, our evaluation shows various linear SSMs have strong performance, often surpassing the bost reported scores Morad et al. (2023). Our observation is consistent with the strong performance of PPO with the S5 model reported by Lu el al. (2023).
5.5 Ablation
We conduct an ablation on vSSM+KF where we vary two hyperparameters: the latent state size and the number of stacked KF layers L,5. We select four representative tasks from POPGym that test different. memory capabilities. The final scores are presented in Figure 10 and the full training curves are included in Appendix K. Performance is most sensitive to these hyperparameters in the RepeatFirstMediun task, where the agent must recall information from the first observation over several steps. The general trend is that using more than one layer improves final performance and increascs sample-cfficinny (sce the training curves in Figure 19). Our results are aligned with the good performance of stacked S5 layers reported by Lu et al. (2023), but differ from the observations in (Ni et al., 2023), where both LSTM and transformer models performed best with a single layer similar long-term memory task (T-maze passive). From these observations, we believe an interesting avenue for future work is study what mechanisms enable effective stacking and combination of multiple recurrent layers.
6 Conclusion
We investigated the of Kalman filter (KF) layers sequence models in recurrent actor-critic architecture. These layers perform closed-form Gaussian inference in latent space and output filtered state representation for downstream RL components, such as value functions and policies. Thanks to the associative nature of the Kalman filter equations, the KF layers process sequential data efficiently via parallel scans, whose runtime scales logarithmically with the sequence length. To handle trajectorics with variable length in off-policy RL, we introduced Masked Associative Operators (MAOs), a general-purpose method that augments any associative operator recover the correct hidden state when processing padded input data. The KF layers are used as drop-in replacement for RNNs and SSMs in recurrent architectures, and thus can be trained similarly in end-to-end, modcl-free fashion for return maximization.
We evaluated and analysed the strengths and weaknesses of several sequence models in wide range of POMDPs. KF layers excel in tasks where uncertainty reasoning is key for decision-making such as the Best Arm Identification task and control under observation noise, significantly improving performance over
We RMSNorm output block in vSSM*KF since it was critical ensure stable learning when
15

---

Published in Transactions on Machine Learning Research (02/2025)
stateful models like RNNs and deterministic SSMs. In more general tasks, including long-term memory and associative recall, KF layers typically match the performance of transformers and other stateful sequence models, albeit with lower sample-efficincyy
Limitations and Future Work. We highlight notable limitations our methodology and suggest avenues for future work. First, we investigated two design decisions KF layers related to vire-igrying process noise and posterior covariance as output features. While they resulted in worse performance (see Appendix E),, in principle they generalize KF layers and may bring benefits in other tasks contexts, we believe it is worth further investigation. Second, is standard in RL but not other supervised learning tasks. It may be possible that deeper models with larger parameter counts enable new capabilities, e.g., probabilistic reasoning, without explicit probabilistic filtering mechanisms. Third, vSSM*KF uses KF layers as standalone history encoders, but more complex architectures may be needed stabilize training at larger parameter counts. Typical strategics found in models like Mamba include residual connections, layer normalization, convolutions and non-lincarities. Fourth, our evaluations were limited to POMDPS with relatively low-dimensional observation and action spaces, where small models have enough capacity for learning. Future work could further evaluate performance in more complex POMDPs (e.g., with image obscrvations) and compare with our findings.
models with relatively low parameter count (< 1M) which
Acknowledgments
We would like to thank Philipp Becker for providing an efficient parallel scan routine in Pytorch which we use on all our SSM-basod experiments.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pages 29304 293200. Curran Associates, Inc., 2021.
François Auger, Mickael Hilairet, Josep M. Guerrero, Eric Monmasson, Teresa Orlowska-Kowalske, and Seiichiro Katsura, Industrial Applications of the Kalman Filter: A Review, IEEE Thansactions on Industrial Electronics, 60(12):5448-5444 544--4411 2013.
Bram Bakker. Reinforcement Learning with Long Short-Tern Memory. In Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001.
Philipp Becker and Gerhard Neumann. On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning. Transactions on Machine Learning Research, July 2022.
Philipp Becker, Harit Pandya, Gregor Gebhardt, Cheng Zhao, C. James Taylor, and Gerhard Neumann. Re- current Kalman Networks: Factorized Inference in Dimh-DDimnsional Deep Feature Spaces. In International Conference on Machine Learning, volume 97, pages 652-552. PMLR, June 2019.
Philipp Becker, Niklas Freymuth, and Gerhard Neumann. KalMamba: Towards Efficient Probabilistic State Space Models for RL under Uncertainty. In ICML Workshop on Aligning Reinforeement Learning Experimentalists and Theorists (ARLET), June 2024.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253 279, June 2013.
Christopher M. Bishop. Pattern Recogrition and Machine Learning, volume 29. Springer, 2006.
Noam Brown and Tuomas Sandholm. Superhuman AI for Multiplayer Poker. Science, 365(6456))..) 890, 2019.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. TransDreamer: Reinforcement Learning with Transformer World Models. In DeeplL Workshop NewrIPS. arXiv, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2025)
S. Y. Chen. Kalman Filter for Robot Vision: A Survey. IEEE Transactions on Industrial Electronics, 59(11): 4409-4420, 2012.
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the Properties of Neural Machine Translation: Encoder- Encoder-Decoder Approaches. In 8th Workshop on Syntar, Semanties and Structure in Statistical Translation, pages 103 111. Association for Computational Linguistics (ACL), 2014.
Petros Christodoulou. Soft Actor-Critic for Discrete Action Scttings. arXiv:1910.07200 October 2019.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G. Bellemare, and Aaron Courville. Sample-Eificient Reinforcement Learning by Breaking the Replay Ratio Barrier. September 2022.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charles Dock, Joel Z Leibo, and Charles Blundell. Generalization of Reinforcement Learners with Working and Episodic Memory. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards Language Modeling with State Space Models. In International Conference on Learning Representations, 2023.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Re. It's Raw! Audio Generation with State-Space Models. In International Conference on Machine Learning, volume 162, pages 7616-7633. PMTR, July 2022.
Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.000 20000005, 2023.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. HiPPO: Recurrent Memory with Optimal Polynomial Projections. In Advances in Neural Information Processing Systems, volume 33, pages 1474-1487. Curran Associates, Inc., 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference on Learning Representations, 2022a.
Albert Gu, Ankit Gupta, Karan Goel, and Christopher Ré. On the Parameterization and Initialization of Diagonal State Space Models. In Advances in Neural Information Processing Systems, volume 35, 2022b.
David Ha and Jürgen Schmidhuber. Recurrent World Models Facilitate Policy Evolution. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018.
Tuomas Haarnoja, Amurag Ajay, Sergey Levine, and Pieter Abbeel. Backprop KF: Learning Discriminative Deterministic State Estimators. In Advances in Newral Information Processing Systems, pages 4383 4391, 2016. Curran Associates Inc.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In International Conference Machine Learning, volume 80, pages 1870- PMLR, July 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications. arXi::1822.0552 January 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning Latent Dynamics for Planning from Pixels. In International Conference on Machine Learning, volume 97, pages 2555 2565. PMLR, June 2019.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Danijar Hafner, Timothy Lillicra,, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learning
Behaviors by Latent Imagination. In International Conference on Learning Represenlations, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering Diverse Domains through World Models, January 2023. URL http://arxiv.
Matthew Hausknecht and Peter Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In AAAT Fall Symposium Series, 2015.
Nicolas Heess, Jonathan J. Hunt, Timothy P. Lillicrap, and David Silver. Memory-Based Control with Recurrent Neural Networks. In NIPS Deep Reinforcement Learning Workshop, December 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Terme Memory, Neural Computation, 9(8) 1780, 1997.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic Domains. Artificial Intelligence, 101(1):99 134, 1998.
R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35 45, March 1960.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data. February 2017.
Bahador Khaleghi, Alaa Khamis, Fakhreddine O. Karray, and Saiedeh N. Razavi. Multisensor Data Fusion: A Review of the State-of-the-art. Information Fusion, 14(1):28 44, 2013.
Diederik P. Kingma and Max Welling. Auto- Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cseke, and Patrick van der Smagt. Latent
Matters: Learning Deep State-Space Models. In Advances in Neural Information Processing Systems, volume 34, pages 10234-10245. Curran Associates, Inc., 2021.
Nya Kostrikov. PyTorc Implementtations of Reinforcement Learning Algorithms,, 2018. URL https: //github.com/ http://ossttkoo//yooooo http:///.....ttoeoooottorroaaaap awt.p-----atttttaaa..a.aa
Rahul Krishnan, Uri Shalit, and David Sontag. Structured Inference Networks for Nonlinear State Space
Models. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), February 2017.
Rahul G. Krishnan, Uri Shalit, and David Sontag, Deep Kalman Filters, November 2015. URL http: wwwpd/.org/bbs/111..00.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective Mismatch in Model-based Reinforcement Learning. In Learning for Dynamics and Control, pages 761 770. PMLR, 2020.
Michael Laskin, Aravind Srinivas, and Picter Abbocl. CURL: Contrastive Unsupervised Representations for Reinforcement Learning. In Internalional Conference on Machine Learning, volume 119, pages 5639 5650. PMLR, July 2020.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A Contextual- Bontextual-Banditt Approach to Personalized News Article Recommendation. In International Conference on World Wide Web, pages 661 (61-670, 2010. Association for Computing Machinery.
Long-Ji Lin and Tom M Mitchell. Reinforcement Learning with Hidden States. In International Conference on Simulation of Adaptive Behavior, pages 271-280. MIT Press, 1993.
Qinghua. Liu, Alan Chung, Csaba Szepesvari, and Chi Jin. When Is Partially Observable Reinforcement Learning Not Scary? In Conference on Learning Theory, pages 5175-5220. PMLR, June 2022.
18

---

Published in Transactions on Machine Learning Research (02/2025)
Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured State Space Models for In-Context Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36, pages 47016 47031. Curran Associates, Inc., 2023.
Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, and Jan Peters. Value- Distributional Model-Based Reinforcement Learning. arXi::208.0008 066900 August 2023.
Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. Mega: Moving Average Equipped Gated Attention. In International Conference on Learning Represenlations, 2023.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning, In NIPS Deep Learning Workshop, December 2013.
Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POPGym: Benchmark- ing Partially Observable Reinforcement Learning. In International Conference on Learning Representations, 2023.
Kevin Murphy, Machine Learning: A Probabilistic Perspective. MIT Press, 2012.
Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. SAND: Modeling Images and Videos as Multidimensional Signals with State Spaces. In Advances in Neural Information Processing Systems, volume 35, pages
Curran Associates, Inc., 2022.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. International Conference on Machine Learning, pages 16691 16723. PMLR, June 2022.
Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Lue Bacon. When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment. Advances in Neural Information Processing Systems, volume 36, pages 50429-50452. Curran Associates, Inc., 2023.
Tianwei Ni, Benjamin Fysenbach, Erfan Seyedsalehi, Michel Ma, Clement Gehring, Aditya Mahajan, and Pierre-Luc Bacon. Bridging State and History Representations: Understanding Self-Predictive RL. In International Conference on Learning Representations, 2024.
Junhyuk Oh, Valliappa Chockalingam, Satinder, and Honglak Lec. Control of Memory, Active Perception, and Action in Minecraft. In International Conference on Machine Learning, volune 48, pages 2790 2799, June 2016. PMLR.
Christos H. Papadimitriou and John N. Tsitsiklis. The Complexity of Markov Decision Processes. Mathemalics of Operations Research, 12(3):441-450, 1987.
Emilio Parisotto and Russ Salakhutdinov. Efficient Transformers in Reinforcement Learning using Actor- Learner Distillation. In International Conference on Learning Representations, October 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Mohammad Reza Samsami, Artem Zholus, Janarthanan Rajendran, and Sarath Chandar. Mastering Memory Tasks with World Models. In International Conference on Learning Representations, 2024.
Simo Sarkka and Angel F. Fernaae-e Temporal Parallelization of Bayesian Smoothers. IEEE Transactions on Automatic Control, 66(1):299 Jamuary 2021.
19

---

Published in Transactions on Machine Learning Research (02/2025)
J. Schmidhuber. Curious Model-Building Control Systems. In IEEE International Joint Conference Newral Networks, pages 1458-1463 vol.2, 1991. IEFE.
Jurgen Schmidhuber. Networks Adjusting Networks. In Distributed Adeptive Newral Information Processing, pages 1990.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXie 1rxie:1777.0634/ 707.06347, August 2017.
Ginalber Luiz de Oliveira Serra. Kalman Filters Theory for Advanced Applications. February 2018.
Vaisakh Shaj, Philipp Becker, Dieter Büchler, Harit Pandya, Niels van Duijkeren, C. James Taylor, Marc Recurrent Kalman Networks For Forward and Inverse
Hanheide, and Gerhard Neumann. Dynamics Learning. In Conference on Robot Learning, volume 155, pages 765-781. PMLR, November 2021a.
Vaisakh Shaj, Dieter Büchler, Rohit Sonker, Philipp Becker, and Gerhard Neumann. Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios. In International Conference on Learning Representations, October 2021b.
Vaisakh Shaj, Saleh Gholam Zadeh, Ozan Demir, Luiz Ricardo Douat, and Gerhard Neumann. Multi Time Scale World Models. In Advances in Newral Information Processing Systems, November 2023.
Jimmy T. H. Smith, Andrew Warrington, and Scott Linderman. Simplified State Space Layers for Sequence Modeling. In International Conference on Learning Representations, 2023.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction, volume 7. MIT Press, 2018.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. control: Software and Tasks for Continuous Control. Software Impacts, 6:100022, 2020.
Claudio Urrea and Rayko Agramonte. Kalman Filter: Historical Overview and Review of Its Use in Robotics 60 Years After Its Creation. Journal of Sensors, 2021(1), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Fwalds, Petko Georgiev, and hers... Grandmaster Level in StarCraft. 1l using Multi-Agent Reinforcement Learning. Nature, 575(7782))350 35443 2019.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Daan Wierstra, Alexander Foerster, Jan Peters, and Jürgen Schmidhuber. Solving Deep Memory POMDPs with Recurrent Policy Gradients. In Artificial Neural Networks, pages 697 706, 2007. Springer Berlin Heidelberg.
Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning. In International Conference on Robotics and Automation, 2017.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. VariBAD: Variational Bayes-Adaptive Deep RL via Meta-Learnin Journal of Machine Learning Research, 22(29)):1-39, 2021.
20

---

Published in Transactions on Machine Learning Research (02/2025)
A Associativity of Masked Associative Operators
Let a, b, E E refer to clements in the space of the MAO à as in Definition 1, with ā = (a, ma), b = (b, m²), ē= (c, me). We show that if the sequence {mai my; meh, is a right-padding mask, that is: ma = = my =
m. = 1, and my =1 = me = 1, then it holds that (a ē = ā o (b Z)2 i.e., the MAO is associative. Similar to the proof in Lu et al. (2023) we consider all possible values for {mas my: moj.
Case 1: my 1 and m. The binary masks of and e are on, so b ē b, a and à a. Then,
(8)
(9)
Case 2: my and me 1. The binary mask of b if off while that of is on, so b=c= -b, then:
(10)
(11)
Case 3: m, 0 and me = 0. No mask is applied, then the MAO is equivalent to the underlying operator which is associative by Definition 1.
Note the case my = and m. =0 violates associativity, but it is impossible under our initial assumption of right-padding mask sequence {m, ma; MeJ.
B Implementation Details
In this section we provide details of various components of the RAC architecture and the specific implemen- fations of history encoders. All methods are implemented in common codebase written in the Pytorch framework (Paszke et al., 2019).
Embedder. We embed the concatenated observation-action history with simple linear layer mapping from the combined observation-action dimension to the embedding dimension E.
Soft Actor-Critic. We use a standard SAC implementation with optional automatic entropy tuning (Haarnoja al., 2019). For discrete act spaces, we use the discrete version of SAC by (Christodoulou, 2019) and one-hot encode the actions.
VSSM, VSSM+KF & vSSM+KF-u. These methods share similar implementation, with an input linear layer, linear recurrence and an output linear layer. vSSM is equivalent to only using the "Predict" block from the KF layer, while vSSM+KF-u removes the input signal ug. For all methods, we discretize the SSM using the zero-order hold method and a Icarnable scalar stop size A. In practice we use an auxiliary learnable parameter Å and define A = softplus (Ā) to ensure positive step size. as similarly done in Mamba. We initialize Á with a negative value such that after passing through the softplus and after ZOH discretization, the SSM is initialized with eigenvalues close to (i.e., slow decay of state information over time).
Mamba. Standard Mamba. model from Gu and Dao (2023). We use reference open-source implementation" and modify the parallel scan to use the associated MAO.
GRU. Standard implementation included in Pytorch.
VTransformer. Default implementation of a causal transformer encoder from Pytorch. We additionally inchude sinusoidal positional encoding, as done prior work using transformers for RL (Ni et al., 2023).
21

---

Published in Transactions on Machine Learning Research (02/2025)
C Hyperparameters
Table 1: Hyperparameters used for Section 5. For the Mamba parameters, we use the notation from the code by Gu and Dao (2023) and select parameters to match effective state size N 128. GRU and vTransformer use default parameters from Pytorch unless noted otherwise.
Parameter
BrstArm
DMC
POPGym
Training
Buffer size
Adam learning rate
3e-4
Env. steps
500K
1M
Batch size
64
32
Update-to-data (UTD) ratio
0.25
1.0
# Eval episodes
100
16
RAC
Embedding size (E)
16
Latent size (N)
128
Activations
ReLU
Context length
256
64
Actor MLP
[128]
[256, 256
Critic MLP
[256]
256, 256
SAC
Discount factor
0.99
Entropy temp.
0.1
Auto
Target entropy (continuous)
N/A
(dim((A)
Target entropy (discrete)
N/A
0.7 log (A))
History Encoders (common)
Latent size
128
# layers
VSSM, VSSM+KF & vSSM+KF-u
A init
-7 HiPPO (diagonal)
A init
B init
Σ init
Inital state belief
OO,1
RMSNorm output?
No
Yes
No
Mamba
A init d model (embedding size)
HiРPO (diagonal)
16
d state (per-channel hidden size)
Expand factor E.
Size of projection
1D Conv kernel size
4
vTransformer
# heads Feedlorward size
128
256
WW use lower value of -0.35 log (1/dim(A)) in the KineSweeper environment from POPGym, as the default value resulted in divergence during training.
22

---

Published in Transactions on Machine Learning Research (02/2025)
D
Best Arm Identification Training Curves
BestArm cost=0.0
BestArm cost=0.01
BestArm cost=0.05
In- In-distribution Norm. avg. retuun
0.5
0.0
Out-of- distribbtii Norm. avg, return
0.5
0.0
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
VSSM+KF
VSSM
GRU
Memoryless
VSSM+KF-u
Mamba
vTransformer
Oracle
Figure 11: Normalized average return over 100 episodes in and out of distribution, for increasing costs. We report the mean and standard error over 5 random sceds.
BestArm- cost=0.0
BestArm cost=0001
BestArm cost=0005
In-distribution Norm. avg. return
102
10a
10a
Out-of Out-of-distribuee distribution Norm. avg. rehurn
10a
10a
10a
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
vSSM+KF
vSSM
GRU
Memoryless
VS
Mamba
vTransformer
Oracle
Figure 12: Average (log) episode length over 100 episodes in and out of distribution, for increasing costs. We report the mean and standard error over 5 random seeds.
23

---

Published in Transactions on Machine Learning Research (02/2025)
E
KF Layer Design Ablation
0.8
1.00
Norm. IQM Return
Fraction of runs
0.6
0.75
with score >
0.4
0.50
0.2
0.25
0.0
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Env. steps (millions)
Normalized score (T)
cartpole-balance
cartpole-swingup
cheetah-run
Avg. Return
750
500
250
500
250
0
0
finger-spin
pendulum-swingup
point-mass eeasy
Avg. Return
500
500
500
0
0
reacher-easy
walker-rum
walker-walk
Avg. Return
500
250
500
250
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
output features: mean and variance; selective process noise? yes
output features: mean and variance; selective process noise? no
output features: mean; selective process noise? yes
output features: mean; selective process noise? no
Figure 13: Ablation on design considerations for KF layers. (Top) Aggregated performance in noisy DMC benchmark (9 tasks) with 95% bootstrap confidence intervals over five random scods. (Top-Left) Inter-quartile mean returns normalized by the score of Oracle. (Top-Right) Performance profile after 1M environment steps. (Bottom) Training curves. We show mean and standard error over five random seeds. Based on these results, our final design for the KF layer uses only the posterior mean state as the output feature and a time-invariant process noise.
24

---

Published in Transactions on Machine Learning Research (02/2025)
F
DMC Training Curves
cartpole- cartpole-balance
sartpole-swingup
cheetah-run
Avg. Return
1000
500
500
500
0
finger-spin
pendulum-swingup
point-mass casy
1000
Avg. Return
500
500
0
0
reacher-casy
walker-run
walker-wald
Avg. Return
1000
1000
500
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
vSSM+KF
vSSM
GRU
Memoryless
VSSM*KF-u
Mamba
vTransf ormer
Oracle
Figure 14: Training curves for the noisy DMC benchmark. We show mean and standard error over five random seeds. For all lasks, we add zero-mean Gaussian noise to the observations with scale of 0.3, except the pendulum swinggg and point-mass where the scale is 0.1.
25

---

Published in Transactions on Machine Learning Research (02/2025)
G
DMC Noise Ablation
Obs noise Ox
Obs noise 0.5x
Obs noise 1x
1000
finger-spin Retur
cheetah-run Return
1000
walker-run
1000
Return
pendulum
1000
Return
1000
reacher
Return
point- mass Return
1000
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
VSSM+KF
Oracle
VSSM*KF-U
Mamba
vTransformer
Figure 15: Training curves in six environments from the DMC benchmark with increasing levels of noise. We
show mean and standard error over five random seeds (ten for pendulum). The base noise scale for all tasks is 0.3, except the pendulum-svingup and point-mass environments where the scale
DMC Comparison to Model-Based Approaches
In Figure 16, we compare performance between VSSM, vSSM+KF and the following model-based baselines reported in Becker et al. (2024)8:
Kalmamba. Uses a Mamba (Gu and Dao, 2023) backbone that outputs the SSM matrices, which are then used within the VRKN architecture proposed in Becker and Neumann (2022). The representation is trained end-to-end in variational loss to produce plausible dynamic predictions. The learned representation is then frozen and used within SAC policy optimizer.
8The experimental data was provided by the authors on personal communication.
26

---

Published in Transactions on Machine Learning Research (02/2025)
RSSM+SAC.
The Recurrent SSM as proposed in Hafner et al. (2019). Similarly, the representation is trained on variational loss for reconstruction of observations, while the policy is trained with SAC using the learned latent representation. The gradients from SAC are not used to update the representation.
VRKN+SAC The Variational Recurrent Kalman Network (Becker and Neumann, 2022). Similarly to RSSM+ and Kalmamba, the representation trained with variational loss for reconstruction while the policy uses the learned representation for control.
SAC. Equivalent to the Memoryless baseline; SAC agent with no memory.
The benchmark includes both observation and action noise with a standard deviation of a 0.3.
cartpolo-swingup
Palker-walk
walker-run
quadruped-wai walk
1000
Avg. Return
750
500
250
0
0
2
0
Env. steps (x IM)
Env. steps (xIM)
Env. steps (xIM)
Env. steps (x1M)
KalMamba
VRKN+SAC
SAC
vSSM+KF - RSSM+SAC
Figure 16: Comparison of performance in DMC environments with observation and action noise. For the mosel-based baselines, we report the final performance after 1M environment steps, as reported in Becker et al. (2024).
From Figure we observe that in three out of four tasks, vSSM+KF close or matches the asymptotic performance of the best model-based baseline, albeit with less sample-efficiency These results highlight that good performance can be achieved in these tasks without the representation learning objectives from model-based approaches. In quadruped-walk we found that probabilistic filtering hurts performance, given the performance difference between vSSM*KF and VSSM. We hypothesize that to handle the larger observation space in quadruped-walk (~ 4x larger than walker-run), vSSK+KF would require further hyperparameter tuning and potential regularization techniques we do not explore in this work. Moreover, the sample-efficient of the architecture could further improved using recent developments such as higher update-to-data to-data ratios and network resets (D'Oro al., 2022).
27

---

Published in Transactions on Machine Learning Research (02/2025)
POPGym Training Curves
AutoencodeEasy
CocalRecall
HigherLowerEasy
0.5
MMER
00.55
0.0
0.50
Easyy
MulbaarmdDaandttaasyy
MultiarmedBanditlllan
MMER
0.5
0.5
0.0
0.0
Posiiionsii CartPoleHard
Noisy PositionOnly PaaiiiionOOlyPPPeuullmnHaaaa PendulumHard
Repeat SresatFFiss First Easy
MMER
0.75
0.50
0.25
0
PedpeatiirstMedium
Repeat Easy
RepeatProviousMedinn
MMER
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Env. steps (millions)
Env. steps (millions)
Env. steps (millions)
vSSM+KF
Mamba
vTransformer
Oracle
vSSM+KF-u
GRU
Memoryless
Best POPGym
vSSM
Figure 17: POPGym training curves. We show mean and standard error over five random sceds.
28

---

Published in Transactions on Machine Learning Research (02/2025)
J
POPGym Scores
Busy
CountRecallEasy
IowherLowerrasy
MineSweeperFasy
MultiarmedBanddtaaaay
MultiarmedBandiitHardd
VoosyPPottiooOOnlyyatPPlllllll
RepeatFFist Medium
Repeat TreviousEasy
Repeat RepeatPrevionsMediim
0.0
0.5
1.0
0.0
0.5 MMER
1.0
0.0
0.5 MMER
1.0
MMER
vSSM+KF
Mamba
vTransformer Memoryless
Dracle
VSSM+KF-U
GRU
Best POPGym
VSSM
Figure 18: POPGym final MMER after 1M training steps. We show mean and standard error over five random seeds.
29

---

Published in Transactions on Machine Learning Research (02/2025)
Table Scores on POPGym tasks after 1M environment steps. For each environment, we report the MMER mean and standard error over random seeds after 1M steps of training. The MMER is calculated from 16 test episodes. For reference, we include the best MMER score reported by Morad al. (2023) (mean and standard deviation over three random scods). We bold the highest score(s) per environment obtained by sequence model.
EutoencodeEasy
CountRecallEasy
HigherL.owerBasy
MineSweeperEasy
VS
0.299 + 0.012
0.983 + 0.002
0.588 + 0.002
0.818 + 0.014
vSSM+KF-u
0.247 + 0.036
0.978 + 0.001
0.581 + 0.004
0.864 + 0.020
VSSM
- 0.320 + 0.006
0.984 + 0.002
0.592 + 0.003
-0.307 = 0.012
Mamba
0.335 0.023
0.982 + 0.002
0.580 + 0.003
0.783 + 0.039
GRU
-0.317 0.025
0.984 = 0.001
0.586 + 0.002
0.916 + 0.034
vTransformer
0..19 + 0.065
0.982 = 0.001
0.590 + 0.001
0.676 = 0.031
Oracle
-0.420 + 0.008
0.984 = 0.002
0 0.003
1.000 = 0.000
Memoryless
-0 -0.463 + 0.001
0.887 = 0.001
0.571 + 0.004
-0.382 = 0.008
Best. POPGym
-0.283 0.029
0.509 + 0.062
0.529 + 0.002
0.693 + 0.009
Bandit
BanditHard
NoikyCarPPoleHard
DoisyPendulumHare
vSSM+KF
0.766 0.005
0.541 + 0.040
0.535 0.023
0.677 = 0.003
vSSM+KF-u
0.771 + 0.019
0.579 + 0.032
0.531 0.007
0.675 + 0.003
vSSM
0.612 = 0.013
0.501 + 0.025
0.528 + 0.013
0.639 = 0.004
Mamba
0.764 0.011
0.608 0.043
0.516 = 0.010
0.658 = 0.009
GRU
0.763 0.012
0.705 + 0.017
0.486 + 0.010
0.701 0.001
vTransformer
0.580 + 0.030
0.384 0.049
0.454 + 0.009
0.604 + 0.004
Oracle
0.889 = 0.005
0.892 + 0.006
1.000 + 0.000
0.946 = 0.001
Memoryless
0.324 + 0.013
0.399 + 0.031
0.225 + 0.003
0.406 + 0.012
Best POPGym
0.631 + 0.014
0.574 0.049
0.404 + 0.005
0.657 + 0.002
RepeatFirstEasy
RepeatFirstMMddum
Repeattrreioossass
Repeat|PrevvoosMMeddum
vSSM+KF
1.000 + 0.000
0.726 + 0.127
1.000 + 0.000
-0.423 + 0.006
VSSM+KF-u
1.000 + 0.000
0.607 + 0.186
1.000 + 0.000
0..29 = 0.008
vSSM
0.989 + 0.009
0.495 + 0.034
1.000 + 0.000
0..200 + 0.008
Mamba
1.000 + 0.000
0.575 0.160
0.993 + 0.001
-0.441 0.005
GRU
1.000 + 0.000
1.000 + 0.000
1.000 0.000
00.440 0.004
vTransformer
1.000 + 0.000
1.000 + 0.000
1.000 + 0.000
0.426 + 0.006
Dracle
1.000 + 0.000
1.000 + 0.000
1.000 + 0.000
1.000 - 0.000
Memoryless
0.093 + 0.085
0.100 +: 0.061
-0.434 + 0.013
-0.450 + 0.007
Best POPGym
1.000 + 0.000
1.000 + 0.000
1.000 + 0.000
0.789 + 0.288
30

---

Published in Transactions on Machine Learning Research (02/2025)
K
POPGym Ablation
MineSweeperEasy
BanditHard
NoisyPenduuumHard
RepeatFinstMedhum
1.0
1.0
0.6
0.50
0.5
MMER
0.5
0.25
0.4
0.0
0.0
0.5
0.00
-0.5
0.2
0
1
0
Env. steps (millions)
Env. steps (millions)
Env.
steps (millions)
Env.
steps (millions)
N = 64, L =1
N = 128, =
N = 256, =1
N = 256, = 3
N 64, 2
128, =
256, = Best POPGym
N 64, =3
128, =
Figure 19: POPGym training curves for vSSM+KF ablation over latent state size N and number of KF layers L. We show mean and standard error over five random seeds. For this experiment, vSSM+KF uses an RMSNorm output block to ensure stability for 1.
31


Reviews:
[{"review_id": "sl0EsyRdXv", "reviewer": "Reviewer_FQjn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Your observations about the paper's strengths and weaknesses provide important insights that could significantly enhance the depth and impact of the research. Here’s a structured feedback that could be included in a review for the authors:\n\n### Strengths\n\n1. **Comprehensive Evaluation:**\n   The paper's extensive experimental evaluations underscore its robustness, particularly highlighted in scenarios like the AI chatbot case. This extensive testing across different scenarios not only demonstrates the effectiveness of the proposed methods but also helps validate the approach under various conditions.\n\n2. **Scalability and Efficiency:**\n   The implementation of Kalman filter layers to process sequential data efficiently is a major strength. The ability to scale logarithmically with the sequence length could be pivotal for real-time applications and handling large datasets, which is crucial in modern AI systems.\n\n### Weaknesses\n\n1. **Comparison with KalMamba:**\n   The paper introduces a model-free version of what seems to be conceptually similar to KalMamba, yet lacks a clear rationale or motivation for preferring a model-free approach over a model-based one. A comparative analysis between these two methodologies could clarify the benefits or trade-offs involved, particularly concerning performance impacts. It would be helpful to discuss situations where a model-free approach may outperform model-based systems and vice versa.\n\n2. **Lack of Comprehensive Benchmarks:**\n   The absence of direct comparisons with existing methods, both deterministic SSMs like CURL and probabilistic SSMs like DreamerV3, is a significant limitation. These comparisons are essential to position the new method within the landscape of existing research, providing a clearer understanding of its advantages or potential drawbacks.\n\n3. **Simplistic Task Environments:**\n   The noisy DMC tasks used in the experiments, characterized merely by true states plus Gaussian noise, may not adequately challenge the denoising capabilities of the proposed method. This setup might oversimplify the problem, making it easier for the model to perform well without truly demonstrating its effectiveness in more complex, real-world-like scenarios. It is suggested to utilize image-based observations with noisy backgrounds, which better mirror the challenges present in real-world applications, to test the robustness and utility of the proposed model under more demanding conditions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper discusses the limitations of current reinforcement learning architectures, such as recurrent neural networks, deterministic state-space models, and transformers, in handling environments with partial observability due to their lack of mechanisms to manage uncertainty in hidden state representations. To address this, the authors propose integrating a Kalman filter layer into model-free architectures. This Kalman filter layer conducts closed-form Gaussian inference in linear state-space models and can be trained end-to-end to maximize returns. Unlike traditional sequence models, the Kalman filter layer uses a parallel scan for processing, which enhances scalability by scaling logarithmically with sequence length. By incorporating explicit mechanisms for probabilistic filtering, the Kalman filter layers can directly replace other recurrent layers in typical model-free setups, offering significant advantages in scenarios where understanding uncertainty is crucial for decision-making. The effectiveness of this approach is demonstrated through experiments in various tasks with partial observability, where Kalman filter layers outperform other stateful models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- **Inclusion of Model-Based Comparisons:**\n  Adding a discussion about why a model-free version is necessary alongside or instead of a model-based system like KalMamba would provide clarity. Including scenarios where the model-free approach provides distinct advantages would strengthen the paper's contribution.\n\n- **Expanded Benchmarking:**\n  Broadening the scope of benchmark comparisons to include a range of state-space models would provide a more comprehensive understanding of where the proposed method stands relative to existing techniques.\n\n- **Enhanced Realism in Experimental Design:**\n  Employing more complex and realistic task environments for testing, such as DMC tasks with image-based observations and variable backgrounds, would better assess the method’s practical efficacy and denoising capabilities.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4A7mGn7BGk", "reviewer": "Reviewer_CzkF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper integrates a Kalman filtering-type algorithm with model-free reinforcement learning, potentially opening new avenues in RL applications.\n2.\tIt contributes new insights into the application of these methods within the context of model-free RL.\n3.\tThe use of the Deep Mind Control (DMC) suite supports an extensive experimental setup that provides a broad evaluation across multiple scenarios, showcasing the adaptability and potential of the proposed methods.\n\n\nMajor Weaknesses:\n\nPaper Organization and Placement of Information: The organization of the manuscript requires improvement. Essential elements such as the paper's objectives and main contributions are not positioned effectively for clear comprehension. For instance, critical statements like, \"In this paper, we propose a history encoder implemented via Kalman filtering layers that perform simple probabilistic inference on the latent state,\" are unexpectedly located in the background section. Such a pivotal statement should ideally be highlighted either at the end of the introduction or as a conclusive remark in the theoretical background to guide the reader smoothly into the core of the paper.\n\nRelated Work and Literature Review:\nThe related work section of the manuscript is notably brief, spanning only about 1.3 pages in a 28-page document. While it primarily focuses on literature from the machine learning community, featuring contributions from well-known conferences like NeurIPS and ICML, it significantly overlooks the rich and longstanding contributions from other disciplines that have extensively engaged with the Kalman Filter, such as signal processing, control, and information fusion.\nThe discussion revisits earlier studies on Deterministic SSMs, Probabilistic SSMs, and Inference in Linear SSMs but falls short of critically evaluating the underlying assumptions of these works or providing a thorough perspective on these topics.\nThis section could be greatly enhanced by broadening its scope to include a wider range of literature, especially emphasizing diverse applications and developments related to DNN implementations of the Kalman Filter. Such an expansion would not only offer a more comprehensive and informative background but also better reflect the interdisciplinary impact and breadth of current research in this field. Moreover, a more inclusive literature review would set the stage for a more solid and informative empirical benchmark that effectively compares with the state of the art.\n\nBias Towards Prior Work:\nThe manuscript demonstrates a considerable bias towards the work of Becker, repeatedly referencing it to the exclusion of other significant contributions in the field. This focus suggests a narrow design choice and a lack of familiarity with a broader range of relevant literature. While building on previous research is standard practice, the extent of reliance on Becker’s work in a Transactions paper of this magnitude raises concerns. Such reliance should be balanced with a broader integration of diverse studies to ensure a comprehensive approach that enriches the paper’s contributions and relevance.\n\nSelf-Contained Scope:\nFor a contribution to be valuable in a high-caliber publication, it must be both comprehensive and self-contained. The current manuscript's approach limits its scope and diminishes its potential to serve as a definitive reference on the topic. Although I am familiar with Becker's work, this narrow focus might alienate readers who are not well-versed in these particular studies, consequently narrowing the paper’s appeal.\n\nTo enhance its accessibility and relevance, the manuscript should strive for a more autonomous exposition that robustly supports its findings and arguments without excessive reliance on prior work—unless such work is indeed foundational and universally recognized within the field.\n\nTheoretical Modeling Clarity and Evaluation of Novelty:\nTo robustly evaluate whether a solution is truly novel or state-of-the-art, a clear understanding of all model assumptions is essential, including any simplifying assumptions made for tractability. These can significantly narrow the scope of problems for which the algorithm is effective. It is crucial that the manuscript distinctly outlines these assumptions, allowing for a precise assessment of the algorithm’s applicability and limitations.\n\nFurthermore, to confirm if an algorithm represents a genuine advancement in the field, it must be benchmarked against not only minor variations of the same type but also against the known state-of-the-art for the test case or relevant theoretical guarantees. This broader comparison is vital to accurately gauge the true impact and innovation of the proposed solution.\n\nIsolating the problem formulation from the solution design choices will help to correctly identify the relevant benchmarks and theoretical bounds for performance and clarify the true significance of the paper and its contribution to the broader community. Additionally, a true decoupling of necessary model assumptions from design choices will help mitigate bias in the authors' approach, potentially leading to more innovative and effective design choices.\n\nThe manuscript currently fails to adequately separate modeling or model assumptions from algorithms and inference techniques, which undermines its scholarly rigor. A robust problem formulation, distinct from the solution technique, is essential in the background section to ensure a comprehensive and unbiased exploration of solutions.\n\nThe paper would significantly benefit from a clear initial presentation of problem formulation, explicitly stating the optimization objectives and modeling assumptions—particularly detailing how the physical world is modeled and how agents interact within it. The model statement should remain independent of the class of algorithms used.\n\nBy adhering to these guidelines, the manuscript can enhance its scholarly rigor and provide a clearer, more impactful contribution to the field. This structured approach will ensure the research is comprehensively evaluated for both its methodological soundness and its innovative value.\n\nExample from the paper:\nThe manuscript does not effectively differentiate between foundational model assumptions, which describe the world or class of problems being modeled, and simplifying assumptions made for tractability. Understanding the implications of these choices is crucial. Additionally, such simplifying assumptions, including the use of structured SSMs and diagonal matrices discussed in subsection 3.4’s Kalman Filter analysis, should be explicitly noted in the problem formulation section. These choices potentially narrow the class of problems that can be addressed, and if these assumptions do not impact the class of problems or expected performance, it should be clearly justified.\n\nEvaluation of Novelty and Design Choices:\nThe paper asserts a novel implementation of a history encoder via Kalman filtering layers for model-free RL, claiming, \"To the best of our knowledge, this is the first work that empirically evaluates the performance of such probabilistic inference layers in a model-free RL architecture.\" While the application in model-free RL is appreciated, the novelty of this approach raises several critical questions:\n\n1. Optimality of Design Choice: The paper leverages a specific type of Kalman Filter algorithm, heavily based on Becker’s work without apparent modification or improvement. It is essential to discuss why this particular algorithm was chosen over other learned Kalman Filter algorithms that might also fulfill these roles. The omission of a broad range of pertinent literature in this context raises concerns about the comprehensiveness of the empirical evaluation.\n\n2. Trade-offs in Simplifying Assumptions: What are the performance trade-offs involved in restricting the Kalman Filter to a linear type? Additionally, what complexities are reduced by these assumptions, and how do these choices impact the overall effectiveness of the solution?\n\n3. Theoretical and Empirical Validation: Is the assumption that a Kalman Filter type algorithm would be an optimal history encoder under all circumstances valid? The manuscript would benefit from more robust theoretical arguments and empirical evidence supporting this premise.\n\n4. State-of-the-Art Comparison: Does the final algorithm as implemented stand up as state-of-the-art for the test cases provided in the empirical sections, or are there gaps in performance when compared to existing solutions?\n\nWhile focusing on a specific algorithm from the outset and exploring its effectiveness is acceptable, such intent should be explicitly stated. This approach sets realistic expectations about the scope and novelty of the research but may limit perceived innovation unless significant enhancements or insights are demonstrated.\n\n\nIn-Depth Analysis Suggestion:\nThe proposed history encoder relies on a Kalman Filter, which is known to be optimal for linear Gaussian models. To better understand how your implementation compares to classical solutions, such as in LQG control scenarios where the Kalman Filter is the optimal tracker and LQR is the optimal controller, testing your methodology in these settings could be insightful.\n\nGiven the high-level architecture agreement that a Kalman Filter is suitable and acknowledging that a linear Kalman Filter is a reasonable choice, exploring how alternative learnable Kalman Filters could integrate into or enhance the proposed design would be valuable. A comparative analysis of these implementations would help clarify whether the design choices made, including simplifying assumptions for better computational efficiency, do not unduly compromise the solution's effectiveness.\nThese discussions and analyses are crucial for validating the design choices made and for ensuring that the proposed solution not only advances the field by integrating novel approaches but also stands rigorous evaluation against established methodologies.\n\nExperiments Evaluation:\nI commend the effort and resources dedicated to conducting the experiments presented in this paper, as they significantly contribute to its richness. However, I hold several reservations about the experimental design and its implications:\n\n1. Relevance of RL Approaches: The paper predominantly focuses on reinforcement learning (RL) based strategies. However, not all benchmarks necessitate a full RL approach, and this narrow focus may omit crucial opportunities to benchmark against more diverse and possibly superior state-of-the-art methodologies. It would be beneficial if the paper expanded its scope to include broader problem formulations. This would provide a more balanced view, contrasting RL solutions against the best available solutions for the specific problems addressed or exploring test cases where assumptions integral to other methods are challenged.\n\n\n2. Evaluation Scope and Comparisons: The statement, “we evaluate the RAC architecture under different history encoders,” suggests a limited exploration confined to variations within a predetermined design framework, specifically focused on a particular type of history encoders. For each test case examined, it is crucial to clarify the state of the art and how it compares to your architecture, irrespective of the high-level design. Where applicable, I am interested in seeing a performance gap analysis relative to an oracle, which could provide insights into the maximum potential improvement over current results.\n\n3. Benchmarking Integrity: Concerning the benchmark algorithms used for comparative analysis, there is an underlying assumption that these are optimally trained and tuned. It is imperative to confirm whether these benchmarks are indeed reflective of the best possible configurations as evidenced by previously published results or through the use of pre-trained models. This validation is essential to ensure that the comparisons made are fair and that the conclusions drawn about the efficacy of the proposed architecture are sound.\n\nBy addressing these points, the paper can enhance its contribution to the field, offering a more comprehensive evaluation that not only tests the proposed methods against current best practices but also provides a clear benchmark for future innovations. This approach will not only solidify the validity of the experimental results but also potentially elevate the perceived novelty and applicability of the research.\n\nRegarding the MAB Experiment 5.2:\nThe significant gap to the oracle in the Multi-Armed Bandit (MAB) experiment raises questions about the chosen methods and their effectiveness. Given that bandit problems are classically addressed with state-of-the-art solutions that do not necessitate the use of neural networks or extensive reinforcement learning, but rather simpler bandit policies, this discrepancy merits closer examination.\nI suggest conducting an additional experiment where the setting involves a noisy observed state that aligns more closely with typical bandit scenarios, avoiding the need for an embedding from a textual user recommendation. This approach would allow for a more controlled assessment of performance and enable a direct comparison with state-of-the-art solutions tailored to this specific problem type. Such an experiment would not only test the robustness of the proposed methods under more typical conditions but also provide a clearer benchmark against established techniques.\n\nFurthermore, it would be beneficial to include a discussion on where similar experiments have been conducted, outlining the results obtained in those studies and how they compare to the findings presented here. Understanding these comparisons is crucial for contextualizing the results and assessing the true novelty and impact of the experimental approach used in this study.\nBy addressing these points, the paper could greatly enhance its empirical credibility and provide a more solid foundation for the claims of innovation and effectiveness in tackling MAB problems.\n\nRegarding the Control Experiments 5.3:\nControl theory is a thoroughly researched area, yet the manuscript lacks references to pertinent control literature. For a more comprehensive evaluation, it would be beneficial to know where similar experiments have been conducted and how those results compare to the findings presented here. When possible, comparing these results to the optimal benchmarks is essential to ascertain the effectiveness of the proposed methods.\n\nThe manuscript mentions the use of the Deep Mind Control (DMC) suite but fails to provide detailed context about this reference. For the paper to be as self-contained and accessible as possible, more information is needed. Specifically, it should clarify whether the observations involve noisy visual frames or noisy state variables. Additionally, an understanding of the underlying physical model used in these experiments would greatly aid in appreciating the constraints and limitations of the modeling approach employed.\n\nMinor Weaknesses:\n\n1.\tWriting Consistency: The paper exhibits inconsistencies in the presentation of acronyms, with some capitalized and others not, such as \"reinforcement learning (RL),\" \"state-space models (SSMs),\" \"Kalman filter (KF) layer,\" and \"Markov Decision Process (MDP).\" To improve readability and maintain a professional tone, I suggest standardizing the format in which acronyms are introduced and utilized throughout the document.\n\n2.\tMathematical Rigor: The manuscript describes dynamical systems using differential equations, which lacks rigor in the context of the extensive literature that employs stochastic differential equations for such descriptions. Considering there is no further reference to continuous-time systems, it may be more appropriate to omit this less rigorous description and commence directly with a discussion on discrete-time systems.\n\n3.\tOverly Specific Methodological Focus: Subsection 3.1, titled \"RL in POMDP,\" seems narrowly focused on specific reinforcement learning methods. This could limit the paper's appeal and applicability, suggesting a need for a broader examination of relevant techniques within this context.\n\n4.\tTechnical Clarifications: Regarding the statement “vSSM+KF. Probabilistic SSM via the KF layers described in Figure 2, which is equivalent to adding Kalman filtering on top of vSSM,” it would be more accurate to specify that this integration involves adding an update step, which clarifies the actual technical process involved.\n\n5.\tConsistency in Terminology: The paper uses the terms \"running example\" and \"motivating example\" interchangeably. For clarity and to enhance the paper's structure, sticking to one term and consistently using it throughout would be beneficial. Additionally, labeling this example clearly would aid in precise references throughout the discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a history encoder utilizing Kalman filtering layers for simple probabilistic inference on the latent state within a model-free RL architecture. This work is reported to be the first empirical evaluation of such probabilistic inference layers. The authors explore the integration of model-free RL techniques with structured state space models (SSMs), combining deterministic and probabilistic SSMs with inference techniques. They investigate the performance implications of these layers across various benchmark settings, thereby expanding the traditional applications of RL and Kalman filters.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As part of the review process, I have carefully evaluated the manuscript and appreciate the efforts put forth by the authors. However, to enhance the quality and impact of the paper, I believe it requires a major revision that addresses several critical concerns outlined in my comments. Addressing these issues will not only strengthen the manuscript but also align it more closely with the standards expected in high-caliber publications.\n\nFor ease of reference and to ensure a comprehensive revision, I have summarized the main issues that need attention. I expect that the concerns raised will be thoughtfully addressed, and the revisions will incorporate substantial changes in the following areas:\n\nRequested Changes:\n1.\tImprove Paper Organization:\nReorganize to clearly present the paper's objectives and main contributions in appropriate sections, such as the introduction or conclusion.\n\n2.\tEnhance Literature Review:\nBroaden the scope of the related work section to include significant contributions from disciplines like signal processing and control, not just machine learning.\n\n3.\tReduce Bias Towards Previous Work:\nIntegrate a wider range of studies and reduce the heavy reliance on Becker’s work to ensure a balanced and comprehensive approach.\n\n4.\tClarify Theoretical Modeling:\nClearly differentiate between foundational model assumptions and simplifying assumptions for tractability, and discuss the implications of these choices.\n\n5.\tSpecify Algorithm Justifications and Comparisons:\nJustify the choice of specific algorithms used and compare them against a broader range of state-of-the-art methods, including theoretical guarantees.\n\n6.\tValidate Experimental Design:\nExpand the scope of experiments to include alternative strategies and provide benchmarks against state-of-the-art methods to ensure robust evaluation.\n\n7.\tAddress Minor Issues:\nReview and address minor issues as noted in the feedback, such as inconsistencies in writing style and terminology", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CXpsV2YOQE", "reviewer": "Reviewer_AeJf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. The paper is easy to follow thanks to a clear structure and careful explanations.\n2. Using a kalman filter in a POMDP setting is novel and has good empirical performances.\n3. The methodoogy is thoroughly evaluated in the paper.\n\n# Weaknesses\n1. The main tool to asses the proposed methodology is using ablations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper shows that augmenting a model-free algorithm with a Kalman filter layer allows it to deal with partial observability thanks to improved uncertainty reasoning capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A comparison with a different sota methodology would allow the reader to better understand the potential of the proposed method.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Dear authors, 

The reviewers are generally satisfies with the changes made in the paper. 

However, you mentioned the comparisons to model-based approaches in the rebuttal and made some arguments about the benefits of using model-free RL. However, without verifying theoretical proof or experiments, the arguments are not convincing. Furthermore, planning with a model could help to navigate to deal with the problem of reward sparsity and how you make a trade-off between using model-based and model-free methods according to their pros and cons should be clarified. Therefore, please add those experiments and discussion in the paper to support the arguments.

I will check whether these changes are made in the final version.

Thanks,
AE
Decision: Accept (poster)