Recurrent networks, hidden states and beliefs in partially observable environments
Authors: Gaspard Lambrechts
Adrien Bolland
Damien Ernst
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning aims to learn optimal policies from interaction with environments whose dynamics are unknown. Many methods rely on the approximation of a value function to derive near-optimal policies. In partially observable environments, these functions depend on the complete sequence of observations and past actions, called the history. In this work, we show empirically that recurrent neural networks trained to approximate such value functions internally filter the posterior probability distribution of the current state given the history, called the belief. More precisely, we show that, as a recurrent neural network learns the Q-function, its hidden states become more and more correlated with the beliefs of state variables that are relevant to optimal control. This correlation is measured through their mutual information. In addition, we show that the expected return of an agent increases with the ability of its recurrent architecture to reach a high mutual information between its hidden states and the beliefs. Finally, we show that the mutual information between the hidden states and the beliefs of variables that are irrelevant for optimal control decreases through the learning process. In summary, this work shows that in its hidden states, a recurrent neural network approximating the Q-function of a partially observable environment reproduces a sufficient statistic from the history that is correlated to the relevant part of the belief for taking optimal actions.

Published in Transactions on Machine Learning Research (08/2022)
Recurrent networks, hidden states and beliefs in partially observable environments
Gaspard Lambrechts
haspartllambreehht@@hheee.... be
Montefiore Institute, University of Liège
Adrien Bolland
adrien. siteee.bblllndd@aaiieeee be
Monteftore Institude, University of Liëge
Damien Ernst
dernst@rulegee be
Montefiore Institute, University of Liëge
LTCI, Telecom Poris, Institut Polytechnique de Paris
Reviewed OpenReview: https: openrevveee. net/ ser/orrmm? id= V3HB23
Abstract
Reinforcement learning aims to learn optimal policies from interaction with enviromments whose dynamics are unknown. Many methods rely on the approximation of a value function to derive near-optimal policies. In partially observable environments, these functions de- pend on the complete sequence of observations and past actions, called the history. In this work, we show empirically that recurrent neural networks trained to approximate such value functions internally filter the posterior probability distribution of the current state given the history, called the bolief. More precisely, we show that, as a recurrent noural network learns the @-function, its hidden states become more and more correlated with the beliefs state variables that are relevant optimal control. This correlation measured through their mutual information. In addition, we show that the expected return of an agent increases with the ability of its recurrent architecture to reach high mutual information between its hidden states and the beliefs. Finally, we show that the mutual information between the hidden states and the beliefs of variables that are irrelevant for optimal control decreases through the learning process. In summary, this work shows that in its hidden states, a re- current neural network approximating the O-function of a partially observable environment reproduces a sufficient statistic from the history that is correlated to the relevant part of the belief for taking optimal actions.
1 Introduction
Latest advances in reinforcement learning (RL) rely heavily on the ability to approximate a value function (i.e., state or state-action value function). Modern RL algorithms have been shown to be able to produce approximations of the value functions of Markov decision processes (MDPs) from which high-quality policies can be derived, even the case of continuous and high-dimensional state and action spaces (Mnih al. 2015 Lillicrap et al., 2015 Mnih et al. 2016 Haarnoja et al., 2018 Hessel al., 2018). The adaptation of these techniques to partially observable MDPs (POMDPs) is not straightforward. Indeed, in such environments, the agent only receives partial observations of the underlying states of the environment. Unlike MDPs where the value functions are written as functions of the current state, in POMDPs the value functions are written as functions of the complete sequence of observations and past actions, called the history. Moreover, the value functions of a history can equivalently be written as functions of the posterior probability distribution over the current state given this history (Bertsckas) 2012). This posterior probability distribution called the belief and is said to a sufficient statistic from the history for the value functions of the POMDP, However, the computation of the belief requires to know the POMDP model and is generally intractable

---

Published in Transactions on Machine Learning Research (08/2022)
with large or continuous state spaces. For these two reasons, practical RL algorithms rely the definition of the value functions as functions of the complete history (i.e., history or history-action value function), while the definition of the value functions as functions of the belief (i.e., belief or belief-action value function) is more of thcorctical interest.
Approximating the value functions as functions of the histories requires one to use function approximators that are able to process sequences of arbitrary length. In practice, RNNs are good candidates for such approximators [Bakker] 2001; Hausknecht & Stone 2015 Heess et al. 2015). RNNs are parametric approx- imators that process sequences, time step by time step, exhibiting memory through a hidden state that is passed recurrently over time. The RNN is thus tasked with outputting the value directly from the history. We focus on the approximation of the history-action value function, or O-function, in POMDPs using a para- metric recurrent Q-learning (PRQL) algorithm. More precisely, RNNs are trained with the deep recurrent Q-network (DRQN) algorithm (Hausknecht & Stone 2015
Zhu et al. 2017).
Since we know that the belief is a sufficient statistic from the history for the Q-function of this history [Bertsekas] 2012), we investigate if RNNs, once trained, reproduce the belief filter when processing a history, This investigation is conducted in this work by studying the performance of the different agents with regard to the mutual information (MI) between their hidden states and the belief. We focus on POMDPs for which the models are known. The benchmark problems chosen are the T-Maze environments (Bakker] 2001 p and the Mountain Hike environments (Igl et al.) 2018). The first ones present a discrete state space, allowing one to compute the belief using Bayes' rule, and representing this distribution over the states in a vector whose dimension is equal to the number of distinct states. The second ones present a continuous state space, making the belief update intractable. We thus rely on particle filtering in order approximate the belief by set of states, called particles, distributed according to the belief distribution. The MI between the hidden states and the beliefs is periodically estimated during training, using the mutual information neural estimator (MINE) algorithm (Belghazi et al.) 2018). The MINE estimator is extended with the Deep Set architecture (Zahoer et al., 2017) in order to process sets particles in the case of POMDPs with continuous stale-spaces. This methodology allows one to measure the ability and lendency of recurrent architecture to reproduce the belief filter when trained to approximate the Q-function.
In (Mikulik et al. 2020), similar study is performed in the meta-learning setting. In this setting, MDP is drawn from distribution of MDPs at each episode. This problem can be equivalently modeled a particular subclass of POMDP. The authors show empirically, among others, that the hidden state of an RNN-based policy and the statistic of the optimal policy can be mapped one into the other with a low dissimilarity measure. In contrast, we consider arbitrary POMDPs and show empirically that information about the belief, a statistic known to be sufficient for the optimal control, is encoded in the hidden states.
In [Section 2 we formalise the problem of optimal control in POMDPs, we present the PRQL algorithms for deriving near-optimal policies and we explain the MINE algorithm for estimating the MI. In Section 3. the beliefs and hidden states are defined as random variables whose MI is measured. Afterwards, Section 4 displays the main results obtained for the previously mentioned POMDPs. Finally, Section 5 concludes and proposes several future works and algorithms motivated by our results.
2 Background
In SSbseccion 2.1, POMDPs are introduced, along with the belief, policy, and Q-functions associated with such decision processes. Afterwards, in [Subscction 2.2, we introduce the DRQN algorithm that is used in our experiments. This algorithm is a particular instance of the PRQL class of algorithms that allows to approximate the Q-function for deriving a near-optimal policy in a POMDP. Finally, in [Subsection 2.3, we present the MINE algorithm that is used for estimating the MI between the hidden states and beliefs in our experiments.
2.1 Partially observable Markov decision processes
In this work, the environments are modelled as POMDPs. Formally, a POMDP P is an &-tuple P (S, O.po, po, T, R, O, where S is the state space, A is the action space, and O is the observation space.
2

---

Published in Transactions on Machine Learning Research (08/2022)
The initial state distribution Po gives the probability Pu(so) of So S being the initial state of the decision process. The dynamics are described by the transition distribution T that gives the probability T(s++1 St, ar) of St+1 E & being the state resulting from action a, E A in state Se E S. The reward function R gives the immodiate reward re = R(s,, a,, S(+1) obtained after each transition. The observation distribution O gives the probability O 8,) to get observation O2 E in state 8, E S. Finally, the discount factor 7 E [0, [ gives the relative importance of future rewards.
Taking sequence of t actions (a0=-1) in the POMDP conditions its execution and provides sequence of + 1 observations (0o:)). Together, they compose the history no: (0o:: a0:-11 E Hox until time step 1, where Hox is the set of such historics. Let E H denote history of arbitrary length sampled in the POMDP, and let. H = U Ha.t denote the set of histories of arbitrary length.
A policy æ E in a POMDP is a mapping from histories to actions, where II H + is the set such mappings. A policy x² E II is said to be optimal when it maximises the expected discounted sum of future rewards starting from any history 7/o.t E Ho. at time & No
DQ Σ
n¹ E arg max
E T.P
you
Vnoi
Haus Vi No.
FETI
The history-action value function, or O-function, is defined the maximal expected discounted reward that be gathered, starting from history noa E Hox at time t No and an action a, A
xc Σ
Q a²) = max E hEll R.P
Tv
no:1 a Vnoa E Hoa, Va A, Vt No.
(2)
The Q-function is also the unique solution of the Bellman equation (Smallwood & Sondik et al. 1998 Porta al. 22044)
1973
Kaelbling
( a)
E P
r + max Q(n', a') 17, 7, An E H. Va E a'EA
where ry' = (a, and r is the immodiate reward obtained when taking action a in history n,. From equation 1 and equation it can be observed that any optimal policy satisfies
x (n) E arg max 2(1,a), An E H.
(4)
aEA
Let P(S) be the set of probability measures over the state space S. The belief b E P(S) of history E H is defined as the posterior probability distribution over the states given the history, such that b(s) = p(s
n), Ys E S (Thrun 2002). The belief filter f* is defined as the function that maps a history ? to its corresponding belief
(n) = b, An E H.
(5)
Formally, for an initial observation 17 = (o), the belief b = f (n) is defined by
(s s)
b(s) Po ds' ES
(6)
and for history 1/ 17 o') the belief = ("')) is recursively defined by
O(o' 5') s') T(s' s, b(s) ds
0'(5') Is O s') T(s' s) b(s) ds Vs E S.
where b (n). Equation observing new information (a.o)
provides way to update the belief b to &' through filter step once
8' s(b; (b:a,o').
(8)
3

---

Published in Transactions on Machine Learning Research (08/2022)
A statistic from the history is defined any function of the history. The belief is known to be sufficient statistic from the history in order to act optimally (Bertsekas) 2012). It means that the Q-function only depends on the history through the belief computed from this same history, It implies in particular that the Q-function takes the following form
217 Q() Vn E H, Va E A
(9)
where Q P(S( x A - R is called the belief-action value function, or Q-function. This function gives the maximal expected discounted reward starting from a belief b P(S) and an action a E A, where the belief = (n) results from an arbitrary history E H. Although the exact belief filter is often unknown or intractable, this factorisation of the Q-function still motivates the compression of the history a statistic related to the belief, when processing the history for predicting the Q-function.
2.2 Parametric recurrent Q-learning
We call PRQL the family of algorithms that aim at learning an approximation of the Q-function with a recurrent architecture 22. where 9 E 12 the parameter vector. These algorithms are motivated by equation 4 that shows that optimal policy can be derived from the Q-function. The strategy consists of minimising, with respect to 0, for all (1. (), the distance between the estimation Qo(n, a) of the LHS of equation (3, and the estimation of the expectation Ep|r + max.'EA Qa(1', a')) of the RHS of equation 33. This is done by using transitions (17, a, aa,, 7') sampled in the POMDP, with ny = 7 (a, In its simplest form, given such a transition, the PRQL. algorithm updates the parameters E TR of the function approximator according to
0 0 a r r+ max (0((1/aa')) 20(7,a) volo(). a'EA
(10)
This update corresponds to a gradient step in the direction that minimises, with respect to the squared distance between ()) and the target + y maxa (Qo(r',a)) considered independent of 8. It can be noted that, in practice, such algorithms introduce a truncation horizon H such that the histories generated in the POMDP have maximum length of H. From the approximation Qs. the policy aa is given by To (7) = arg max Q0(1) a). Equation (4) guarantecs the optimality of this policy if Qo = Q. Even though it will alter the performance of the algorithm, any policy can be used to sample the transitions (17 a, r,,', ry").
The function approximator Qo of PRQL algorithms should be able process inputs rj E H of arbitrary length, making RNN approximators suitable choice. Indeed, RNNs process the inputs sequentially, ex- hibiting memory through hidden states that are outputted after each time step, and processed at the next time step along with the following input. More formally, let xp:: = _xo...., xt] with i E No be an input sequence. At any step & E (O....,d (), RNNs maintain an internal memory state h& through the update function [11 and output a value yı through the output function 12 The initial state h is given by the initialization function 13
h = uo (h:h-11 Xx), Vk E No,
(11)
y& (bx)), Wk E No,
(12)
h = ig.
(13)
These networks are trained basod on backpropagation through time where gradicnts are computed a backward pass through the complete sequence via the hidden states (Werbos) 1990). The following recurrent architectures are used in the experiments: the long short-term memory (LSTM) by Hochreiter & Schmid-
huber (1997), the gated recurrent unit (GRU) by Chung et al. (2014), the bistable recurrent cell (BRC) and 2021 and the minimal gated
recurrently neuromodulated bistable recurrent cell (nBRC) by Vecoven et al.
unit (MGU) by Zhou et al. (2016).
In the experiments, we use the DRQN algorithm (Hausknocht Stone 2015 Zhu et al. 2017 to learn policies. This algorithm is PRQL algorithm that shows good convergence even for high-dimensional problems. The DRQN algorithm is detailed in Algorithm 1 Appendix B In this algorithm, for given
4

---

Published in Transactions on Machine Learning Research (08/2022)
history To. of arbitrary length t, the imputs of the RNN are x (ak-1,0k) k == =,....., and X = (0, ou):
and the output of the RNN at the last time step y: og (h (h,) E TRA gives y; = -o((oa, at), for any a, E A.
We also define the composition up : H. + Rd of equation 13 and equation [1] applied on the complete history, such that
xt); 121 1
h, (Mo:))
(14)
xi).
t= 0
2.3 Mutual information neural estimator
In this work, we are interested in establishing a recurrent function approximator reproduces the belief filter during PRQL. Formally, this is performed by estimating the MI between the beliefs and the hidden states of the RNN approximator Qo- In this subsection, we recall the concept of MI and how it can be estimated in practice.
The MI is theoretically able to measure any kind of dependency between random variables Kraskov et al. 2004j. The MI between two jointly continuous random variables X and Y is defined as
I(X;))= S p(z, 3) log p(z, v) dz dy px (x) pr (V)
(15)
where X' and y are the support of the random variables X and Y respectively, is the joint probability density function of X and Y, and px and py are the marginal probability density functions of X and Y, respectively. It is worth noting that the MI can be defined in terms of the Kulback-Leibler (KL) divergence between the joint p and the product of the marginals = px @ py, over the joint space X' y
T(X;Y) = (x 9) - p(2) log
p(2) {(2)
dz
(16)
In order to estimate the MI between random variables X and Y from a dataset {(r,, w)) we rely on the MINE algorithm [Belghazi et al.) 2018). This technique is parametric approach where neural network outputs& lower bound on the MI, that is maximised by gradient ascent. The lower bound is derived from the Donsker- Donsker-Varhadan Varhadan representation of the KT-divergence (Donsker & Varadhan 1975
DK ll () = sup E T-2+R sup
[T(2)] - log
E Long e [
(17)
where the supremum is taken over all functions T such that the two expectations are finite. The lower bound (X;; Y) on the true MI Y( is obtained by replacing T by parameterised function Tø 2 - IR with o E 2, and taking the supremum over the parameter space & of this function. If & corresponds to the parameter space of a neural network, then this lower bound can be approached by gradicnt ascent using empirical means as estimators of the expectations. The resulting procedure for estimating the MI given in Algorithm 3 in Appendix
3 Measuring the correlation between the hidden states and beliefs
In this work, we study PRQL implicitly approximates the belicf filter by reaching a high MI between the RNN's hidden states and the beliefs, that are both generated from random histories. In this section, we first explain the intuition behind this hypothesis, then we define the joint probability distribution over the hidden states and belief' that defines the MI.
As explained in Section 2 the belief filter is generally intractable. As a consequence, PRQL algorithms use approximators 2o that directly take the histories as input. In the DRQN algorithm, these histories are processed recurrently according to equation 11]), producing a new hidden state h, after each input x; = (a.-1. 0,)
h, no (a))
(18)
5

---

Published in Transactions on Machine Learning Research (08/2022)
These hidden states should thus summarise all relevant information from past imputs in order to predict the O-function at all later time steps. The belief is known to be a sufficient statistic from the history for these
predictions [9p. Moreover, the belief be is also updated recurrently, according to equation ( after each
transition (a)
be = 1(h-1; 0z)-
(19)
The parallel between equation 18] and equation 19 knowing the sufficiency of the belief (0. justifies the appropriateness of the belief filter f as the update function ue of the RNN approximator Qo. It motivates the study of the reconstruction of the belief filter by the RNN.
In practice, this is done through the measurement of the MI between the hidden state h, and the belief b, at any time step E No. Formally, for a given history length & E No, the policy Te of the learning algorithm, as defined in (Sussccttion 2.2 induces a distribution Pra (1) 1) over histories 1) E H. This conditional probability distribution zero for all history of length t' # t. Given a distribution p(t) over the length of trajectories, the joint distribution of h and b is given by
p ph, b) = Σ p(t) / p(h,b) m) Pro t) dn
20
(20)
where p(h, b 1) is Dirac distribution for h ug) and b= (n) given by equation [4] and equation respectively. In the following, estimate the MI between h and b under their joint distribution (20
4 Experiments
In this scction, the experimental protocol and environments are described and the results are given. More specifically, in [Subsection 4.1 describe the estimates that are reported in the figures. The results are reported for four different POMDPs: the T-Maze and Stochastic T-Maze in (Sbsecction 4.2 and the Mountain Hike and Varying Mountain Hike in [Subsection 4.3) Afterwards, in [Subsection 4.4] irrelevant state variables and observations are added to the decision processes, and the MI is measured separately between the hidden states and the belief of the relevant and irrelevant variables. Finally, in (Subsection 4.5) we discuss the results obtained in this section, and propose an additional protocol to study their generalisation.
4.1 Experimental protocol
As explained in SSubsection 2.2] the parameters 0 of the approximation Qo are optimised with the DRQN algorithm. After e episodes of interaction with the POMDP, the DRQN algorithm gives the policy (1) arg max Qo, a)) In the experiments, the empirical cumulative reward J(O.) of the policy no, is reported, along with the estimated MI i(0.) between the random variables and b under the dis- tribution (20) implied by Te.- Fach estimate is reported averaged over four training sessions. In addition, confidence intervals show the minimum and maximum of these estimates.
The empirical return is defined as j(0.) + E E'' y'ri, where I is the mumber of Monte Carlo rollouts, H the truncation horizon of the DRQN algorithm, and rt is the reward obtained at time step t of Monte Carlo rollout i. As far as the estimation of the MI is concerned, we sample time steps with equal probability p(t) = 1/H, E {o..... H - 1}, where H is the truncation horizon of the DRQN algorithm. The uniform distribution over time steps and the current policy mh. define the probability distribution (20) over the hidden states and beliefs. The MI estimated from samples of this distribution using the MINE estimator î(0.) (see Subsection D.1 for details). The hyperparameters of the DRQN and MINE algorithms are given in Appendix E
For POMDPs with continuous state spaces, the computation of the belief b is intractable. However, a set of state particles 5 that follows the belief distribution (r( can be sampled, using particle filtering (see Appendix C. This set of particles could be used to construct an approximation the belief in order to estimate the MI. This density estimation procodure is nonetheless unnocessary as the MINE network can directly process the set of particles by producing a connuntationnivvriian embedding of the belief using the Deep Set architecture (Zaheer et al. 2017), see Subsection D.2 for details.
6

---

Published in Transactions on Machine Learning Research (08/2022)
4.2 Deterministic and Stochastic T- T-Mazes
The T-Maze is POMDP where the agent is tasked with finding the treasure in T-shaped maze (see Figure 1). The state is given by the position of the agent in the maze and the maze layout that indicates whether the treasure lies up or down after the crossroads. The initial state determines the maze layout, and it. never changes afterwards. The initial observation made by the agent indicates the layout. Navigating in the maze provides bouncing onto wall, in which case a reward of -0.1 1 is received. Finding the treasure provides a reward of 4. Beyond the crossroads, the states are always terminal. The optimal policy thus consists of going through the maze, while remembering the initial observation order to take the correct direction at the crossroads. This POMDP is parameterised by the corridor length N and stochasticity rate A [0, 1] that gives the probability of moving in a random direction at any time step. The Deterministic T-Maze (A 0) was originally proposed in [Bakker 2001). The discount factor is = 0.98. This
00.0)
0,00
8.0
(4,00
(6,00
(2.-1)
reward, except when
Down
[C.1]
(0.0)
(1.0)
(2.0)
(3.0)
[4.0)
[2.0)
Figure 1: T-Maze state space.
POMDP is formally defined in Subsection A.2
As explained in (Subsection 2.2, the histories can be sampled with an arbitrary policy in PRQL algorithms. In practice, the DRQN algorithm uses an E-greedy stochastic policy that selects its action according the current policy with probability 1 e, and according to the exploration policy E(A) with probability Usually, the exploration policy is chosen to be the uniform distribution U(A) over the action. However, for the T-Maze, the exploration policy E(A) is tailored to this POMDP to alleviate the exploration problem, that is independent of the study of this work. The exploration policy forces one to walk through the right of the corridor with E (Rigght) = 1/2 and E(Other) = 1/6 where Other E (Up, Left, Down].
1.46
Retaum jio.)
LSTM
GRU
BRC
Return (0.)
NBRC
MGU
-1
MI ((0.) [bitt]
2.5
-1
1000
2000
3000
4000
5000
Episode e
MI 1 (00) [bit]]
Figure 2: Deterministic T-Maze (L= 50). Evolution of the return J(O.) and the MI f(O2) after e episodes
(left), and the return I(0.) with respect to the MI f(0x) (right). The maximal expected return is given by the dotted line.
On the left in Figure the expected return is shown along with the MI between the hidden states and the belief function of the number of episodes, for T-Maze of length L = 50. In order to better disambiguate between high-quality policies, the empirical return is displayed with exponential scale in the following graphs. Both the performance of the policy and the MI increase during training. We also observe that, at any given episode, RNNs that have a higher return, such as the nBRC or the BRC, correspond to cells that have a higher MI between their hidden states and the belief. Furthermore, the LSTM that struggles to achieve high return has a significantly lower MI than the other cells. Finally, we can see that the evolution of the MI and the return are correlated, which is highlighted on the right in Figure 2] Indeed, the return increases with the MI, with linear correlation coefficient of 0.8233 and rank correlatio coefficient. of

---

Published in Transactions on Machine Learning Research (08/2022)
0.6419. These correlations coefficients are also detailed for each cell separately in Appendix G be noted that no RNN with less than 5 bits of MI reaches the maximal return.
It can also
0.53
Roturn j(,)
Return j(0.)
-1-
-2
MI ( [bit]]
LSTM
5.0
GRU
BRC
-1
NBRC
-2
MGU
0.0
2000
4000
6000
8000
10000
2
5
Episode
MI I(O.) [bit]
Figure 3: Deterministic T-Maze (L= 100). Evolution of the return J(..) and the MI f(0.) after episodes (left), and the return J(O.) with respect to the MI I(Os) (right). The maximal expected return is given by the dotted linc.
In Figure 3 we can see that all previous observations also hold for a T-Maze of length L = 100. On the left, we can see that the lower the MI, the lower the return of the policy. For this length, h, addition to the LSTM, the GRU struggles to achieve the maximal return, which is reflected in the evolution of its MI that increases more slowly than for the other RNNs. is also interesting to notice that, on average, the MGU overtake the BRC in term of return after 2000 episodes, which is also the case for the MI. Here, the linear correlation coefficient between the MI and the return is 0.5347 and the rank correlation coefficient is 0.6666. Once again, observe that minimum amount of MI between the hidden states and the belief is required for the policy to be optimal. Here, at least 5.0 bits of MI is necessary.
0.28
Return ((.)
GRU
BRC
NBRC
Return j(O.)
-1-
MGU
MI I(e.) [bit]]
-1
-2
3000 Episode e
4000
MI f(00)) [bil]
Figure 4: Stochastic T-Maze (L 50, = 0.3). Evolution of the return j(e.) and the MI î(0x) after episodes (left), and the return j(O.) with respect to the MI î(0..) (right). The maximal expected return is given by the dotted line.
In Figure 4 the results are shown for the Stochastic T-Maze with L = 50 and À 0.3. On the contrary to the Deterministic T-Maze, where the belief is a Dirac distribution over the states, there is uncertainly on the true state this environment. We can nevertheless observe that previous observations hold for this environment too. The MI and the expected return are indeed both increasing throughout the training process, and the best performing RNNs, such as the BRC and nBRC, have a MI that increases faster and stays higher, while the LSTM struggles to reach both high return and high MI. Here, the linear correlation
8

---

Published in Transactions on Machine Learning Research (08/2022)
coefficient between the MI and the return is 0.5460 and the rank correlation coefficient is 0.6403. It can also be noticed on the right that the best performing policies have MT of at least 4.5 bits in practice.
In the Deterministic T-Maze, it can be observed that the estimated lower bounds (o b) on the MI that are obtained by the MINE estimator are tight. Indeed, in this environment, the hidden state and belief are discrete random variables and their mutual information thus upper bounded by the entropy of the belief. Moreover, the belief is a Dirac distribution that gives the actual state with probability one. Under the optimal policy, each state is visited with equal probability, such that the entropy of the belief is given by logo (102) = 6.6724 for the Deterministic T-Maze of length L = 50, where 102 is the number of non terminal states. As can be seen in Figure 2 the optimal policies reach an estimated MI around 6.5 at maximum, which nearly equals the upper bound. The same results is obtained for the Deterministic T-Maze of length L = 100, where the entropy of the belief is given by logz(202) = 7.658 and the optimal policies reach an estimated MI around 7.0 at maximum, as can be seen in Figure 3 We expect this result to generalise to other environments even if this would be difficult to verify in practice for random variables with large or continuous spaces.
4.3 Mountain Hike and Varying Mountain Hike
The Mountain Hike environment is POMDP modelling an agent walking through mountainous terrain. The agent has position on two-dimensional map and can take actions to
1.00
0.75
move in four directions relative to its initial orientation: For- ward, Backward, Right and Left. First, we consider that its initial orientation always North. Taking an action results in a noisy translation in the corresponding direction. The transla- tion noise is Gaussian with standard deviation of ør 0.05. The only observation available is a noisy measure of its rel- ative altitude to the mountain top, that is always negative. The observation noise is Gaussian with a standard deviation of 00 = 0.1. The reward is also given by this relative altitude, such that the goal of this POMDP is to to obtain the highest possible cumulative altitude. Around the mountain top, the Figure 5: Mountain hike altitude function.
0.50
0.25
x
0.00
00.25
00.50
-0.75
-1.00
-1.0
00.5
0..0
0.5
1.0
X1
states are terminal. The optimal policy thus consists of going fast as possible towards those terminal states while staying on the crests in order to get less negative rewards than in the valleys. This environ- ment is represented in Figure 5) This POMDP is inspired by the Mountain Hike environment described in (Igl et al., 2018). The discount factor is = 0.99. We also consider the Varying Mountain Hike in the experiments, a more difficult version of the Mountain Hike where the agent randomly faces one of the four cardinal directions (i.e., North, West, South, East) depending on the initial state. The agent does not observe its orientation. As a consequence, the agent needs to maintain a belief about its orientation given the observations in order to act optimally. This POMDP is formally defined in /Susecction A.3
Figure shows on the left the expected return and the MI during training for the Mountain Hike environment. It is clear that the DRQN algorithm promotes high MI between the belief and the hidden states of the RNN, even in continuoss-saate environments. It can also be seen that the evolution of the MI and the evolution of the return are strongly linked throughout the training process, for all RNNs. We can also see on the right in Figure G that the correlation between MI and performances appears clearly for each RNN. For all RNNs, the lincar correlation cocfficient is 0.5948 and the rank correlation coefficient is 0.2965. In particular, we see that the best policies, with a return around -20, are clearly separated from the others and have significantly higher MI on average.
In Figure 7, we can see the evolution and the correlation between the return and the MI for the Varying Mountain Hike environment. The correlation is even clearer than for the other environments. This may be due to the fact that differences in term of performances are more pronounced than for the other experiments. Again, the worse RNNs such as the LSTM and the BRC have a significantly lower MI compared to the other cells. In addition, the performances of any RNN is strongly correlated to their ability to reproduce the belief filter, as can be seen on the right, with sharp increase in empirical return the MI increases from 2.5 to
9

---

Published in Transactions on Machine Learning Research (08/2022)
-10
Retam J((.))
-20
LSTM
GRU
-40-
BRC
NBRC
Return j(..
-30
60
MGU
-40
MI i( [bit]]
50
2-
60
2000
4000
6000
S000
10000
2
5
Episade
MI T(O.) [bit]
Figure 6: Mountain Hike. Evolution of the return i(22) and the MI i(0c) after e episodes (left), and the
return (Oo)) with respect to the MI 7(0.) (right).
Roturn ((.)
LSTM
GRU
BRC
60
NERC
Return (
MGU
80
MI ((O,) [bit]]
70
10000
20000
30000
40000
2
Episode
MI ((0.) [bit]
Figure 7: Varying Mountain Hike. Evolution of the return j(O,) and the MI I(0e) after episodes (left), and the return ((2) with respect to the MT f(0.) (right).
4.5 bits. More precisely, the linear correlation coefficient between the MI and the return is 0.5982 and the rank correlation coefficient is 0.6176. This increase occurs throughout the training process, as can be seen the left.
4.4 Belief of variables irrelevant for the optimal control
Despite the belief being a sufficient statistic from the history in order to act optimally, it may be that only the belief some state variables is necessary for optimal control. In this subsection, we show that approximating the Q-function with an RNN will only tend reconstruct the necessary part, naturally filtering away the belief of irrelevant state variables.
In order to study this phenomenon, construct state variables, independent of the original ones, and irrelevant for optimal control. More preciscly, we add d irrelevant state variables of that follows a Gaussian random walk. In addition, the agent acting in the POMDP p' obtains partial observations of of the new state variables through an unbiased Gaussian observation model. Formally, the new states and observations are distributed according to
POMDP P' from POMDP P by adding new
p(s = o(s6; 0.
(21)
p(s/11 st) o(s/411 st. 1), VI No,
(22)
10

---

Published in Transactions on Machine Learning Research (08/2022)
p(o: s/) s6 1), Vt E No.
(23)
where (1 Σ) is the probability density function of a multivariate random variable of mcan μ R and covariance matrix E E Rdxd evaluated at RR, and is the identity matrix.
1.46
1.46
Return J
Return j
MI ( [bit]]
MI ( [bit]]
10
5.0
Belief for s
Belief for s
Belief for
2.5
Belief for
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode e
Episode e
(a) d=1
(b) d=
Figure 8: Deterministic T-Maze (L = 50) with d irrelevant state variables. Evolution of the return (0e) and the MT ((0,) for the belief of the irrelevant and relevant state variables after e episodes, for the GRU cell. The maximal expected return is given by the dotted line.
Figure 8) shows the return and the MI measured for the GRU on the T-Maze environment with L = 50. It can be observed, as for the classic T-Maze environment, that the MI between the hidden states and the belief of stale variables that are relevant to optimal control increases with the return. In addition, the MI with the belief of irrelevant variables decreases during training. It can also be seen that, for 4, the MI with the belief of irrelevant variables remains higher than the MI with the belief of relevant variables, due to the high entropy of this irrelevant process. Finally, it is interesting to note that the MI continues to increase (resp. decrease) with the belief of relevant (resp. irrelevant) variables long after the optimal policy is reached, suggesting that the hidden states of the RNN still change substantially. Similar results are obtained for the other cells (see Appendix H
Return J (0.)
-00
Return ((..)
20
40
MI ((0.) [bitt]
MI i(0.) [bit]
10
Reliel for s
Belief
Relief for
Belief s
2000 4000
6000
8000
10000
2000
4000
6000 8000
10000
Episode
E Episode e
(a) d 1
(b) d
Figure 9: Mountain Hike with with irrelevant state variables. Evolution of the return J(O.) and the MI I(0.) for the belief of the irrelevant and relevant state variables after episodes, for the GRU cell.
Figure 9 shows the return and the MI measured for the GRU on the Mountain Hike environment. The same conclusions as for the T-Maze be drawn, with clear increase of the MI for the relevant variables
11

---

Published in Transactions on Machine Learning Research (08/2022)
throughout the training process, and a clear decrease of the MI for the irrelevant variables. In addition, it can be seen that the optimal policy is reached later when there are more irrelevant variables. It is also clear that adding more irrelevant variables increases the entropy of the irrelevant process, which leads to a higher MI between the hidden states and the irrelevant state variables. Similar results are obtained for the other cells (see Appendix
4.5 Discussion
As shown in the experiments, under the distribution induced by a recurrent policy trained using recurrent Q-learning, its hidden state provide a high amount of information about the belief of relevant state variables, at any time step. The hidden state of the RNN is thus statistic from the history that encodes information about the belief. In addition, at any time step, the network performs update of this statistic, based on the actions and observations that are observed. The RNN thus implements a filter that provides a statistic encoding the belief.
However, it was only shown that the RNN produces such a statistic under the distribution of histories induced by the learned policy. For the sake of robustness of the policy to perturbations of histories, we might want this statistic to also provide information about the belief under other distribution of histories. In Appendix F] we propose an experimental protocol to study the generalisation of the learned statistics. The results show that the MI between the hidden states and the beliefs also increases throughout the training process, under distributions induced by various E-greedy policies, even the fully random policy. We impute those results to the following reasons. First, the DRQN algorithm approximates the O-function, which generally requires a richer statistic from the history than the optimal policy. Second, the DRQN algorithm makes use of exploration, which allows the RNN to learn from histories that are diverse. However, we still observe that the higher the noise, the lower the MI. From these results, we conclude that the statistic that is learned by the network generalises reasonably well to other distributions of histories.
5 Conclusions
In this work, we have shown empirically for several POMDPs that RNNs approximating the Q-function with recurrent Q-learning algorithm (Hausknecht & Stone. 2015 Zhu et al., 2017) produces statistic in their hidden states that provide high amount of information about the belief of state variables that are relevant for optimal control. More precisely, we have shown that the MI between the hidden states of the RNN and the belief of states variables that are relevant for optimal control was increasing throughout the training process. In addition, we have shown that the ability of a recurrent architecture to reproduce, through a high MI, the belief filter conditions the performance of its policy. Finally, we showed that the MI between the hidden states and the beliefs of state variables that are irrelevant for optimal control decreases through the training process, suggesting that RNNs only focus on the relevant part of the belief.
This work also opens up several paths for future work. First, this work suggests that enforcing high MI between the hidden states and the beliefs leads to an increase in the performances of the algorithm and in the return of the resulting policy. While other works have focused on an explicit representation of the belief in the hidden states (Karkus et al. 2017 [gl et al.) 2018), which required to design specific recurrent architectures, we propose to implicitly embed the belief in the hidden state of any recurrent architecture by maximising their MI. When the belicf or state particles are available, this can be done by adding an auxiliary loss such that the RNN also maximises the MI. In practice, this can be implemented by backpropagating the MINE loss beyond the MINE architecture through the unrolled RNN architecture, such that the hidden states are optimized to get a higher MI with the beliefs.
Moreover, this work could be extended to algorithms that approximate other functions of the histories than the Q-function. Notably, this study could be extended to the hidden states of a recurrent policy learned by policy-gradient algorithms or to the hidden states of the actor and the critic actor-critic mothods. We may nevertheless expect to find similar results since the value function of policy tends towards the optimal value function when the policy tends towards the optimal policy.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Acknowledgments
Gaspard Lambrechts gratefully acknowledges the financial support of the Hallonia-Brusseer Federation for his FRIA grant and the financial support of the Walloon Region for Grant No. 2010235 - ARRIC by DW4AI Adrien Bolland gratefully acknowledges the financial support of the Wallonia- -arnnilals Federation for his FNRS grant. Computational resources have been provided by the Consortium des Equipements de Calcul Intensif (CÉCI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.FNRS)) under Grant No. 2502011 and by the Walloon Region.
References
Bram Bakker.
Reinforcement learning with long short-term memory. Advances in neural information pro- cessing systems, 14, 2001.
Mohamed Tshmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information ncural cstimation. In International conference on machine learning, pp. 531 540, 2018.
Dimitri Bertsckas. Dynamic programming and optimal control: Volume I, volume 1. Athena scientific, 2012.
Junyoung Chung, Caglar Gulcchre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprinl 1rXiv:1412.3553 2014.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain Markov process expecta- tions for large time, I. Communications on Pure and Applied Mathematics, 28(1):1 47, 1975.
Tuomas Haarnoja, Aurick Zhou, Pieler Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861 1870, 2018.
Matthew Hausknecht and Peter Stone. Deep recurrent Q-lcarning for partially observable MDPs. In Asso- ciation for the advancement of artificial intelligence fall symposium series, 2015.
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory- control with recurrent neural networks. ar preprint orXiv:1512.00 04455, 2015.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second association for the advancement of artificial intelligence conference on artificial intelligence, 2018.
Sepp Hochreiter and Jürgen Schmidluber. Long short-term memory. Neural computation, 98):11335-1780. 1997.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational re- inforcement learning for POMDPs. In International Conference on Machine Learning, pp. 2117-2126, 2018.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting partially observable stochastic domains. Artificial intelligence, 101(1-2):99 134, 1998.
Peter Karkus, David Hsu, and Wee Sun Lec. QMDP-net: Deep learning for planning under partial observ- ability. Advances in neural information processing systems, 30, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprint arXiv:1412. 6980, 2014.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review F, (9(6):066135 2004.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. arXiv preprinl orXiv:1509.020 2015.
Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg, and Pedro Ortega. Meta-trained agents implement Bayes-optimal agents. Advances in neural information processing systems, 33: 33:18691-18777 2020.
Volodymyy Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et. al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Volodymyy Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforeement learning. In Inter- national conference on machine learning, pp. 1928 1937, 2016.
Josep M. Porta, Matthijs T. J. Spaan, and Nikos Vlassis. Value iteration for contimuous-state POMDPs. Technical Report LAS-UVA-04-02 December 2004.
Richard D Smallwood and Edward J Sondik. The optimal control of partially observable markov processes over a finite horizon. Operations research, 21(5):1071-1088, 1973.
Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52-517 2002.
Nicolas Vecoven, Damien Ernst, and Guillaune Drion. A bio-inspired bistable recurrent cell allows for long-lasting memory. Plos one, 16()))e0 2021.
Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the institute of electrical and electronics engineers, 15600) 1990.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R. Salakhutdinov, and Alexan- der Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Guo-Bing Zhou, Jianxin Wu, Chen-Lin Zhang, and Zhi-Hua Zhou. Minimal gated unit for recurrent neural networks. International Journal of Automation and Computing, 13(3):226 234, 2016.
Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learning for POMDPs. arXiv preprint Xiv: 079.07978, 2017.
14

---

Published in Transactions on Machine Learning Research (08/2022)
A
Environments
In this section, the class of environments that are considered in this work are introduced. Then, the envi- ronments are formally defined.
A.1 Class of environments
In the experiments, the class of POMDPs that are considered is restricted to those where we can observe from or a state S, is terminal. A state s E S is said to be terminal if, and only
T(s' ss = 6s(s), As' E s) = O, va E
USS. Va EA
(24)
(25)
where & denotes the Dirac distribution centred in E S. As can be noted, the expected cumulative reward of any policy when starting in a terminal state is zero. As a consequence, the Q-function of a history for which we observe a terminal state is also zero for any initial action. The PRQL algorithm thus only has to learn the Q-function of histories that have not yet reached a terminal state. implies that the histories that are generated in the POMDP can be interrupted as soon a terminal state observed.
A.2 Mazz environments
The T-Maze environment POMDP (S,A,O, Po. T, R, O, parameterisod by the maze length N and the stochasticity rate À [0, 1]. The formal definition of this environment is given below.
Up
(E. 11
(0.0)
(1.0)
(2.0) (3.0)
14.0)
(5.0)
(6.0)
e- (1,0)
-(4,-1))
Down
-- (1.1)
(0.0)
c=10.0)
(2.0)
cm13.0)
14.0)
e-15.00
16.0)
e- (1.0)
(1.-1)
Figure 10: T-Maze state space. Initial states in blue, terminal states in grey, and treasure states hatched.
State space. The discrete state space composed of the set of positions C for the agent in cach of the two maze layouts M. The maze layout determines the position of the treasure. Formally, we have
S M x C
(26)
M (Up, Down}
(27)
C (00.0) 0) ... (L, U (2..))
(28)
A state s; S is thus defined by S; (m, ci) with m, M and e₁ E C. Let us also define {s: (m, c.) S c, E {(T, 1), (I, -1)}} the set of terminal states, four in number.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Action space. The discrete action space A is composed of the four possible moves that the agent can take
A {(1,0), (0. (0,, -0)
(29)
that correspond to Right, Up, Left and Down, respectively.
Observation space. The discrete observation space O is composed of the four partial observations of the state that the agent can perceive
= (Up, Down, Corridor, Junction].
(30)
Up = (Up. (0, and So (Down, (0, 0)), depending on the maze in which the agent lies. The initial state distribution S - [0. 1] is thus given by
Initial state distribution. The two possible initial states are Sp
Down
0.5 if So = 5g
(s0)
0.5
if So so Down
(31)
otherwise
Transition distribution. The transition distribution function T:SxAxx [0 is given by
80, ($1+1)
if E otherwise
T(s++1 st, ar)
(32)
(11 ()8 (S--1) + (Enes & (sr,a)))
where 8, E S, a, E A and 81+1 E S, and ʃ is given by
f(s = 81+1 (m,, Ct aa)
f a EC otherwise
(33)
S1+1 (m,, ct)
where s; (m, c.) E and a A.
Reward function. The reward function R:Sx A x S R is given by
if EF
0
if F,5t11 & F,s; St+1
-0.1
if & F, S:+1 F,s = St41
R(st, at, S1+1) =
(L, 1)
if m = Up
(34)
4
if & F, 8:41 E F,,c1+1
(I,-1)
if m Down
(I,--)
if m Up
-0.1 sf F, 8:41 F, c141
(L,+1)
if mm+1111 Down
where s; (m,, c.) E S, A and S (m,1,, c41) S.
Observation distribution. In the T-Maze, the observations are deterministie. The observation distribu- tion S 0 - [0, 1] is given by
if o, Up. c, = (0, m; Up
1 if Down, C, (0, m, Down
O st) =
1
if O: Corridor, c, E ((1,0)..... (L 1,0)
(35)
1
if O Junction, c, {LL, 0), (L, 1), (L, (L)
0 otherwise
where s, (my E S and o, O
16

---

Published in Transactions on Machine Learning Research (08/2022)
Exploration policy. The exploration policy & A [0, is stochastic policy that is given by (Right)) 1/2 and ((ther)) = 1/6 where Other E {Up, Left, Down). It enforces the exploration of the right hand side of the maze layouts. This exploration policy, tailored to the T-Maze environment, allows one to speed up the training procedure, without interfering with the study of this work.
Truncation horizon. The truncation horizon H of the DRQN algorithm is chosen such that the expected
displacement of an agent moving according the exploration policy in a T-Maze with an infinite corridor both sides is greater than L. Let [(Right) and = E(Left). In this infinite T-Maze, the probability of increasing its position is p (1 X)r + X and the probability of decreasing its position is (1 A A As consequence, starting at O, the expected displacement after one time step is I (1 A) - (). By independence, Zи = H such that, for TF 2 L, the time horizon is given by
H
(36)
A)(r
A.3 Mountain Hike environments
The Varying Mountain Hike environment is POMDP AS, T, R,R,,,,, parameterised by the sensor variance do E R and the transition variance ot R. The formal definition of this enviromment is given below.
1.00
0.75
0..00
0.25
0.25
0.50
0.75
X2
0.00
1.00
0.25
1.25
0.50
1.0
0.5
0.75
-1.0
0.0
-0.5
+>
1.00
0.0
0.5
1.0
0.5
0.0
0.5
1.0
0.5
-1.0
X1
1.0
Figure 11: Mountain hike altitude function h X.
The state space S is the set of positions X' and orientations C that the agent can take. Formally, we have
State space.
S = X X
(37)
X' = 1|
(38)
C = (0%, 9900, 180" 180",2709)
(39)
The orientation c = 0,, 1,, 180 and 270 corresponds to facing East, North, West and South, respectively. The set of terminal states is F = {s = (x. c) E
|x (0.8, 0.8) 0.13.
Action space. The discrete action space is composed of the four possible directions in which the agent move
A = (00 (-0100)) (0, (01,0))) that correspond to Forward, Left, Backward and Right, respectively.
(40)
17

---

Published in Transactions on Machine Learning Research (08/2022)
Observation space. The continuous observation space is O IR.
Initial state distribution. The initial position is always is always (0.8 and the initial orientation is sampled uniformly in C, such that the initial state distribution po : - [0, 1] is given by
Po(so) =
Σ
C 0(0.00.. CEC
(41)
Transition distribution. The transition distribution T: S x Axx - [0, is given by the conditional probability distribution of the random variable ($1+1 at) that is defined as
if E clamps (s: R(c) a, + +V(0, otherwise
St
St+1
(42)
where clamps(s) is the function that maps to the point in S that minimizes its distance with s, and
(cos sin c
sin c) cos
R(c)
(43)
is the two-dimensional rotation matrix for angle
Reward function. The reward function R: 8 Ax
IR is given by
10
sf otherwise
R(s, ac Sc+1) (((441))
(44)
where S, S,a, E A, S E 8, and 8 RT the function that gives the relative altitude to the mountain top in any state. Note that the altitude is independent of the agent orientation.
Obscrvation distribution. The obscrvation distribution O: 8 O [0, is given by
0(o; ss)) o(o,; h(s)).
(45)
where 8; S and or O, and where o(; μ, o²) denotes the probability density function of a univariate Gaussian random variable with mcan j and standard deviation
Mountain Hike, The Mountain Hike environment is a POMDP PS,AA.OP,,TTRR.O, parameterisod by the sensor variance do and the transition variance oT E R. The formal definition of this environment is identical to that of the Varying Mountain Hike, except that the initial orientation of the agent is always North, which makes it an easier problem. The initial state distribution is thus given by
Po(so)
(46)
Exploration policy. The uniform distribution U(A) over the action space A chosen as the exploration policy E(A).
The truncation horizon of the DRQN algorithm is chosen equal to H = 80 for the Mountain Hike environment and Hl 160 for the Varying Mountain Hike environment.
Truncation horizon.
B Deep recurrent Q-network
The DRQN algorithm is an instance of the PRQL algorithm that introduces several improvements over vanilla PRQL. First, it is adapted to the online setting by interleaving the generation of episodes and the
18

---

Published in Transactions on Machine Learning Research (08/2022)
update of the estimation Qo. In addition, in the DRQN algorithm, the episodes generated with the greedy policy of H + P(A), derived from the current estimation Qo- This stochastic policy selects actions according to arg aa Qa(-; a) with probability 1 E, and according to an exploration policy E(A) E P(A) with probability E. In addition, a replay buffer of historics is used and the gradient evaluated on a batch of histories sampled from this buffer. Furthermore, the parameters are updated with the Adam algorithm Kingma Ba 20144. Finally, the target rt + maxae 2o (10=++1,a) a) computed using past version Or of the estimation Qa with parameters 0' that are updated to less frequently, which eases the convergence towards the target, and ultimately towards the Q-function. The DRQN training procedure is detailed in Algorithm 1
Algorithm DRQN- Q-function approximation
Parameters:
N the buffer capacity.
C' E N the target update period (in episodes).
E' E N the number of episodes.
H E N the truncation horizon.
I E N the number of gradient steps after each episode.
E C R the exploration rate.
R. the learning rate.
B N the batch size.
Inputs
(S, A, O. T, R, 7.. POMDP.
((A) E P(A) the exploration policy.
Initialise empty replay buffer B
Initialise parameters randomly
for 0...., F- do
if mod C then
Update target network with 0'
// Generate new episode, store history and rewards Draw an initial state so according to po and observe op Let 10:0 = 10:0=(0)))
for t= O....,
do Select a, ~ E(A) with probability e, otherwise select a = arg a (No a}} Take action ar and observe r, and o
10
Let m0.++1 = (oo, 01,..., 0++1]
if [8] < N then add (Taxr, a,, (0411,00441) 1) in replay buffer B else replace oldest transition in replay buffer B by (Maa, ar, rt, 0141, 10441) if o,41 is ferminal then break
// Optimise recurrent Q-network
16
i= 0=....,, do Sample transitions (v af of veser) uniformly from the replay buffer B
17
r + maxaed { Do (1)))1)a)) if of is not terminal otherwise
18
Compute targets bd
Compute loss EB-1 (v' Co 2²))²
Compute direction using Adam optimiser, perform gradient stop 0 <c 0 ag
C Particle filtering
As explained in Section 2 with continuous state space require to perform integration over the state space. Furthermore, in these environments, the belief should be represented by function fnite-dimensional vector. Such arbitrary beliefs cannot be represented in digital computer.
the belief filter becomes intractable for certain POMDPs. In particular, POMDPs
continuous domain instead of
To overcome these two difficulties, the particle filtering algorithm proposes to represent an approximation of the belief by a finite set of samples that follows the belief distribution. In other words, we represent b, P(S) by the of M samples
S. = (57" (sm 1
(47)
where si E S, 0...., M being independent realisations of the distribution by
19

---

Published in Transactions on Machine Learning Research (08/2022)
Particle filtering is procedure that allows one to sample set of states St that follow the belief distribution b, The set is thus updated each time that new action a,-1 is taken and new observation Or is observed. Although this procedure does not require to evaluate expression ( it is necessary to be able to sample from the initial state distribution Po and from the transition distribution T, and to be able to evaluate the observation distribution O. This process, illustrated AAlgrrithm 2 guarantees that the successive sets , have (weighted) samples following the probability distribution by...., by defined by equation
Algorithm 2: Particle filtering
Parameters: E the number of particles
Inputs
AS,A, O, T; R, O.po, 7) a POMDP.
N the number transitions
Tu. (ou, ..,..., -, 11,,,, og) ) c Hu. a history 1/ Cenerate seighted samples following the initial belief by Sample 56 1p.....5
Pu
m 0, ...,
do
"i0 t O(0g 188
7 - 1) 4 m0
for o...., M do
"0 + "0 /n
So { w m=0
W-1
Generate successive weighted samples following the beliefs by
1...., 17
10
11
m ................................ .............................. do
12
Sample E (0....; M - 13 according to p(1) = w
Sample sp" T(: 8 14-1.-a0 ac-1)
14
w O(o s
7 1 wi
for 0, 0,...,
do
17
w < w /n
18 S. [ST with M-1
Algorithm starts from N samples from the initial distribution Po- These samples initially weighted by their likelihood O si)) Then, have three steps that are repeated at each time step. First, the samples are resampled according to their weights. Then, given the action, the samples are updated by sampling from T(- az). Finally, these new samples are weighted by their likelihood 0(0 si+1) given the new observation o1+1, as for the initial samples. As stated above, this method ensures that the (weighted) samples follow the distribution the successive beliefs.
D Mutual information neural estimator
In Subsoction D.1 the MI estimator that is used in the experiments is formally defined, and the algorithm that is used to derive this estimator is detailed. In Subseetion D.2 we formalise the extension of the MINE algorithm with the Deep Set architecture.
D.1 Estimator
As explained in Subsection 2.3
the ideal MI neural estimator, for parameter space T is given by
(x(X;Y) sup is (X;Y) 69$
(48)
(X;Y) = E [T) - log (2((3)))
E
(49)
zmp
However, both the estimation of the expectations and the computation of the supremum are intractable, practice, the expectations thus estimated with the empirical means the set of samples {(x", "))
N-1
20

---

Published in Transactions on Machine Learning Research (08/2022)
drawn from the joint distribution p and the set of samples {(x", y)") obtained by permuting the samples from Y, such that the pairs follow the product of marginal distributions px PY- In order to estimate the supremum over the parameter space 4, the MINE algorithm proposes to maximise (X;;;) by stochastic gradient ascent over batches from the two sets of samples, as detailed in Algorithm 3 The final paramoters o² obtained by this maximisation procedure define the estimator
N-1 E
N-1
To (x" yx" log
Σ
The (x",
(50)
N
that is used in the experiments. This algorithm initially proposed in Belghazi et al.
2018
Algorithm 3: MINE lower bound optimization
Parameters: E E N the number of episodes.
B E N the batch size.
a E R the learning rate.
Inputs
N C N the number of samples.
D == yx)] the set of samples from the joint distribution. Initialise parameters é randomly.
for
o...., 1- do
Let random permutation of (0, N- 1}. Let D1 random permutation of (0, (O....,N N 1). Let P2 random permutation (O....,
while = o,..., N do
Let { (xplk)) yrlk)) (411)8-1 batch of samples from the joint distribution. Let { (x
k-B8
(411)8-1
a batch samples from the product of
k=i8
marginal distributions Evaluate the lower bound
L(d)
Σ To(x,y) log (x,y)S
Σ
(o(3.9)
B
B
(F.5)eS
Evaluate bias corrected gradients G(d) voL(o)
Update network parameters with & + o aG(
D.2
Deep sets
As explained in (Subsection 4.1 the belief computation is intractable for environments with continuous state spaces. In the experiments, the belief of such environments is approximated by set of particles S = (s m==1 that are guaranteed to follow the belief distribution, such that sT ~ b, Vss" E S (see Appenddi Cj. Those particles could be used for constructing an approximation of the belief distribution, a problem known as density estimation. We nonetheless do not need an explicil estimate of this distribution. Instead, the particles can be directly consumed by the MINE network. In this case, the two sets of input samples of the MINE algorithm take the form
AM
((x",")) ((",,"")) S") N-1
(51)
N-1 MM mm=1) n=o
(h", nn M
(52)
In order to process particles from sets S as input of the neural network Ta, we choose an architecture that guarantees its invariance to permutations of the particles. The deep set architecture (Zaeerr et al.) 2017).
that is written as Ps (Eaes & (s)), provides such guarantoes. Morcover, this architecture theoretically able to represent any function on sets, under the assumption of having representative enough mappings Ps and tha and the additional assumption of using finite sets S when particles come from uncountable set.
21

---

Published in Transactions on Machine Learning Research (08/2022)
in this work. The function To is thus given by
Σ (0())) ) LES
(h, Má h, Ps
(53)
when the belief approximated by set of particles.
E Hyperparameters
The hyperparameters of the DRQN algorithm are given in Table and the hyperparameters of the MINE algorithm are given in Table 2 The value of those hyperparameters have been chosen a priori, except for the number of episodes of the DRQN algorithm and the number of epochs of the MINE algorithm. These were chosen so as to ensure convergence of the policy return and the MINE lower bound, respectively. The parameters of the Mountain Hike and Varying Mountain Hike environments are given in Table 3
Name
Value
Description
S
Number RNN layers
D
Number linear layers (no activation function)
H
32
Hidden state size
N
8192
Replay buffer capacity
C
10
Target update period in term of episodes
T
10
Number of gradient steps after each episode
0.2
Exploration rate
B
32
Batch size
10-3
Adam learning rate
Table DRQN architecture and training hyperparameters.
Name
Value
Description
L
2
Number of hidden layers
H
256
Hidden layer size
N
10000
Training set size
E
200
Number of epochs
B
1024
Batch size
x
10-3
Adam learning rate
R
16
Representation size for the Deep Set architecture
a
0.01
EMA rate for the bias corrected gradient
Table 2: MINE architecture and training hyperparamoters.
Name
Value
Description
do
0.1
Standard deviation of the observation noise Standard deviation of the transition noise
OT
0.05
Table 3: Mountain Hike and Varying Mountain Hike parameters.
22

---

Published in Transactions on Machine Learning Research (08/2022)
F Generalisation to other distribution of histories
In this section, study if the hidden state still provides information about the belief under other distribu- tions of histories than the one induced by the learned policy (20). This generalisation to other distributions is desirable for building policies that are more robust to perturbations of the histories.
We propose to study the evolution of the MI between the hidden state and the belief when adding noise to the policy used to sample the histories. Formally, instead of sampling the hidden states and beliefs according to 20
propose to sample those according to
p(() J p(h,b v) Paz(n) ()
(h
b) - 1=
(54)
where p(t) is once again chosen to the uniform distribution over the time steps p(t) = 1/H, te (O,....,H-. of is the e-greedy policy as defined in Appendix B. and Poş (1) t) gives the conditional probability distribution induced by the policy of over histories 1) E H. given that their length is t E No- Note that the training procedure remains unchanged.
The results of this additional study can be found in Figure 12. for & E {0.0, 0.2, 0.4, 0.8, 1.0). It can be noted that Po.o is the distribution of hidden states and beliefs induced by the learned policy (20). and P1.o is the distribution of hidden states and beliefs induced by fully random policy. For reasons of computational capacity, this analysis was carried out for the GRU cell only. This cell was chosen for being a standard cell that performs well in all environments in terms of return, unlike the LSTM. As can be seen in Figure 12 the MT wetw the hidden states and the beliefs increases throughout the training process, under all considered policies, even the fully random policy. We conclude that the correlation between the hidden states and beliefs gencralises reasonably well other distributions. In other words, the hidden states still capture information about the beliefs even under other distributions of histories.
G Correlations between the empirical return and the estimated mutual information
The correlation between the empirical return and the estimated MI are computed with the Pearson's linear correlation cocfficient and the Spearman's rank correlation cocfficient. These coefficients are reported for all environments and all cells in Table 4 and Table The columns named aggregated give the correlation coefficients measured over all samples of and / from all cells.
Environment.
Aggregated
LSTM
GRU
BRC
nBRC
MGU
T-Maze (L 50, A = 0.0)
0.8233
0.7329
0.8500
0.8747
0.9314
0.9178
T-Maze (L- = 100, A = 0.0)
0.5347
0.3624
0.6162
0.6855
0.6504
0.6299
T-Maze (I. = 50, A >= =033))
0.5460
0.2882
0.8008
0.7229
0.7424
0.6159
Mountain Hike
0.5948
0.7352
0.6177
0.4338
0.5857
0.5485
Varying Mountain Hike
0.5982
0.6712
0.4530
0.4446
0.3669
0.3006
Table 4: Pearson's 's linear correlation coefficient for each environment and cell.
Environment.
Aggregated
LSTM
GRU
BRC
nBRC
MGU
T-Maze (L = 50, X = 0.0)
0.6419
0.7815
0.5963
0.5403
0.4009
0.5002
T-Maze (L = 100, = 0.0)
0.6666
0.5969
0.7108
0.5058
0.4605
0.5534
T-Maze (L = 50, X >= 0.3)
0.6403
0.3730
0.6600
0.5090
0.4706
0.6497
Mountain Hike
0.2965
0.5933
0.1443
0.2762
0.4337
0.2630
Varying Mountain Hike
0.6176
0.6869
0.3677
0.4355
0.2955
0.2266
Table 5: Spearman's rank correlation coefficient for each environment and cell.
23

---

Published in Transactions on Machine Learning Research (08/2022)
1.46
0.53
Retum J(fd)
Return j(B.)
-2
MI I bit
MI [ [btt]]
2-
1000
2000
3000
4000
5000
0
2000
4000
6000
8000
10000
Episode
Episode
(a) Deterministic T- P-Maze (L 50)
(b) Deterministic T-Maze (L 100)
Return J(0.)
Return j((.))
-2
-40
6
MI T(0 [mitt]
MI ((a)) [btt]
2-
0
1000
2000
3000
4000
5000
2000
4000
6000
8000
10000
Episode
Episode
(c) Stochastic T-Maze (L 50, À = 0.3)
(d) Mountain Hike
Return (.)
-40
-00
80
MI ((0,) bbt]
2
10000
20000
30000
40000
Episode
(e) Varying Mountain Hike
Figure 12: Evolution of the return ((O)) and the MI i(0.) after episodes, under distribution of histories
induced by several e-greedy policies, for the GRU cell. The maximal expected return is given by the dotted line.
24

---

Published in Transactions on Machine Learning Research (08/2022)
H Belief of variables irrelevant for the optimal control
In this section, we report the evolution of the return and the MI between the hidden states and the belief of both the relevant and irrelevant variables for the LSTM, BRC, nBRC and MGU architectures. It completes the results obtained for the GRU cell Subsection 4.4
1.46
1.46
Return J(
Return J(O.)
0
0
-1
MI [(0.) [bitt]
((0.) [bit]
Belief for
Belief for
Belief for
Belief
2.5
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode
Episode
(a) 1=1
(b) d=
Figure 13: Deterministic T-Maze (L = 50) with d irrelevant state variables. Evolution of the return (0.)
and the MI f(0z) for the belief of the irrelevant and relevant state variables after episodes, for the LSTM cell.
1.46
1.46
Return j
Return j(0.)
MI ((0)) [bit
i(0.) [bit
7.5
Belief for sf
Belief for s
5.0
Belief for
Belief for
2.5
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode
Episode:
(a) d=
(b) d=
Figure 14: Deterministic T-Maze (T. 50) with d irrelevant state variables. Fvooution of the return (0.) and the MI ((0.) for the belief of the irrelevant and relevant state variables after episodes, for the BRC cell.
Figure 15 and Figure 16 show the evolution of the return and the MI for T-Maze of length L 50 with E (1,4) irrelevant state variables added to the process for these cells. These results are reported for the GRU cell in Figure 8 (sce (Ssseccbtt 4.2). As can be seen from these figures, the return generally increases with the MI between the hidden states and the belief of state variables that are relevant for optimal control. Moreover, as for the GRU cell, the MT between the hidden states and the belief of irrelevant state variables generally decreases throughout the learning process.
Figure 13 Figure 14
Additionally, it can be observed that the LSTM and BRC cells fail in achieving a near-optimal return when d 4. As far the LSTM is concerned, it is reflected in its MI that reaches lower value than the other
25

---

Published in Transactions on Machine Learning Research (08/2022)
RNNs. Likewise, the BRC cell does not reach high return, and the MI does not increase all. For this cell, it can be seen that the MI with the belief of irrelevant state variables is not decreasing, even with d = 1. The inability of the BRC cell to increase its MI with the belief of relevant variables and to decrease its MI with the belicf of irrelevant variables might explain its bad performance this environment.
1.46
1.46
Return J (0,)
Return j(O.)
1
MI ((0.) [bit]
MI [(0,) [bit]
7.5
Belief for s
Belief for s
5.0
Belief for
Belief
2-
2.5
3000
4000
5000
2000
3000
4000
Episode e
Episode e
(a) d=1
(b) =
Figure 15: Deterministic T-Maze (L = 50) with d irrelevant state variables. Evolution the return (0e))
and the MI ((0.) for the belief of the irrelevant and relevant state variables after episodes, for the nBRC cell.
1.46
1.46
Return J(B.)
Return j(
MI ((0.) [bit]]
MI I(02) [bit]
10
Belief s
Bolief for
Belief for
Belief for
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Episode e
Episode e
(a) d
(b)
Figure 16: Deterministic T-Maze (L = 50) with d irrelevant state variables. Evolution of the return ((o)
and the MI i(0_) for the belicf of the irrelevant and relovant state variables after e episodes, for the MGU cell.
I Figure 18 Figure 19 and Figure 20 show that all previous observations also hold for this environment with the LSTM, BRC, nBRC and MGU cells. These results are reported for the GRU cell in Figure (see (Sussection 4.2). As can seen from these figures, the return clearly increases with the MI between the hidden states and the belief of relevant state variables, for all cells. In contrast, the MI with the belief of irrelevant state variables decreases throughout the learning process.
As far as the Mountain Hike is concerned, Figure 17
26

---

Published in Transactions on Machine Learning Research (08/2022)
Return J(O.))
Return j(e.)
20
MI [(0,) [bitt]
5.0
Belief sf
MI ((,) [bit]
Belief for
Belief for
Belief for
0.0
2000
4000
6000
8000
10000
2000
4000
6000 8000
10000
Episode
Episode e
(a) d =
(b) d=4
Figure 17: Mountain Hike with with irrelevant state variables. Evolution of the return (0.) and the MI I(0.) for the belief of the irrelevant and relevant state variables after episodes, for the LSTM cell.
Return j(0.)
Return j(0.)
-20
20
40
MI ((0.) [bit]]
MI ((0)) [bit]
7.5-
Belief for s'
Belief
Belief for
Belief for
2000
1000
6000
8000
10000
2000
4000
6000 8000
10000
Episode
Episode
(a) d 1
(b) d =
Figure 18: Mountain Hike with with irrelevant, state variables. Evolution of the return (0.) and the MI i(0.) for the belief of the irrelevant and relevant state variables after episodes, for the BRC cell.
27

---

Published in Transactions on Machine Learning Research (08/2022)
Returr J(O.)
20
Return j(e.)
-20
MI ( [bitt
MI [( [bit]
7.5
Belief for
5.0
Belief for
Belief for s
Belief for
2.5
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Episode
Episode e
(a) d ==
(b) d=4
Figure 19: Mountain Hike with with irrelevant state variables. Evolution of the return (0.) and the MI I(0.) for the belief of the irrelevant and relevant state variables after episodes, for the nBRC cell.
Return ((..)
-20
Return j(0.)
20
MI ((0.) [bitt]
MI ((0.) [bit]
Belief for s
7.5
Belief for
5.0
Belief for
Belief for
2.5
2000 1000
6000
8000
10000
2000
4000
6000 S000
10000
Episode
Episode
(a) d =
(b) d =
Figure 20: Mountain Hike with with irrelevant, state variables. Evolution of the return ((.)) and the MI i(0.) for the belief of the irrelevant and relevant state variables after episodes, for the MGU cell.
28


Reviews:
[{"review_id": "1Bf3Mfh58X", "reviewer": "Reviewer_VF6u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The core contribution is convincing, clear and interesting. Providing a deeper understanding of a well-established method of learning to act in POMDPs. The one exception being that the results in the varying mountain hike environment are less clearly correlated. The learning curves in Figure 5 suggest that the agent was still learning, perhaps this experiment would have benefited from a longer run?\n\nSome results are also included to show that the mutual information between the learnt representations and irrelevant variables input decreases as the agent's performance improves with a GRU policy. It is unclear why these results are limited to this one network architecture, given the broader range of architectures experimented with in the rest of the paper. This is still an interesting insight, but I currently have less confidence if the observations in this section (4.4) generalise.\n\nThe paper is well written and provides sufficient background material to understand the technical contribution but includes no discussion of related studies making it hard for readers unfamiliar with the area to appreciate how this study compliments others or where to go next to explore this topic further. In particular I would note:\n\n1. \"Meta-trained agents implement Bayes-optimal agents\" [NeurIPS 2020] also investigates the internal representations learnt by recurrent policies;\n2.  \"AMRL: Aggregated memory for reinforcement learning\" [ICLR 2020] also investigates the effect of noisy observations on learning recurrent policies in the T-maze environment.\n\nThe appendix provides extensive implementation details on the experiments that were run, but limited explanation for how these precise values were chosen. Improved documentation of these experimental design choices may reveal important insights into the rigor of the study. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides empirical insight into the representations learnt by DRQN in partially observable environments across four variants of two domains and 5 recurrent policy architectures. Providing convincing evidence that the agent's performance in these environments is correlated with high mutual information between the learnt hidden state and belief.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Critical To Securing My Recommendation**\n\n1. Please extend the experiments in section 4.4 to all 5 network architectures experimented with earlier in the paper or justify why a subset of these is chosen.\n\n2. Please elaborate on how the environment settings and algorithm hyperparameters provided in the appendix were chosen. \n\n**Other Opportunities To Strengthen The Work**\n\n1. Review the literature to identify and discuss related studies to further clarify the contribution of the paper.\n\n2. Please expand all acronyms on first use (e.g. the recurrent architectures introduced at the end of Section 2.2 as they are key prior work this paper builds on and some are very recent work that may not be familiar to all readers.)\n\n3. Are the dotted lines in Figures 1-3 the return of the optimal policy? Please clarify if so. It would also be more informative to include the value of these policies on the y-axis\n\n4. In Figure 2, there appear distinct clusters of policies with 0 return (and again about halfway between 0 and the max return) that have constant return across a wide range of MI. Can the authors provide any insight in to the types of policies that have been learnt here? \n\n5. Does running the varying mountain hike experiments longer enable the agent's return to converge and improve the correlation between the MI and return?\n\n6. Subjectively, I personally think the illustrations of the environments in the appendix would be beneficial to include in the main body of the paper as a concise way to communicate the more precise environment details documented in the appendix. \n\n7. The appendix provides good clarity on the environment and algorithm hyperparameters used. The reproducibility of the experiments could be further improved by providing open source access to the code. \n", "broader_impact_concerns": "No immediate broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ae6JDUejf", "reviewer": "Reviewer_Vmp3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " Are the claims made in the submission supported by accurate, convincing and clear evidence? \n+ The claims are supported by some evidence\n- The evidence is not fully convincing. See my requested changes, which include both moderating claims and including more evidence.\n+ The design of the experiments is otherwise very nice, especially Sec 4.4. This was a compelling result, to illuminate that the RNN's aren't approximately filtering _all_ state variables, only  (reward-)_relevant_ state variables (for this particular setting studied, at least). \n\n Would some individuals in TMLR's audience be interested in the findings of this paper?\n+ Yes. At the very least, I am interested, therefore, I can answer this question with certainty. I am fairly confident that some others would be interested in the findings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Using two POMDPs in a few different settings, this paper shows that empirical returns are correlated with the estimated mutual information between hidden states of RNNs (used by a particular recurrent DQN algorithm) and a Bayes filter's beliefs of control-relevant states. This is a contribution because it brings new understanding to how these algorithms work, and introduces a new tool for introspecting these RNNS (estimating the MI of their hidden states with beliefs). Notably, the paper also analyzes MIs with beliefs of control-irrelevant states by augmenting the POMDPs with state components that do not affect the reward or transition dynamics, and shows that these MIs decrease during optimization.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nMore evidence needed:\n1.1. As far as I can tell, the experiments are all conducted with a single run of each algorithm. Therefore, the results could be different for different initializations of the Q-networks. This weakens the conclusions that we can draw. I request that the authors provide more evidence in the form of including more trials, with different initialization schemes (e.g. change the random seed).\n\n1.2. Furthermore -- because the MI is estimated via a lower-bound, it's not clear how tight this lower bound is with the particular hyperparameters chosen (see Alg 3). Can the authors comment and justify the parameters adopted? For example, imagine running just one step of optimization of this MI estimation procedure. This seems like it would invalidate the claims made in this paper, since the resulting lower-bound would likely be quite loose. In contrast, the original MINE algorithm runs until some convergence criterion is met. It could make sense to also run this paper's experiments in a problem setting in which we know that the MI estimation error is very low and the Bayes filter is exact, e.g. 1-dimensional hidden state and 1-dimensional control-relevant belief state, although I am not actually requesting this experiment be run (it would be helpful, but it is not necessary).\n\nClaim reduction needed:\n2.1 The statement of conclusions needs to be adjusted, as these results apply not to all recurrent deep q-learning algorithms, but just to the one investigated; also, it doesn't show that the underlying beliefs are _recovered_ (the h's aren't direct stand-ins for the control-relevant belief states, they just share high MIs with them when the policy is closer to optimal --- given a good policy's h, we don't yet have a way to translate that h to a belief state, unless we apply some additional machinery). This statement appears in the first sentence of the conclusion: \"In this work, we have shown, empirically, that the approximation of the Q-function by recurrent neural networks in partially observable environments comes with the approximation of the belief filter of the state variables that are relevant to optimal control.\" I believe it is more accurate to say something like (although it is quite verbose..) \"In this work, we have shown empirically that approximate mutual information between the hidden states of an RNN-based deep recurrent Q-learning algorithm <cite DRQN papers> and the beliefs of a Bayes filter of control-relevant state variables is correlated with that policy's higher returns in two POMDPs.\"\n\nMinor requests:\npage 1: \"one to known the POMDP\" -> \"one to know the POMDP\"\n\npage 2 \"We thus rely on particle filtering in order to approximate the belief by a set of states, called particles, distributed according to the belief distribution.\"\nSome discussion of why we should still expect the MI estimates to be good enough to draw conclusions is missing here.\n\nEq 2,3 -- it's not standard to omit the policy from the value function notation -- i.e. it's standard to use Q^{\\pi^*} or Q^*, instead of just Q. I think it could improve clarity to include the policy in the notation of these Q-functions, although the definition of Eq 2 is fine.\n\nIt might be nice to add a reference to a definition and explanation of Bayes filters near Eq 5. For example, \"Burgard, Wolfram, Dieter Fox, and Sebastian Thrun. \"Probabilistic robotics.\" The MIT Press (2005).\" is a good introductory textbook.\n\nThe notation for the estimated MI is \\hat I(\\theta_e). I think this should instead be a function of two arguments, like the standard MI, e.g. \\hat I(h, b), since the RVs are the h's and the b's coming from the filter.\n\n\n\n\n\n\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fcQEZQxuhN", "reviewer": "Reviewer_DNRi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nOverall I really liked the paper. The scope and contributions are clearly stated in the abstract and introduction and the rest of the paper follows through on the statements made in the beginning. I particularly liked the writing, which I found to be clear and precise, with good mathematical notation. \nSufficient background is given in the paper and whenever I was missing some additional information (with a few exceptions - see below), it was provided in the appendix (for example the observation spaces of the environments). \nExperimental results are clearly explained and discussed, supporting the claims made in the introduction (see below for one exception). \n\n# Weaknesses\n\nOne area in which I believe the paper could be strenghtend is in discussing the connection between the mutual information $I(h,b)$ between the rnn-state $h$ and the belief $b$, and the conclusion that $h$ captures information about $b$ by _filtering_. At the moment, the authors mostly equate the MI and that \"such value functions internally filter the posterior probability distribution of the current state given the history\", which I believe is not necessarily true.  \n\nThree possible pitfalls that I can come up with are the following, although there might be others:\n* The time $t$ might be a confounding factor as the rnn-state might simply count the number of steps since the beginning of the episode. In this case, for near-deterministic policies the correlation between $h$ and $b$ might still be high, without $h$ capturing anything about $b$. I believe this could be countered by not always measuring $I(h,b)$ for histories $\\eta$ starting from the beginning of the episode. The author's most likely are already doing that, but I couldn't find the exact information or discussion thereof in the paper.\n* Secondly, and this could be supported by additional experiments, is that I believe there are actually two claims: a) The latent state $h$ captures information about the current (belief) state $b$, which is what the MI measures. And b), the rnn updates perform something that resembles _filtering_ to update the latent state.  For example a) can be (and likely is) true in deterministic environments, even without b) being true. Furthermore, I believe T-Maze is not sufficient to show b), as it really only requires memory and not something that aggregates information over time to perform belief inference (equation 7). On the other hand, the Mountain Hike environment likely does require some form of inference, but this depends on the values of the transition and observation noise (i.e., $\\sigma_T$ and $\\sigma_O$), which I couldn't find in the paper nor appendix (though apologies if I missed it). Lastly, the fact that rnn-transitions do indeed perform something approximating inference could, I think, be further supported by showing some 'belief-like' characteristics. For example, it might be interesting to measure the accuracy of trying to predict the $x,y$ state in the MH environment, based on the rnn-state, and see how this value changes over history-length. If the rnn-state performs something like filtering, I would expect this accuracy to increase over time, at least up to a point.\n* More generally, I think to what extend the rnn-update resembles filtering might strongly depend on the environment, with some environments allowing for \"shortcuts\". For example, maybe in the MH environment, the agent is not measuring anything about the $x,y$ position, but learned to perform local hill-climbing, which only requires knowledge of the last few altitude values and movement directions. \n\nA last, independent point: It would be helpful to have one ablation study showing the accuracy of the MINE estimation. I.e. taking the same policy-checkpoint and running MINE multiple times, to get an estimate of the measurement noise. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors investigate whether the rnn-state in recurrent neural networks approximates the \"belief state\" (i.e., the posterior distribution over the latent environmental state) in POMDPs and the rnn-update performs something related to updating the belief state by filtering. They do so by measuring the mutual information (as measured by MINE) between the rnn-state and a computed 'ground truth' belief state for a variety of RNN architectures, on two different environments, each in a deterministic and stochastic variant. They use DRQN as algorithm to train the architectures.\n\nThey find that rnn-states and ground truth belief states have high mutual information and that higher MI is correlated with better performance of the agent. They also find that mutual information with additionally added 'nuisance features' in the environment decreases over time, in contrast to the MI with relevant features, which increases. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "# Critical for recommendation of acceptance\n\nA discussion should be included about why (or why not) high mutual information means that $h$ captures information about $b$ and does so by _filtering_ similar to equation 7, instead of relying on shortcuts or not having to perform filtering due to insufficient observation noise (see above for details).\n\nAlso, please provide the missing environmental parameters $\\sigma_T$ and $\\sigma_O$.\n\n# Strenghtening the paper\n\nI belive the paper could be futher strengthed with additional experiments strenghtening the connection between high MI and the claim about filtering. For example, but not necessarily, the one I suggested about measuring the state-prediction accuracy as a function of history lenght. \n\nThe MINE accuracy ablation study would also be helpful, but certainly not critical.\n\nOverall, I really liked the paper and hope the authors find my suggestions helpful. ", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how much RNN recurrent state corresponds to belief states, and through the experiments presented by the authors find strong support that it does.

Through discussion with the reviewers, several improvements to the paper were suggested both in the main body, and in the appendix, and to the best of my understanding, they have been incorporated in the current version. As such, and due to the unanimous consent from the reviewers, I am happy to recommend acceptance without further revision.

Nonetheless, I encourage the authors to consider whether any final changes or improvements to the writing are called for based on the reviewer comments and discussions, and to ensure that they present the strongest camera ready paper possible on the basis on incorporating all suggestions from the reviewers. I will operate on the basis of trust here and not require a second look at the final draft.

Congratulations.
Decision: Accept (poster)