Agent-State Construction with Auxiliary Inputs
Authors: Ruo Yu Tao
Adam White
Marlos C. Machado
Venue: Accepted by TMLR ()

Abstract: In many, if not every realistic sequential decision-making task, the decision-making agent is not able to model the full complexity of the world. The environment is often much larger and more complex than the agent, a setting also known as partial observability. In such settings, the agent must leverage more than just the current sensory inputs; it must construct an agent state that summarizes previous interactions with the world. Currently, a popular approach for tackling this problem is to learn the agent-state function via a recurrent network from the agent's sensory stream as input. Many impressive reinforcement learning applications have instead relied on environment-specific functions to aid the agent's inputs for history summarization. These augmentations are done in multiple ways, from simple approaches like concatenating observations to more complex ones such as uncertainty estimates. Although ubiquitous in the field, these additional inputs, which we term auxiliary inputs, are rarely emphasized, and it is not clear what their role or impact is. In this work we explore this idea further, and relate these auxiliary inputs to prior classic approaches to state construction. We present a series of examples illustrating the different ways of using auxiliary inputs for reinforcement learning. We show that these auxiliary inputs can be used to discriminate between observations that would otherwise be aliased, leading to more expressive features that smoothly interpolate between different states. Finally, we show that this approach is complementary to state-of-the-art methods such as recurrent neural networks and truncated back-propagation through time, and acts as a heuristic that facilitates longer temporal credit assignment, leading to better performance.

Published in Transactions on Machine Learning Research (05/2023)
Ageet--State Construction with Auxiliary Inputs
Ruo Yu Tao', Adam White 1, 2, Marlos C. Machado4.
Department of Computing Science, University of Alberta
Canada CIFAR AI Chair, Alberta Machine Intelligence Institute (Amii)
{rtao3, amw8, boochaddd@uabbbrta..co
Reviewed on OpenReview: http:://ppeaanrrs.....e///////tp
Abstract
In many, if not every realistic sequential decision decision-making task, the decision- decision-making agent is not able to model the full complexity of the world. The environment is often much larger and more complex than the agent, a setting also known as partial observability. In such scttings, the agent must leverage more than just the current sensory inputs; must. construct an agent state that summarizes previous interactions with the world. Currently, a popular approach for tack- ling this problem is to learn the agent-state function via a recurrent network from the agent's sensory stream input. Many impressive reinforcement learning applications have instead relied on environment-specifc functions to aid the agent's 's inputs for history summarization. These augmentations are done in multiple ways, from simple approaches like concatenating observations to more complex ones such as uncertainty estimates. Although ubiquitous in the field, these additional inputs, which we term auriliary inputs, are rarely emphasized, and not clear what their role or impact is. In this work we explore this idca further, and relate these auxiliary inputs to prior classic approaches to state construction. We present series of ex- amples illustrating the different ways of using auxiliary inputs for reinforcement learning. We show that these auxiliary inputs can be used to discriminate between observations that would otherwise be aliased, leading to more expressive features that smoothly interpolate between dif- ferent states. Finally, we show that this approach is complementary to state-of-the-at meth- ods such as recurrent neural networks and truncated back- procpppoppggtion through time, and acts as a heuristic that facilitates longer temporal credit assignment, leading to better performance.
1 Introduction
In reinforcement learning, an agent must make decisions based only on the information it observes from the environment. The agent interacts with its environment in order to maximize special numerical signal called the reward. This problem formulation is quite general and has been used in several high-profile success stories, such agents capable of achieving impressive performance controlling fusion reactors (Degrave et al. 2022) beating Olympians in curling (Won et al., 2020), and when navigating balloons in the stratosphere (Bellemare et al. 2020). An important feature of these problems is that the environment the agent is in the real world is much bigger than the agent itself. In this setting, the current observation from the data stream the agent. experiences does not contain all the relevant information for the agent to act on, making the environment partially observable.
In such settings, the agent must leverage more than just its current observations, it must construct an internal state that summarizes its previous interactions with the world, sometimes known as history summarization.
We refer to this internal stale as the agent state. One way to learn agent-stale functions is to leverage recurrent neural network architectures (Hausknecht & Stone 2015 Vinyals et al. 2019 Degrave et al., 2022 to summarize an agent's 's history. These rocurrent functions calculate latent states (also called hidden states), and learn the function used summarize history. Another approach that has been used in real-life use cases of reinforcement learning to help resolve partial observability is to model predictive information about the

---

Published in Transactions on Machine Learning Research (05/2023)
agent's uncertainty over its effectiveness (Won et al. 2020 observations (Bellemare et all 2020 allowing the agent to reason about what information the agent does not know. Explicitly learning and leveraging pre- dictions (Rafols et al. 2005b) is another approach that has been considered for history summarization. Two popular approaches to this are predictive state representations (Littman et al.) 2001) and general value func- tions (Sutton et al. 2011), which represent agent state with future predictions. All these different approaches for constructing agent-state functions have different limitations and assumptions, but the same purpose: embedding necessary information from observations by expanding the feature space into a richer class of features with auziliary inputs to ameliorate the issues of partial observability for better decision making.
Feature expansion has been considered in many different contexts, and has been widely used and investigated for neural networks over the years. Early incarnations of neural networks used random projections in the first layer of the neural network as "associator features" to map imputs to random binary features and expand the input space (Block, 1962). Expanding the input space to specifically tackle time-scries data has been considered in the prodiction context, where convolution over the history of inputs (Mozer 1996. has been proposed as an approach to incorporating simple, non-adaptive forms of memory for neural networks. In reinforcement learning, feature space engineering and expansion for agent-state construction was widely used before the advent of deep reinforcement learning. Techniques range from tilc coding (Sutton & Barto 2018) to radial and fourier basis functions (Sutton & Barto 2018 Konidaris et al., 2011), having even been applied to Atari games (Liang et al. 2016]- The largest drawback of these feature expansion techniques is the fact. that they are static they do not adapt to the problem setting at hand.
In this work we consider how we might resolve different forms of partial observability by augmenting the inputs to function approximator (i.e. a deep neural network) with feature expansion techniques for reinforcement learning. We revisit earlier formulations of explicit history summarization (Mozer 1996), and connect these ideas with modern approaches to agent-state functions. We look to combine the simplicity and performance of simple feature expansion techniques with the natural adaptivity and flexibility of neural network function approximation.
In this paper, we explore the idea that many approaches to tackle partially observable problems can be viewed as form of auxiliary input. In the context of agent-state construction for reinforcement learning in partially observable environments, we define auxiliary inputs as additional inputs, beyond encironment observations, that incorporale or model information regarding the past, present und/or fulure of reinforcemend learning agent. Auxiliary imputs have been ubiquitous across recent, real-world applications of reinforcement learning. Recent work in stratospheric superpressure balloon navigation with deep reinforcement learning has explored using not only the average magnitude and direction of the observed (or predicted) wind columns over time as input features, but also the variance of this wind column as an auxiliary input for successfully navigating balloons (Bellemare et al., 2020). In robotic curling, distance errors from previous throws were used as features to help mitigate the partial observability induced by environment conditions such changing ice sheets over time (Won et al., 2020). In biomedical applications, both a sime-decayer trace of joint activity (Pilarski ct al. 2012) and future predictions of prosthesis signals (Pilarski et al., 2013)) were used as additional input features for controlling or aiding in the control of robotic prostheses. All these approaches were successful due to carefully thought out auxiliary inputs that were fitting for their respective domains.
We present the following contributions in this work: (1) we survey auriliary input techniques used for resolving partial observability in successful real-world reinforcement learning applications. (2) We unily these approaches under single formalization for auxiliary inputs.
(3) We demonstrate empirically, through illustrative examples, how a practitioner might leverage this formalism to create auxiliary inputs, and the efficacy of these approaches in de-aliasing states for better value function learning and policy representation.
We first introduce a formalism for the auxiliary inputs used throughout reinforcement learning. This formalism gives practictioners a mechanism to parse the aspects of a partially observable environment an agent may need to consider for both learning a value function and successful policy, and as well as the type of auxiliary input to use in given partially observable domain. We use simple partially observable environment to elucidate how few simple, fast, and general instantiations of auxiliary inputs (a decaying trace of observations, particle filters, and likelihoods predictions) summarizes history, as well as general trajectory information needed for decision making. Through demonstrations on this environment, we show that auxiliary inputs allow an agent
2

---

Published in Transactions on Machine Learning Research (05/2023)
to discriminate between observations that would otherwise be aliased, and also allow for smooth interpolation in the value function between different states. Next, we demonstrate the efficacy of uncertainty-based auxiliary inputs on two classic, partially observable environments. Finally, we show that particular auxiliary inputs (specifically exponential decaying traces) can intograte well with recurrent neural networks trained with truncated backpropagation through time (T-BPTT), potentially allowing for a significant performance increase as compared to using only one or the other. Code and implementation for this work is publically available
To summarize, auxiliary imputs can be simple, performant, and should likely be the first approach most reinforcement learning practitioners take to tackle partial observability, In many cases, simple auxiliary inputs may be good enough if not better than more complex approaches such as recurrent function approximation, and they can also be easily combined with these approaches for better performance.
2 Background and Notation
We model the agent's interaction with the world as a sequential decision Decision-making problem. On each time step i, the agent takes an action 0 E A and receives an observation of the environment 0g E O. Partially in response to the agent's taken action, the environment transitions into a new state 5t+1 E S and receives a reward rr+1 E R, both according to the dynamics function p R S x A S - [0. 1]. The agent does not observe the state, only the current observation, which we specify as a vector in this w E EOc O". This obscrvation vector is constructed from the underlying state St according to an unobservable function o: S + O, where o(S.)= = O,, Periodically the environment enters a terminal state St, = 1 resetting the enviromment to a start state Sp. The agent's isss interaction is thus broken into a sequence of episodes.
The agent's primary goal is to learn a way of behaving that maximizes future reward. In our sctting, the policy is defined over the history of interactions because the agent cannot. observe the underlying state. Let h (Oo, Ag, Oi,..., O.) E TT where T denotes the space of all possible trajectories of observations and actions of all possible lengths. The return from timestep & is the discounted sum of rewards G, = Re+1+ Ru
R2, where y E 00 1) is the discount and L is the (stochastic) time of termination. The goal is to find a policy :: T + [0, 1] that maximizes the expected return, in expectation across start states Er [Go]. In this paper, we focus on methods that estimate a value function, qu (s, a) = Ex, [G, St = = At = a]. in order to incrementally improve the agent's current policy т.
In our work we primarily focus on the Sarsa (Rummery & Niranjan 1994] algorithm to learn estimates qs from the agent's interaction with the environment for control. Our policy æ is defined by two cases: either we choose greedy action with respect to gr or we choose a random action with probability to ensure sulficient. exploration. After sampling and taking an action A, ~ T[ | Ss),, we receive the next reward and next state Ret1,S+11 ~ p(.. St, At), and pick the next action A441 ~ IT(- SS41). In the more general, function approximation case, the estimate dn is parameterized by 0g E R and is updated with the semi-gradient Sarsa update 0.41 = 0, + a [Re11 + Tūx A4+1, 0,) - Zz (S,, Ar, 0.)) V (Sr, A,, Ot), where a > 0 is the step size, and V is the gradient of the function I with respect to the parameters O,.
In many problems, learning policies over full histories is not tractable and the agent must make use of an agent-state function to summarize the history. The agent-state function maps given hy to agent state vector I; E X. In most approaches to agent-state construction, including recurrent neural networks and general value function networks (Schlegel et al. 2021 the stent-state function has a recursive form:
F+++ - us at, 04+1) E R
(1)
where & E &RL are the parameters of the parametric agent-state function ug. We can easily extend Sarsa to approximate the value function from agent state: ((z,, a,, 0,) = 0T x4 R qr(s), Thus, the estimated value is a linear function of the agent-slate, which itself is a recurrent, potentially non-linear, function of æ;, at, and O.+1- Semi-gradient Sarsa simply adapts 0 and & from the agent's interaction with the environment in order to improve reward maximization, as before. The aim of this work to investigate how including auxiliary inpuls in agent-state construction (as input to u) can improve basue-based reinforcement learning agents.
Inttps://ght
inx-iiputs
2 We denote random variahles with capital letters, and vectors with bolded symhols.
3

---

Published in Transactions on Machine Learning Research (05/2023)
3 Auxiliary Inputs
In our work, we investigate the use of auxiliary inputs as an input into the agent-state function, and its implications in the different types of partially observability. As opposed to only summarizing the past history of experiences, auxiliary inputs also allows agents to summarize the present and/or fulure for decision- making. To do this, auxiliary inputs must summarize entire trajectories, which also includes potential future interactions, rather than only the agent's history. We denote these trajectory of an agent at time & Tr = {Oo, Ap. O1...., Oi, A E T, where L denotes the terminal time step of the trajectory. At the time step i, an agent will have only a partially realized trajectory, T₁ = {oo, 40, ..... Oc, Gt. Oc+1..... O,),, where observations including and before t are actualized variables (denoted with lower case letters), whereas all future observations from L +1 1 to L are still random variables.
Let M : T - R.W denote an auxiliary input as a function of the trajectory of the agent, which maps the trajectory T, to a fixod-length vector. As the input of this function variable with respect to trajectory length, and the output is of fixed length, the auxiliary input function acts as summarizing funclion which maps entire trajectories into some fixed-length vector that summarizes particular aspects of the trajectory. For an auxiliary input at time t, we denote this as M(T,) = M,. With these additional auxiliary inputs, we re-define our agent-stale function to include these auxiliary inputs:
Et+1 = uo( dts O++1, M E R
(2)
We visualize this auxiliary input function, as well the overall agent-environment interface in Appendix A Since trajectories T are likely to include future, unobserved random variables, it is sometimes helpful to define auxiliary inputs as functions that explicitly separate observed events from the past and unobserved random variables from the future:
M, = M(fon, ao.....
..... O., at, Ort1....,0ii Ma((O0, ao...., Oz, a) )}= + EM,
(3)
Or.]]
(4)
Where M,. and My are history summarizing and future summarizing functions respectively. EM, denotes the expectation of future trajectories, summarized by the function M,. We explicitly consider expectations here due to the unobserved random variables (future observations, actions, and terminal time step) past the current. time step, in order to consider concrete auxiliary inputs that do not depend on unobserved, future random variables. Equation 4 makes the distinction between the actualized, observed time steps, and the random variables of the trajectory clear: from O to t our auxiliary input function maps over observed, actualized variables, and from t ++1 1 to L we take the expectation with respect to the sampling distribution over all potential future observations and actions. We also present a further specified formulation of auxiliary inputs as convolution operation in Appendix B for correctness and further clarity. Finally, we note that while the function us is described as a recurrent function in the general case, most of the control algorit used in this work (besides the agents explicitly using an LSTM with auxiliary inputs in Section 5) do not leverage recurrency. This is because in most cases in practice, this not done since auxiliary inputs are enough in terms of de-aliasing state and adding enough state information. Furthermore, in this study, we do this isolate and understand the affects of these auxiliary inputs on state de-aliasing for value function learning by itself.
Many auxiliary inputs can be defined by the function M. To clarify how this formulation might be used, we show how frame stacking (Mnih et al. 2015), widely used in algorithms that succeed in the Arcade Learning Environment (Bellemare et al. 2013 Machado et al. 2018). fits into this formalism.
Frame Stacking. As an auxiliary input, frame stacking only considers the past four observations the agent has seen. This means that the auxiliary input only defined by the function M, = Ma, where Mr. is the function that concatenates the past 3 observations and the current observation together:
Ma((on, .o...., Or,
ar ax)) = o @ 2 O X O2
where the @ operation represents the concatenation operation. This auxiliary input function produces a fixed-length vector M, E 4 where n is the size of the observations. This is exactly the frame stacking technique ubiquitous throughout Atari-2600 experiments.
4

---

Published in Transactions on Machine Learning Research (05/2023)
0. Is the agent in location 0?
1. Is the agent in location 1?
2. Is the agent in location 2?
Collect
the reward in location observable and missing?
Lett
LO
Right
02
the reward in location observable and present?
the reward in location unobservable?
Right
the reward in location 2 observable and missing?
Right
L2
7. Is the reward in location 2 observable and present?
Collect
Callect
8. Is the reward in location 2 unobservable?
(((L visible
2(L2) visible
(a)
(b)
Figure 1: aa): The Lobster environment.
bb)
Binary questions representing the binary observation vector
emitted by the environment at time t, O₁.
Resolution and Depth. Viewing frame stacking as form of auxiliary inputs elucidates an interesting property of our formalization: auxiliary inputs defined by Equation represent differing depths and resolutions of the information you retain with regards to your trajectory. We define depth to be at what temporal length these auxiliary inputs retain information with regards to the agent's trajectory, and resolution to be the extent that information regarding individual observation-action pairs are preserved by the auxiliary inputs. time steps before the
Frame stacking is form of auxiliary inputs that is low in depth (as we only current time step), but high in resolution (since we retain all the relevant information of these 3 observations).
Incremental Functions. One important factor we focus on is the space and time complexity of calculating these auxiliary inputs. While we define these auxiliary inputs to be a function of history, we focus on algorithms that are incremental update functions for producing auxiliary inputs: M, = M(M O,, At).
With this framework in place, we describe three auxiliary inputs in this section. Before we do so, we describe the minimal partially observable environment we use to illustrate the benefits of our auxiliary inputs on: the Lobster environment.
3.1 The Lobster Environment
We now introduce the Lobster environment: a simple partially observable environment we leverage to show the state de-aliasing properties of auxiliary inputs. The environment is shown in Figure Ta In this environment, fishing boat has to travel between locations-represeeeee represented from lobster pots. Only locations L1 and L2 have lobster pots, which refill randomly over time after being collected. The environment starts with both pots filled. The observations emitted by the environment include both the position of the agent, and whether or not a pot is filled if the agent. is in the corresponding location. This observation vector shown in Figure [b An agent this environment has 3 actions: A (left, right, collect). The agent receives a reward +1 if it takes the collect action to collect the lobster pots at location with full lobster pots. We fully specify the details of this environment in Appendix
nodes in the graph- to collect lobsters
3.2 Instantiations of Auxiliary Inputs
With this environment in place, describe three techniques popular throughout reinforcement learning literature for auxiliary inputs with the formalism introduced in Scction 3 that incorporate model information from the past, present and/or or the future. We consider these three techniques throughout our work. Through the demonstration of these auxiliary inputs in a small, partially observable domain called the Lobster environment, in this section we look investigate and understand how these auxiliary inputs help with docision making in a simple and controlled setting. We look to compare the performance of each form of auxiliary input to two agents: one using only the observations described in Figure 1b another using the fully observable environment state.
5

---

Published in Transactions on Machine Learning Research (05/2023)
3.2.1 Exponential Decaying Traces
To incorporate information from the past of the agent, we consider exponential decaying traces of history as auxiliary inpuls to agent-stale. Decaying traces simply keep an exponentially decaying weighted sun of our observations and actions:
M (()
(5)
with À < 1, and is a preprocessing function for the observation, action pair. Written in an incremental form, we have M, AM,1 g(0, (+).
This form of auxiliary input acts a model of the past by acting as an exponential timer for events in the observation. When used ions allows the agent to take into consideration the time in which events occur in the observation vector or action a. This particular form of history summarization is high in depth, but low in resolution, since g(Or, a.) is aggregated together across time steps.
auxiliary imput to the agent-state function, an exponential decaying trace of
3.2.2 Approximate Belief State with Particle Filters
We now consider classic approach to auxiliary inputs for resolving partial observability: constructing approx- imate belief states. In this section, we investigate the use of uncertainty as auxiliary inputs approximating a distribution over all states at every time step. In order to get a measure of uncertainty of the environment state, we leverage a few assumptions with regards to available transition dynamics and known state struc- ture for the particle filtoring approach for approximating a distribution over possible states, or a belief state (Kaellling et al. 1998).
To construct these belief states, we consider Monte- Caante-Carlobbseel approach with particle filtering (itagawa 1996) to approximate a distribution over states (Trun 1999 Pincau & Gordon 2007 as auxiliary inputs for agent state. With this approach, we mainlain an approximale distribution over possible states by incorporating statistics of the current observation and action into this distribution. We do this by approximating this distribution with particles and corresponding weights, and updating these particles with the emission probabilities and dynamics function through particle filtering update. We begin with & particles, which we denote by vector of particle states &o E (1, (1.-.., [S]]*, initialized according to the start state distribution of the environment, and instantiate a vector of weights wo Rt At every step ++ 1, and for every particle Vj E (1,..., k),, with the dynamics function p, we update the particles and weights by first propagaling all particles forward with the action taken:
1 ~ p( 8,D)),
(6)
and updating each particle's weight according to the probability of emitting the observation (emission probability) received:
= = o S. = = w/]-]-
This produces the unnormalized weights of all particles. We get our new set of weights by simply normalizing: These weights are essentially the mechanism in which we summarize our past trajectory.
W++11
=
Wi+1
We+1
With this mechanism to update particles and weights, we form our auxiliary inputs based on these weights. These auxiliary inputs calculate our approximate distribution of states time step t+ 1, M411, by summing over weights for each particle for given state:
M Σ w Ø 1l [ablI]-
(8)
==0
Where the bolded [ corresponds to the one-hot encoding of length [S], with a at state 8. In this case, our auxiliary inputs M4+1 are our agent-state function, and we have T++1 = M
At every step, the auxiliary input function M defined by Equation Our current action and observation are incorporated into our auxiliary input by both the propagation of particles forward given action, and

---

Published in Transactions on Machine Learning Research (05/2023)
20.0
Ground-truth
Likelihood
Returns over 200 steps
17.5
PF
Right
(avg. 30 runss
Stade
15.0
Trace
Observations
12.5-
Normalited
Normalized
10.0
R(J) feature
rL11 feature
7.5
0,
1
5.0
0.0
1.0
2.0
2.5
((12) feature
R(L2) feature
Environment steps
1e5
(a)
(b)
(c)
The impact of auxiliary inputs for agent states in the Lobster environment. Fig. 2 lal: Online returns in the Lobster euvironment for four different agents, in comparison to the optimal policy given the ground-truti state. Dotted line represents the optimal policy given states. See text for details. (b) Normalized action-values for an agent with decaying trace auxiliary inputs in location LO for the left action. Each (r(L1), r[L2) combination represents possible input for the exponential decaying trace agent state. The values for the ground-truth state (crosses) and for using only the observation (diamond) are overlaid for comparison. See Appendix [ for similar plots for particle filter and general value function auxiliary inputs. Fig. (ch Action-values for both left (yellow) and right (blue) actions in location LO.
Figure 2:
the re-weighting of our particles through the emission probability of the current observation given the par- ticle. The actions are incorporated into the particle updates for { and observations are incorporated into the weight updates for W++1- Put logether, M, resolves partial observability through the counter counterfactual updates of the particles and their corresponding weights at the present time step with the current observation and action. This approach has low resolution because individual observations are incorporated into the ap- proximate belief state through its emission probabilities. Its depth depends on the nunber of particles used.
3.2.3 Likelihoods for Incorporating Future Predictions and Past Information
We now considor how to incorporate future predictions together with past information for auxiliary inputs. the reinforcement learning selting, one popular choice of predictions are general value functions (Sutton et al.j 2017). These predictions take on the form of discounted sum over cumulants (which can be a function over observations and actions), with some cumulant termination function. These can both be represented by our definition of auxiliary input functions M.
We describe auxiliary inputs that incorporate both future predictions and past information for the Lobster environment. Let i E 21, be the indices for both our two auxiliary inputs and our rewarding locations. These auxiliary inputs predict whether or not a reward at a given location Li will be present if the agent takes the expected number of steps to that location. These auxiliary inputs essentially amount to answering the question: "Given that saw r(Li) missing (r(Li) 0) some steps ago, if take the mcan number of stops to reach Li, what is the likelihood that r(La) will be regenerated (r(Li) 1)?" We calculate these likelihoods by counting the number of time steps since seeing each reward, and the expected number of time steps to reach Li. We describe the full implementation of this auxiliary input in Appendix C.3.5
Because this approach summarizes its history and future predictions with likelihood functions, it is low in resolution and high in depth. It low in resolution because these likelihoods summarize its history into a probability distribution, and would be hard recover individual observations. It is high depth as ex- ponential decaying traces are also high in depth: the likelihood probability distribution is also an exponential function of time steps, and so will have depth depending on the Poisson process rate.
7

---

Published in Transactions on Machine Learning Research (05/2023)
3.3 Results the Lobster Environment
We summarize the performance of the agents that leverage auxiliary inpuls in Figure 2a The policy learned with any of the three auxiliary inputs converges to a higher return than the agent using only observations. We can see this in the learned policies of our auxiliary-input agents versus the observations-only agent: when using only observations, the agent dithers between location LO and location L1 or L2, but not both. On the other hand, the agent that leverages additional auxiliary inputs collects the reward from one of the rewarding locations, and then traverses to the other rewarding location depending on the value of the inputs.
Comparing the performance of all three auxiliary inputs in Figure 2a we see the similarities in the converged average returns across the three algorithms. From all the visualizations of the action-value functions seen for the three auxiliary input approaches in Figures 2c and
7b we can see similaritics in the learnt value functions between the three auxiliary inputs that incorporate or model different kinds of information. Learn- ing a value function over these features result in similar policies. This implies that these auxiliary inputs all help resolve partial obscrvability in one way or another. All hyperparameters swept and algorithmic details are fully described in Appendix C.2
Value Function Geometry of Auxiliary Inputs To get a better idea of how auxiliary inputs impact our policy, consider the value function learnt over these auxiliary inputs. Specifically, since our value function is approximated with linear function approximation, we visualize and compare how the auxiliary inputs of exponential decaying traces affect the value function geometry of our Iearnt policy in Figs. 2b and 2c In these plots, we compare action-values when the agent is al location LO between the three algorithms: the agent using observations only (the single stamond-shaped point), the agent using ground-truth environment states (the crosses), and finally the agent with exponential decaying traces as auxiliary inputs (the small circular points). We fully describe the details behind this 3D plot in Appendix C.4 We also show similar value function plots for both the particle filtering and likelihood auxiliary inputs in Figure 7 in Appendíx C.3
We first consider Fig. 2b in this plot we calculate and visualize all possible input features mentioned above for the three algorithms over the normalized action-values of the left action, given the agent is at location LO. We can see that augmenting the agent state with auxiliary inputs expands the state space of the agent over two dimensions of time: one dimension for each reward observation that we have our exponential decaying trace over. The four vertices of this expanded state space represent and coincide with the four ground-truth states when the agent is at location LO.
Learning a value function with these exponential decaying traces allows the agent to smoothly interpolate between the values of the actual ground-truth states. The agent does this through resolving (checking out a location) or accumulating (waiting in another location) the decaying trace observations. Learning a value function with such agent states essentially allows the agent to disentangle and discriminate states that would otherwise be mapped to the same observation. This expansion in the state space allows for more expressivity in the value function.
This additional expressivity is also refiected in the policy. This is depicted in Figure 2c in addition to the action-value function for the left action, we also visualize the action- function for the right action of the agent with exponential decaying traces. By overlaying the action-values of both actions we can see how the decaying trace agent, as well as the agent using the ground-truth states, learn action-values that in some corners are greater for the right action and in other corners are greater for the 1eft action, actually leading to a sensible greedy policy. Alternatively, the agent that uses only the environment observations has no choice but to collapse the action-values of both actions into the same value. Finally, we presont results in Appendix C.5 on these same auxiliary inputs, but with a base control algorithm that can represent stochastic policies- Proximal Policy Optimization (Schulman et al. 2017). We show that while stochastic policies help in partially observable settings, auxiliary inputs resolve partial observability that is unresolvable by stochastic policies alone.
In this section, we have presented results and visualizations to help elucidate why these techniques help with decision making in partial observability. We have shown that auxiliary inputs expand the input space to de-alias states, and allow for more fino-grained policy representations and ultimately better performance. With these definitions in place, in this manuscript we further investigate two of these approaches for auxiliary inputs: present (particle filters) and past (exponential decaying traces) modelling.
8

---

Published in Transactions on Machine Learning Research (05/2023)
4 Particle Filtering for Auxiliary Inputs
We first consider the case with stronger assumptions: particle filtering for auxiliary inputs. We know that this form of auxiliary input adds the maximal amount of relevant information (Kaelbling et al. 1998) to our agent state in the partially observable setting. With this approach, we look to answer the following in this section: how much can auxiliary inputs help? How does it compare to using recurrent neural networks for function approximation?
In this section, we look at auxiliary inputs which explicitly represent uncertainty. Uncertainty as auxiliary inpuls has been used in many real-world reinforcement learning use cases, including wind-column uncertainty for stratospheric superpressure balloon navigation (Belemaaee et al.) 2020), and positional entropy as features for robotic navigation with reinforcement learning (Roy & Thrum 19999. In this section, we consider particlo- filtering- based uncertainly features for auxiliary imputs as described in Section Section3.2.2 We evaluate this approach classic partially observable environments: A modified version of the Compuss World (Rafols et al., 2005a] environment and the RockSample (Smith & Simmons 2004) environment. We compare this approach to agents using LSTMs for agent-state construction [Bakker 2001). We begin by describing these two environments.
4.1 Compass World and RockSample
Modified Compass World is a 9 x 9 partially observable environment where the agent can only see the color of the square directly in front of it. The goal of the agent to face the green square. The agent is initialized in a random position that is not the goal position. The agent has 3 actions: fmove forward, turn left, turn right]. Due to this form of partial observability, the goal location, and the random start position and pose, the agent has to resolve both its and y coordinates in order to reach the goal and receive a reward of +1. This environment is modified from the original Compass World environ- ment in that the goal location is in the middle of the west blue wall, as opposed to the top of the west blue wall. This is to add difficulty to this environment; in the original environment, the agent would only need to traverse up to the north facing orange wall (resolve its y-position) and head west, until reached the goal. In this modified version, the agent needs to resolve both coordinates for it to reach the goal. A visualization of this environment is shown in Figure 10a in Appendix D
RockSample(7, 8) is another partially observable environment where the goal of the agent is to collect as many good rocks as possible before exiting to the castern border of the gridworld. In this environment, we have a 7 Ix 7 gridworld with 8 rocks randomly scad throughout, the environment. At the start of an episode, each rock is randomly assigned to be either good or bad. The agent is unaware of whether or not a rock is good or bad, but has individual actions to check the goodness of each rock with an imperfect sensor that gets noisier the farther away the agent is to each rock. Besides these check actions, the agent also has the move actions {up, right, down, 1eft]. Collecting good rock gives a positive reward of + +10, and exiting to the right. also gives positive reward of +10. Collecting bad rock gives a reward of -10. A visualization of the Rock- Sample environment is shown in Figure 10b in Appendix D along with further details of both environments.
4.2 Results and Discussion
We compare our particle-fiteen ciaiicle-fiter--aseee based auxiliary imputs to other agent-state functions for Modified Compass World and RockSample in Figures 3a and 3b respectively. The particle filtering-bssed agent (orange, labelled as "Auxiliary Inputs") is compared to three bascline agents: An agent using only observations (teal, labelled as "Observations"), an agent which uses an LSTM to learn its agent-state function (yellow, labelled "LSTM"), and an agent that uses the ground-truth environment state (blue, labelled as "Ground-truth"),
In these figures, we plot the online returns during training over environment steps. All experimental results shown report the mean (solid lines) and standard error to the mean (shaded region) over 30 runs. Standard error to the mean is shaded but too small to be visible. Hyperparameters for each algorithm were selected based on a sweep. In these experiments, all agents utilize similar neural network architectures as their function approximator. Further details of the experimental setup and hyperparameters swept can be found in A Appendix []] Further details of the particle filter and the agent-state function for each environment in this
9

---

Published in Transactions on Machine Learning Research (05/2023)
0.50
Ground-truth
30
Particle Filter Aux. Inputs
Ground-truth
Episodic returns (30 run run average)
Episodic returns
(30 run average)
Particle Filter Aux. Inputs
20
LSTM
0.25
10
LSTM
Observations
Observations
0.00
0.0
0.5
1.0
0.0
0.5
1.0
1.5
Environment steps
1e6
Environment steps
1e6
(b)
Figure 3: Online discounted returns over environment steps for agents in both the Modified Compass World 3a and RockSamplo(7, 8) 3bb environments. Colors in both plots correspond to the same class of agents.
section can be found in Appendix D.2 We also perform ablation studies over the RockSample half efficiency
distance and LSTM-input action concatenation in Appendix D.3
The agents acting on observations (and not ground-trun ttunh state) must first resolve their partial observability in both environments. We first consider the learnt policies from both the LSTM and particle filter agent states in Modified Compass World. The agents first move forward until they see a wall color. Sceing this wall color resolves one of their position coordinates, and allows the agent to navigate towards the west wall. The agent then traverses either up down, periodically checking the color of the west wall until they resolve the other position coordinate, and get to the goal. This resolving of coordinates is implicitly represented as elements of the distribution over state going to O. In RockSample, using an approximate belief state as auxiliary imputs is particularly useful because these auxiliary inputs allow the agent to learn a policy that takes into account the uncertainty of the rocks in the current time step. The agent will traverse closer to a rock before checking whether the rock good or bad since accuracy of the check decreases with distance.
The particle filter auxiliary imput approach (with the additional assumptions that approximate belief states afford) also consistently outperforms the agent using LSTM agent-state function. Not only do these auxiliary imputs converge faster, but also converges to a higher average return. This implies that these auxiliary inputs are able to represent and add privileged information into the agent state that may be hard to learn and represent in recurrent neural network approaches, and are also beneficial for faster value function lcarning.
Incorporating trajectory information with particle filtering allowed the agent to leverage knowledge of the dynamics of the environment. This results in auxiliary inputs that were highly relevant for decision making, While the particle filtering approximate belief state assumptions may be strong assumptions to make, it is the case in many real-world use cases that this information (or approximations thereof) are not completely unrcasonable (e.g., [Bellemare et al., 2020). In these experiments, we have shown that in the idcal case, auxiliary inputs are able to add complex, relevant information to the agent state given a model of the environment. We now consider the case where we do not have these assumptions available and investigate modelling the past for auxiliary inputs.
5 Scaling Up Auxiliary Inputs and Integration with RNNs
With the efficacy of auxiliary inputs demonstrated in smaller environments, in this scction we now consider how to scale up approaches to auxiliary inputs, as well as the role of auxiliary inputs in gradient-based agent- state functions. Specifically, in this section we investigate ways in which to scale up exponential decaying traces described in Scction 3.2.1 to larger, pixel-basod environments, and how these traces integrate with recurrent neural networks as an agent-state function. We begin by describing the two variations of the partially observable environments we test on.
10

---

Published in Transactions on Machine Learning Research (05/2023)
Ground truth
Raturss staps)
30-
LSTM + Exp Trace
Ground-truth
Cava, 30 seadg)
35
Exp Trace
30
20
Observations
Returns
LSTM
20-
Gbssevvations
15
0.00
0.25 0.50 0.75 1000
1.25
Environment steps
1e6
+
Environment steps
167
(a) Fishing 1.
(b) Fishing results.
(c) Fishing 2.
(d) Fishing 2 results.
The Fishing Boat environments. At every time step the agent receives map with the 5 x 5 area around itself updated. This map includes obstacles, currents and rewards. Currents labelled with multiple directions represent stochastic currents. Dark blue grids represent obstacles, whereas light blue grids represent obstacles the agent is able to see through. b d) Results for the first and second Fishing Boat environments respectively, averagod over 30 runs. Standard errors are shaded for each curve.
Figure 4:
aa
5.1 The Fishing Environments
We first introduce the Fishing euvironments two pixel-based stochastic and partially observable environ- ments, visualized in Figures La and 1c This environment reminiscent of a scaled-up version of the Lobster environment, in that it was designed to test agent's 's ability to reason about partial observability and time, except in scaled-up sctting. In both of these foraging environments, the goal of the agent (a fishing boat) is to continually navigate around the stochastic currents and obstacles to collect fish from fishing locations (denoted by the green circles). Currents push the agent one step in the direction it is facing. After collecting the fish from a net, the not re-cast and fills up over a random amount of time. Partial obscrvability in the domain comes from a few sources: the first source of partial observability is the environment map. every step, the position of the agent is given and map of the environment accumulated into a larger map, much like in robot naviga- tion and mapping (e-g., Elfes 1987 Thrun 1998). In given step, the map only updated with potentially occluded x 5 area around the agent's 's current position. This 5 5 area contains information the direction of the currents, obstacles and rewards at the current time step. As the agent traverses the environment, current. and reward information on the accumulated map unobserved by the agent begins to "stale" since currents and rewards change stochastically with time. Further details of this environment are elucidated in Appendix E.1
Due to the stochasticity of the currents and rewards, the degree of partial observability of each environment. is dictated by the number of stochastic elements throughout the map, and the rate at which these random variables change. In Fishing (Figure 4a), have environment with low levels of partial observability and stochasticity. There is sparse number of currents throughout the map, with most currents acting gateway to the rewarding areas. Rewards and currents in this environment also have relatively slow rate of change. Fishing 2 (Figure 4cp is an environment that is much more partially observable, with several fast-changing currents throughout the environment. Specifics of the stochasticity in each environment are detailed in Appendix E.1.
5.2 Exponential Decaying Traces For Robot Mapping
To encode information with regards to the past history of the agent as an auxiliary input for these environments, we use exponential decaying traces (as per Section 3.2.1 over the past observable regions in the environment map. In this case we have auxiliary inputs that are all updated at once as malriz, which we define as M, idxdd, where d is the width and height of the map. Our auxiliary input is then:
M, Σ 1
1(0,) E {0, 1]axd a d d binary map which indicates which areas of the global map are obscrvable from o, at time т. At each step, all the locations that are not currently observable are decayed by factor of X < This auxiliary input encodes the time since the agent has observed particular location exponentially
11

---

Published in Transactions on Machine Learning Research (05/2023)
decaying timer. The incremental version of this auxiliary input is simply M, = max(AMM-1+11 1), where 1 is a d x d matrix of ones.
Results and Discussion We show our results for both environments in Figures 4b and 4d Experimental details including hyperparameters swept, algorithmic details, and environment details are included in Appendix E Results shown here are offline evaluation returns over environment steps, where we evaluate our agent after every fixed number (10K) of steps. We compare our exponential decaying trace auxiliary inputs (orange) to a few baselines: an LSTM-- based agent with action concatenation (yellow), and an agent with only the observation map as described before as input (tcal), and combination of both the trace auxiliary imputs combined with an LSTM agent-state function (purple). In both environments, exponential decaying traces as auxiliary inputs are comparable to, or performs slightly better than the LSTM-based agent. In the simpler Fishing 1, using exponential decaying traces matches the performance of the LSTM agent, as shown in Figure 4b] While the LSTM agent performs temporal credit assignment by using additional compute with T-BPTT and TD error propagation, our exponential decaying trace can be seen as a simple way of performing temporal credit assignment by only propagating TD error through the input features, reminiscent of eligibility traces (Sutton & Barto 2018). Doing so requires much less computation per time step as compared to T-BPTT. In the Fishing 2 environment, the additional stochasticity seems to harm the performance of the LSTM agent as compared to the agent using exponential decaying traces as auxiliary inputs, with the decaying trace agent slightly outperforming the LSTM agent in this case.
Exponential decaying traces are also able to integrate with LSTMs. One might expect no increase in performance when combining the two approaches, since LSTMs are able to exactly model an exponential decaying trace of observation features. But from the results, we see that in both environments, combining decaying trace auxiliary inputs with LSTM function approximation increased the performance of the agent by quite large margin in these environments. This implies that when combined, the trace features and LSTM hidden states are modelling different, but complementary, aspects of the partially observable environment for even better performance, As pointed out by Rafiee et al. (2022), adding an exponential decaying trace as input to an LSTM learning through T-BPTT seems to add robustness to the truncation window length. In our case, for control, it seems to both increase the rate of learning and to also increase the average returns of the learnt policy, This example suggests that auxiliary inputs can integrate well with gradient- agent- state construction.
To conclude, in this section we have demonstrated the the ability of decaying trace an auxiliary input to scale to a larger environment, and the ability of this auxiliary input to integrate well with RNN agent-state functions.
6 Conclusion and Discussion
To conclude, auxiliary inputs are helpful tools for reinforcement learning practitioners to resolve partial observability which also have the potential to integrate well with existing gradient- based agent-state functions.
In this work we advocate for the general principle of auxiliary inputs as an addition to agent-state construction, and evaluate different instantiations of auxiliary inputs for reinforcement learning. We first introduce auxiliary inputs as a unifying framework for input augmentation, as well as consider three different instantiations of these inputs in Section 3 Using the Lobster environment introduced in Section 3.1 we demonstrate the well as how these auxiliary inputs allow to expand the input feature space of the agent to allow to interpolate between ground-truth states, and for more fine-grained policy. With this formalism in place, we investigate the performance of the particle filtering approach to auxiliary inputs on few classic partially observable environments in Section 4 We show the eflicacy of approximate belief states as auxiliary inputs in these hard, partially observable environments. Finally, in Section | we investigate the use of simple exponential decaying traces of observation features as auxiliary inputs on the scaled-up pixel-based Fishing environment. Besides showing matching or better performance of these trace features in this environment as compared to LSTMs, we also show how this auxiliary input can integrate with recurrent neural network agent-state functions, and improve performance.
efficacy of these auxiliary inputs in resolving partial observability,
12

---

Published in Transactions on Machine Learning Research (05/2023)
As for future work, the most immediate extension of our investigation would be an empirical study the relative efficacies of different auxiliary inputs on different forms of partial observability and different learning algorithms on potentially larger environments. This extension could also consider a more fined-grained comparison of auxiliary inputs to learnt recurrent representations (e.g. LSTMs vs exponential decaying traces). Along this vein, another, langential direction of future work would be to utilize the formalism of auxiliary inputs developed here as guide for the (potentially automatic) selection of auxiliary inputs for a particular domain. Another avenue for future work would be to use more complex predictions as auxiliary inputs. In terms of using future predictions to resolve partial observability, general value functions (Sutton et al. 2011) and predictive state representations (Littman et al. 2001 are two promising approaches for using predictions for resolving partial observability as an auxiliary input.
While a promising area of research, future predictions have their limitations: in the context of using future predictions for next-step predictions as well as for control, general value functions have been shown to be effective in only a very limited scope of environments and predictions (Schlegel al., 2021). Another very promising direction for future work would be to integrate and combine different. forms of auxiliary inputs, which leverage information from all parts of the trajectory, from both past and future. Another promising direction for future work is to leverage auxiliary inputs for better exploration. While techniques to estimate the uncertainty of an agent for exploration are near ubiquitous throughout reinforcement learning, an interesting avenue for exploration is learning policies over altered imputs of an agent for more robust exploration.
Acknowledgments
We would like to thank Martha White for the support with computational resources, ideas, and leedback. We would also like to thank Matthew Schlegel for the continued support throughout this project, especially in their help in developing predictive algorithms for the Lobster environment. We would like to thank Patrick Pilarski for the many insightful comments and suggestions that have improved this work. Finally, we would like to thank Mare G. Bellemare and Taylor W. Killian for early discussions on what is now auxiliary inputs.
13

---

Published in Transactions on Machine Learning Research (05/2023)
References
Bram Bakker. Reinforcement Learning with Long Short-Term Memory. In T. Dietterich, S. Becker, and
Z. Ghahramani (ods.), Advances in Neural Information Processing Systems, 2001.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253 279, 2013.
Mare G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera S. Ponda, and Ziyun Wang. Autonomous Navigation of Stratospheric Balloons Using Reinforcement Learning. Nature, 588(7836):77-82, 2020.
Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, 6(5):679- 684, 1957.
H. D. Block. The Perceptron: A Model for Brain Functioning. Reviews of Modern Physics, 34:123 135, 1962.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsmpouuelli,, Jackie Kay, Antoine Merle, Jean-Marc Moret, Seb Noury, Federico Pesamossaa, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Karukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic Control of Tokamak Plasmas Through Deep Reinforcement Learning. Nature, 602(7897):4144417 2022.
A. Elfes. Sonar-based Real-world Mapping and Navigation. The Institute of Electrical and Electronics Engineers Journal on Roboties and Aulomation, 3(3):249 265, 1987.
Matthew J. Hausknecht and Peter Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In Association for the Advancement of Artificial Intelligence Fall Symposia, 2015.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and Acting in Partially Observable Stochastic. Domains. Artificial Intelligence, 101:99-1344 134, 1998.
Genshiro Kitagawa. Monte Carlo Filter and Smoother for Gan-Gaassian Nonlinear State Space Models. Journal of Computational and Graphical Statistics, 5(1):1-28 25, 1996.
George Konidaris, Sarah Osentoski, and Philip Thomas. Value Function Approximation in Reinforcement Learning Using the Fourier Basis. In Association for the Advancement of Artificial Intelligence Conference, pp. 380 385. AAAI Press, 2011.
Yitao Liang, Marlos C. Machado, Erik Talvitie, and Michael H. Bowling. State of the Art Control of Atari Games Using Shallow Reinforcement Learning. In International Conference Autonomous Agents B Multiagent Systems, 2016.
Long-Ji Lin. Self-improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Machine Learning, 8(3):293 321, 1992.
Michael I.. Littman, Richard S. Sutton, and Satinder Singh. Predictive Representations of State. In Advances in Neural Information Processing Systems, 2001.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journel of Artificial Intelligence Research, 61:523- 562, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level Control Through Deep Reinforcement Learning. Nature, 518:529-533 2015.
Michael C. Mozer. Neural Net Architectures for Temporal Sequence Processing, volume 265, pp. 243-264 Addison-Wesles, 1996.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Patrick Pilarski, Travis Dick, and Richard Sutton. Real-Time Prediction Learning for the Simultancous Actuation of Multiple Prosthetic Joints. In IEEE International Conference Rehabililation Robotics, 2013.
Patrick M. Pilarski, Michael R. Dawson, Thomas Degris, Jason P. Carey, and Richard S. Sutton. Dynamic Switching and Time Machine Learning for Improved Human Control of Assistive Biomedical Robots. In JEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics, pp- 296- 302, 2012.
Joelle Pineau and Geoffrey J. Gordon. POMDP Planning for Robust Robot Control. In Robotics Research, 2007.
Banafsheh Rafiee, Zaheer Abbas, Sina Ghiassian, Raksha Kumaraswanyy Richard Sutton, Elliot A Ludvig, and Adam White. From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learning. Adaptive Behavior, 2022.
Eddie Rafols, Anna Koop, and Richard S Sutton. Temporal Abstraction in Temporal-diiffreence Networks. In Advances in Neural Information Processing Systems, 2005a.
Eddie J. Rafols, Mark B. Ring, Richard S. Sutton, and Brian Tanner. Using Predictive Representations to Improve Generalization in Reinforcement Learning. In International Joint Conferences on Artificial Intelligence, 2005b.
Nicholas Roy and Sebastian Thrun. Coastal Navigation with Mobile Robots. In Advances in Newral Information Processing Systems, pp. 1043-1049, 1999.
G. A. Rummery and M. Niranjan. On-line Q-learning Using Connectionist Systems. Technical Report TR 166, Cambridge University Engineering Department, Cambridge, England, 1994.
Matthew Schlegel, Andrew Jacobsen, Zaheer Abbas, Andrew Patterson, Adam White, and Martha White. General Value Function Networks. Journal of Artificial Intelligence Research, 70:497-5433, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. Computing Research Repository, 2017.
Trey Smith and Reid Simmons. Heuristic Search Value Iteration for POMDPs. In Conference on Uncertainty in Arlificial Intelligence, 2004.
Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick Pilarski, Adam White, and Doina Precup. Horde: A Scalable Real-time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction Categories and Subject Descriptors. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, USA, 2018.
Sebastian Thrun. Learning Motric-topologiaal Maps for Indoor Mobile Robot Navigation. Artificial Intelligence, 99(1):21-77) 71, 1998.
Sebastian Thrun. Monte Carlo POMDPs. In Advances in Neural Information Processing Systems, 1999.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Tvo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukeuoghu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575:350 354, 2019.
Dong- Onn Won, Klaus-Robert Robert Müller, and Seong- Whan Lee. An Adaptive Deep Reinforcement Learning Framework Enables Curling Robots with Human-like Performance in Real-world Conditions. Science Robotics, 5(46)4ebb99764 2020.
15

---

Published in Transactions on Machine Learning Research (05/2023)
A
Auxiliary Input Function in the Agent Environment Interface
Agent
Observation
Environment
Reward
Auxiliary Input Func.
|Potentially Recurrent)
Agent- State
Action
Action
Value Func. /Policy
Agent State
Figure 5: The agent-environment interface, with the auxiliary input function visualized within the agent.
B Auxiliary Inputs as Convolutions Over Trajectories
Another useful formulation and further specification of auxiliary inputs is to view them as convolution operation over trajectorics. While previous work has viewed history summarization as convolution operation [Mozer 1996), in this work we formulate the convolution over trajectories auxiliary inputs which summarize both histories and potential futures.
In the more specified form, can have multiple functions m', i E {0, (O,..., N 1} which correspond to each of our auxiliary inputs over our history. For the ith auxiliary input at time t, we denote this as m = m.. The set of auxiliary inputs at time is then written as the tuple M, = (m....., m,,"1).
We further specify the function m to allow us to formalize our different approaches to auxiliary inputs. Without loss of generality, we modify our definition of history be a sequence of (o, a) E O A observation- action tuples (we can simply append the next action A, the empty set to the final observation): h = {(Ou, Ao), (O1, A1), (O,, E T. With this, ith auxiliary input at time t, m can be as convolution over the history of preprocessed observation-acee pairs:
m Σ (())):(Or..A))
(10)
where k; is the ith kernel function c: N: + R of the convolution, and g is the preprocessing function applied to the observation-action pair before convolving with the rest of the history. Many auxiliary inputs and memory mechanisms can be defined by the function m.
B.1 Frame Stacking as an Auxiliary Input with Convolutions
As an auxiliary input, the number of previous frames stack corresponds to the number of auxiliary inputs (N 33, as the current frame is accounted for). Our preprocessing function for frame stacking is simply the function that just returns the observation: g(:,, 0z) - Or. The kernel function for i E [O....,2] is defined as
k((T) = 1|r=2---1).
where [[codd] is the indicator function, which is 1 if cond is true, otherwise. With these definitions in place, Equation (10) defines 3 auxiliary imputs at time t, M, = (m...... m Furthermore, based on the convolution of this time-indicator function, each of these inputs correspond to the observation O,-4-1 or the previous observations seen by the agent. M, defines the stack of the last 3 observations, which, in addition to the current. observation, is frame stacking described by Mnih et al. (2015).
16

---

Published in Transactions on Machine Learning Research (05/2023)
B.2 Exponential Decaying Traces with Convolutions
Decaying traces wrillen as convolution over trajectory simply keep exponentially decaying weighted sum of our trajectory observations and actions:
gor,
(11)
r=0
with À < 1. The kernel function (T) in this case is simple exponential function past time steps:
t
kx(T)
(12)
0
otherwise.
Our decaying trace auxiliary inputs are simply convolutions over time with this kernel function. Written an incremental form, we have m = Ami-1 + ge(o,,
B.3 Particle Filtering with Convolutions
Given the mechanism update particles and weights in Section 3.2.2 we form our convolution-basel auxiliary inputs based these weights. In this approach, only have single auxiliary input vector where N 1, which simply denote m,
In this case, our convolution-basse auxiliary inputs are the same as our simplified auxiliary inputs, with m, Mr with the kernel function defined as:
k(T) (1 it
(13)
otherwise.
The preprocessing function is defined by g(o,, = Σt whtlbe O [
B.4 Likelihoods as Convolutions
To represent likelihoods and general value functions convolutions, we simply represent our cumulant as a function of observations and actions like our preprocessing function Ii, with some separate cumulant termination function which is represented with our kernel function kg
C Lobster Fishing Environment and Experiment Details
C.1 Environment Details
As this environment partially observable, we now detail the observation vector o, {0, the agent receives at every time step. We list out ordered true false questions which correspond to the elements (either 0 or 1 respectively) in the observation vector:
0. Is the agent in location 0?
1. Is the agent in location 1?
2. Is the agent in location 2?
3. Is the reward in location observable and missing?
4. Is the reward in location observable and present?
(14)
5. Is the reward in location unobservable?
6. Is the reward in location observable and missing?
7. Is the reward in location observable and present?
8. Is the reward in location unobservable?
We now detail the sources of stochasticity in the Lobster environment. Actions that try to transition between locations in the Lobster environment succeed with probability pslip = 0.6; if the transition fails, the agent.
17

---

Published in Transactions on Machine Learning Research (05/2023)
Collect
Laft
Right
Laft
Right
Right
2
3
Collect
Right
Collects
Left, Collect
Right, Collect
Collct
Left
Right
Right
Lent
Right
Right
Left
5
Right
6
Loft
Collect
Collect
Left, Right. Collect
eftvi...
10
Rigght
Collect
Right
Collect
Leften
11 Left. Collect
12
Rughe
Right, Collect
Collect
Collect
Figure 6: The Lobster Environment MDP. "Slippery" transitions at certain states are not pictured. Green nodes represent states where performing the collect action will yield a reward. Particular only this figure, solid lines are deterministic transit whereas dotted lines are stochastic transitions.
18

---

Published in Transactions on Machine Learning Research (05/2023)
"slips" and stays in the same location. At every time step, if reward is not present, it regenerates according to its own Poisson processes, with an expected number of steps for regeneration of A = 10 for each reward.
We also visualize the full MDP as nodes and edges in Figure
C.2 Hyperparameters and Experimental Setup
We now detail the experimental sctup and hyperparameters swept for all agents mentioned in Section first detail all the shared settings and swept hyperparameters between all algorithms:
We
Learning algorithm: Sarsa(()
Environment train steps: 250K
Function approximator: Linear
Max episode steps: 200
Optimizer: Adam
Step sizes: [10-5 10 10 -404,0 10
Discount rate: ? = 0.9
Because this environment is continuing task, evaluate agents based the undiscounted returns over 200 time steps, with no terminal time steps. All hyperparameter sweeps were done over 30 seeds, with the best hyperparameters decided for each algorithm based on mean undiscounted returns over these 200 time steps over these 250K steps and 30 seeds. The results reported are over 30 different additional seeds, with each additional seed run on the selected hyperparameters. We now briefly describe the two baselines that we employ as comparisons the auxiliary input techniques we introduce, as well as minor details of the auxiliary input algorithms usod this section.
C.3
Algorithmic Details and Additional Results
C.3.1
Observations only
As bascline for performance, consider the observations-only agent. This agent simply uses the Sarsa algorithm to learn policy over the observations described in Equation 14 This agent is labelled in teal "Observations'.
C.3.2 Value Iteration with Environment States
We also consider an optimal agent acting on the fully-observable version of this task, with full knowledge of the transition dynamics. We use this agent see how close to optimal our partially observable agents can perform. We use the transition probabilities over the 12 possible states to perform value iteration (Bellman 1957) to calculate the optimal value functions for control. We iterate through all states until our maximum change in value function over all states A is less than the threshold value 0 = 10-10 10 A < 6. We use this optimal value function over the 12 states together with the transition probabilities to get the optimal policy. We evaluate this optimal policy over 200 steps and collect 1000 runs to get both the mean and standard crror to the mean as the shaded dotted line all learning rate plots.
C.3.3 Exponential Decaying Trace
To use decaying traces as an auxiliary input for the Lobster environment, we take decaying traces of the elements in the observation vector that indicate whether or not each reward is collected. With the indexing and observations from Equation 14 our decaying trace for the Lobster environment is defined as:
Xmi_1 if reward is unobservable, o. otherwise.
m m
(15)
for each auxiliary input e {1,2} that corresponds to rewards in locations 1 and 2, respectively. Note that each auxiliary input hore vector of size R!. As reminder, o, [3]] corresponds to the boolean that answers the question "Is the reward in location observable and missing?" Within our Lobster environment experiments, decay rate of X = 0.9.
19

---

Published in Transactions on Machine Learning Research (05/2023)
1
1
Observations
Observations
0
Particle fiter
0 Likelhood
Normalized
Normalized
State
Right
Right
Left
Left
1
0
01
P(r(L))
P(r(L1)
P(r(L2) 1)
P(r(L2) 1)
0
(b)
The action-values for both left (yellow) and right (blue) actions for the value function learnt over particle filtering auxiliary inputs. Again, the observation-only agent is represented by overlapping diamonds in this plot. bl: The same plots for the action-values for the likelihood auxiliary inputs.
Figure 7:
part of our agent-state function. Our agent state for trace decay auxiliary inputs in the Lobster environment is the concatenation of the observation and the two auxiliary inputs in Mr: = fo,, M,] E R The dimension of this agent state is the number of dimensions of the observation, plus two dimensions from the two auxiliary inputs 9 + 2 = 11.
Put together, auxiliary imputs are defined M, = (m, m2), which we
For the agents described in this section, we use the Sarsa (Rummery & Niranjan 1994 algorithm to learn a control policy (with the exception of the ground-trute state agent, which uses value iteration learn the optimal policy). We also linear function approximation for all agents. For this trace decay agent, a step size of a = 10 was selected from hyperparameter sweep, with epsilon of e = 0.1 for the epsilon-greedy policy.
C.3.4 Particle Filtering
With particle filtering, our approximate belief state becomes more accurate with more particles. In this environment, we instantiate the particle filter with 100 particles to begin with. In the rare case of particle depletion, where there are no particles in the current environment state, we reset all particle weights to be uniform and re-weight them from that time step onwards.
For this particle filtering approach in the Lobster environment, we approximate a distribution over the underlying ground-truth truth states (as per Figure 6), based on the approximated distribution over state over Pparticlss 100 particles. At every step, we follow the steps in Section 3.2.2 to get our approximate belief state our auxiliary input. In this case, since we only have single auxiliary input vector, we have that M, M. In addition to this, our auxiliary inputs in this case define the entire agent state, since we already incorporate both observations and actions at each step in the particle filter update: 2; = M, For this particle filtering agent, once again we leverage the Sarsa control algorithm as well as a selected step size of a - 10-3 (from hyperparameter sweep) and an epsilon of e 0.1.
m,-
We also show similar value function geometry plots to Fig. 2c except for the belief distribution features in Fig. 7a In this case, the bottom two x x-y axes correspond to, for each reward E (1,2):
P(r(L))=1)
Σ m ④ l where reward r(Li) = & a where agent is in location 0], SES
which is the sum of the probabilitics over environment states where the agent in location 0 and the reward in location is present. In this particular visualization, we visualize all the features collected from multiple rollouts of the policy learnt by the particle filtering agent.
20

---

Published in Transactions on Machine Learning Research (05/2023)
C.3.5 Likelihood Predictions
To predict the ground-truth reward regeneralion likelihoods, assumie we know the ground-truth rales of our Poisson processes for both rewards, which in this case is 1 = 1 for both rewards (on average, 1 reward regeneration every 10 steps). With this rate of reward, we calculate the likelihood of a reward being present.
π TO
We now describe how we model this future prediction for the Lobster environment as a likelihood. We do this through calculating, in closed form, the likelihood that r(Li) is present, given that some number of steps have elapsed with r(Li) as unobservable. In this auxiliary input scheme, those number of steps will depend on the number of steps since the agent has seen r(Li) missing and the average number of steps needed to reach Li. In this approach, we assume we have the privileged information of the rate r at which either rewards are regenerated. Let E be the event that the reward at location Li is regenerated within T steps, and E be the complementary event, where the reward does not regenerate after T steps. Since this is a Poisson process, this means that the likelihood a reward at location Li regenerating after T steps is:
- -
(16)
where Equation 16 is simply the probability that at least one Poisson process occurs after I steps. To calculate this prediction for our auxiliary imputs, we need the total number of steps in our trajectory between last observing r(La)) as missing, and the average number of steps to reach Li from the current location. We find this total number of steps by summing these two number of steps, and finding the corresponding likelihood. Our auxiliary imput is then defined as:
mi = 1 exp
1 lio- has r(Li) missing) & (r>last observed r(Li) missing]]
r=0
(17)
Em;
Σ 1=t+1
1l (0..34+22=11
where ni corresponds to the policy of going to location Li (for L1 that is simply the policy that always goes left, for I.2 this policy always goes right). 0, + 2] is the observation feature that answers the question "Is the reward at Li unobservable?". Each auxiliary input defined here corresponds to future prediction about the reward. In this case, our kernel function is simply a function which filters time steps based on if the step was in the past/prreeet or future. Our preprocessing function is then defined by the function that returns the two element vector with the two predicates defined in Equation I7 Put together we get our likelihoods (one for each reward) which we use as auxiliary inputs for our agent.
Similar to our previous two auxiliary inputs, this approach also uses the Sarsa algorithm for control, with step size of = 10-3 selected based on hyperparameter sweep. The epsilon used here was also = 0.1.
C.4 Results and Plotting Details
We now describe the axis and plotting details in Figures 2b and 2c The x y axes (bottom two axes) capture two imput features that represent the likelihood of cach reward being present. For the ground-truth environment states, these features simply represent whether or not the rewards are present or not. Since we have two rewards in locations L1 and L2, this corresponds to four possible environment states at location LO: one for cach possible state of the two rewards (since they can be either present or not present). For the observation-only agent, both of these corresponding features can only take on single value due to partial observability: both features are 0 since at location LO this agent will only ever see () clements for the features which correspond to whether or not the rewards in each location are there. We now describe how we plot the exponential decaying trace features.
From Equation [5 our exponential decaying traces decrease the more time elapsed since last observing each reward was missing. This means that our trace features for each reward should be inverscly proportional to the likelihood of each reward being present. Given this, we plot the complement - (1-m') of each exponential decaying trace input in Figures 2b and 2c
21

---

Published in Transactions on Machine Learning Research (05/2023)
Returns over 200 steps
20
Ground-truth
Trace
Likelihood
avo: 100 ruuss
18
16-
Observations
PF
(avg)
14
12
10
0.0
1.0 Environment steps
2.0
2.5
1e5
Figure 8: Results for PPO agent on the Lobster environment.
C.5 Stochastic Policies with Auxiliary Inputs
We show results for an agent that is able learn stochastic policics the Lobster environment in Figure g The agent utilizes the Proximal Policy Optinization (PPO, |Schulman et al. 2017) algorithm, which is sallcy-gradient-based algorithm that explicitly learns policy parameters to represent stochastic policies. The observation-only agent here (in tcal) performs significantly better than the Sarsa-based observation-only agent (also leal, in Figure 2a due to the ability of the agent to learn a stochastic policy. The stochastic policy learnt by the observation-only PPO agent goes left or right at T.0 with equal probability 0.5, allowing the agent to potentially seek out the reward that was not visited most recently, and is more likely to have regencrated. This stochastic policy achieves a much higher return as compared to the e-greedy policy learnt by the Sarsa algorithm, which will almost always dither between L.0 and either L.1 T,2, but not both. This means the agent essentially has to wait for the reward at either L1 L2 to regenerate. These results also reflect the consensus that policy gradient algorithms generally perform better under partial observability than value-based algorithms- this is most likely due to the fact that stochastic policies allow an agent to perform better in partially observable environments.
From these results, the auxiliary input agents still outperforms the observation-only agent. While the increase may be more marginal as compared to the Sarsa algorithm, auxiliary inputs still increases performance over leveraging observations only. This implies that not all of the gains achieved by auxiliary inputs be achieved by just switching to PPO or stochastic policies-there is additional information agents utilize in auxiliary input features.
Results were run over 100 seeds as opposed to 30 seeds used in the other results in this paper due to the increased stochasticity from the stochastic policy. parameter of 0.95 was used, with n-step returns of == max. episode length used. 1-layer neural network was used as the function approximator. Learning rates and hidden sizes were swept over 10 seeds, with values swept over {10-5, 10-4, 10-3 10-2, 1001) and (5, 10} respectively. The 100 seeds were run over a learning rate of 10 2 and hidden size of 10 selected from the best performance (average reward over max. episode length) over the aforementioned sweep. No replay buffer was used for this agent, with the agent making an update after the roll-out of each episode.
D Particle Filtering Environment and Experimental Details
D.1 Environment Details
Below we consider all the details of the RockSample environment. We do not provide further details of the Modified Compass World environment here because the description in Section 4.1 together with Figure 10a fully specifies the environment.
22

---

Published in Transactions on Machine Learning Research (05/2023)
1.00
hed = 20
Probability
0.75
hed = 5
0.50 L2 distance
10
Half efficiency distance function plots: probability of correct sensor reading as function of distances for different half efficiency distances (Ōhed = 5, 20).
Figure 9:
D.1.1 RockSample Environment Details
We now consider the environment and implementation details of the RockSample environment not considered in Section
The position of each rock is determined in this environment based uniform sampling (without replacement) of 8 positions out of all possible 7 x 7 positions in the grid. This position is determinist defined by the seed which the experiment is run on, and these positions do not change for each agent trained on particular seed. The initial goodness and badness of rocks is determined based on a uniform Bernoulli distribution for every rock at every environment reset.
As mentioned when introducing this environment, the sensor available the agent for checking the goodness and badness of rocks has noise proportional to the L2 distance between the agent and the rock in question, which denote as 8. This noise is essentially based another Bernoulli distribution, where with probability p(o) the sensor returns the correct sensor reading of the rock, and with probability - p(d) the agent receives an incorrect reading. The probability function p is defined by the half efficiency distance Shed of the sensor. Overall, this probability function defined by the function:
p = 0.5 x (1+2 Shea ).
(18)
We plot this function in Figure g We also discuss results for different agents introduced in Section 4 in
RockSample for different half efficiency distances in Section D.3 In our results in Section 4.2 we use a half efficiency distance of 5.
D.2
Environment--pecifne Algorithmic Details and Hyperparameters
We now detail the enviromment--peecfic algorithmic details and hyperparameters swept for all agents pertaining Modified Compass World and RockSample.
D.2.1
Modified Compass World Experimental Setup and Hyperparameters
For our Modified Compass World experiments, we use and sweep the following hyperparametors:
Learning algorithm: Sarsa(0)
Discount rate: 7 = 0.9
Function approximator: Neural Network
Environment train steps: 1M
Layers:
Max episode steps: 1000
Hidden units: 100
Step sizes: [10-3 10-1 10-5]
Optimizer: Adam
Number of particles: 1 for each possible start.
state (9 9 x 323)
23

---

Published in Transactions on Machine Learning Research (05/2023)
B
G
Y
EXIT
F
B
R
(a) Modified Compass World
(b) RockSumple(T, 8)
Figure 10: Modified Compass World and (b) RockSample(7, 8) environments for evaluating approximate
belief states as auxiliary inputs.
Ground-truth
Ground-truth
30
Auxiliary Inputs
30
Auxiliary Inputs
Discounted returns [10 run avgerage)
Discounted returns
(10 run avgerage)
20
LSTM
20
LSTM
10
10
Observations
Observations
-10
-10
0.0
1.0 Environment steps
0.5
1.5 1e6
0.0
1.0 Environment steps
0.5
1.5 1e6
(a) Half efficiency distance dned 20 results.
(b) Half efficiency distance 8ted 5 results.
Figure 11: RockSample results for both dned = 5, 20, In a and mean and standard error to the mean are shown over 10 soods. We use 8hed = 5 for the results in our work.
All hyperparameter sweeps were done over 10 seeds, with the best hyperparameters decided for cach algorithm based on mean discounted returns over these 200 time steps and 10 seeds. We then use these selected hyperparameters and run experiments for 30 different seeds obtain the results presented in Figure 3a For the LSTM agents in this environment, the agents all use one-hot action concatenation (Schlegel et al. 2021 with its input fcatures to the LSTM cell. An ablation study for this action concatenation is done in Appendix D.3.2
Ground-truth Agent In Modified Compass World, the ground-truth agent converges to much higher return than the other agents. This is because it is fully observable and has more information available it in its data stream than the other agents namely the position and orientation of the agent. The learnt policy for this agent traverses directly to the green goal state, without having to localize first.
Obsorvations Only With the observations-oniy bascline, our observation vector defined by a vector of size 5, where each feature corresponds to whether the color directly front of the agent is being observed. An all zero vector represents color being shown in front. of the agent.
24

---

Published in Transactions on Machine Learning Research (05/2023)
Particle Filtering We use particle filtering to approximate belief state over the possible pose and position of the agent. Our feature vector for this approach is of size 7 x 7 x 4, where these features represent all possible combinations of positions and poses of the agent. This position and pose belief state is approximated through the particle filtering approach described in Section 3.2.2 where emission probabilities are simply binary variables representing whether or not each position and pose combination can emit the given color. For this environment, we use one particle for each possible starting position and pose combination, 7 x 7 4, since the environment dynamics are deterministic outside of the initial start state.
Recurrent Neural Network Finally, our RNN-based approach uses the same observations as described in Appendix D.2.1] except with an LSTM as the function approximator. In addition to using a recurrent neural network for function approximation, we also use action conditioning (Schlegel et al.) 2021). y. In our setting, actioning conditioning simply consists of concalenating one-hot encoding of the previous time step's action to the observation vector fed into the RNN, As point of clarification, the LSTM-based agent conditions both the observation and action at every time step.
D.2.2 RockSample(7, 8) Experimental Setup and Hyperparameters
In our RockSample(7, 8) experiments, we leverage replay buffer Lin 1992 for all of our experience. We use and/or sweep the following hyperparameters:
Learning algorithm: (O)sa(O)
Favironment train steps: 1.5M
Function approximator: Neural Network
Max episode steps: 1000
Layers:
Step sizes: [10-3, 10-4, 10-9]
Hidden units: 100
Buffer size: [10K, 100K]
Optimizer: Adam
Nunber of particles: 100
Discount rate: = 0.99
All hyperparameter sweeps were done over 10 seeds, with the best hyperparameters decided for each algorithm based on mean discounted returns over these 200 time steps. We then use these selected hyperparaneters and run experiments for 30 different seeds to obtain the results presented in Figure 3b For the LSTM agents in this environment, the agents also use onc-hot action concatenation (c.f. Appendix D.3.2
Ground-truth Agent For the RockSample ground-truth agent, all the actual moralities (goodness or badness) of the rock are available to the ground-truth agent at the start of an episode. The learnt policy for this agent traverses directly to the good rocks, collects them, then exits to the right.
Observations Only With the observalionn-only baseline, our observation vector is defined by a vector of size 7 7 8 22, where the first + 7 features represent one-hot encodings of the and y coordinates of the agent respectively, and the final 8 features reprosent the most recently observed rock moralities (goodnoss or badness). In our implementation of RockSample, these observed rock moralities are initialized at 0.5, and take on values depending on the most recent check of each rock. So if rock number 1 was checked and seen good 5 steps ago, and this was the most recent check of rock 1, then the feature representing this rock would be a feature, representing a good rock.
Particle Filtering To approximate belief distribution over state, leverage particle filtering mentioned in Section 3.2.2 In this approach, our input feature vector is also defined vector of length 7 7 8. The first ++7 7 features are once again one-hot encoding of the x and y coordinates for the first and second features respectively. The final 8 features are an approximate belief state of the current state of the rock moralities, instcad of the underlying state of the environment. This reduce the dimensionality of the input features. These last 8 features are simply the normalized sum over the weights over all particles. For this particle filtering algorithm, we start with 100 partieles well.
25

---

Published in Transactions on Machine Learning Research (05/2023)
Recurrent Neural Network Finally, our RNN-based approach uses the same observations as Ap- pendix D.2.2 except with an LSTM as the function approximator: Similarly to the Modified Compass World LSTM agent, we also use action conditioning here.
Since we use a replay buffer with LSTMs for this algorithm, to fix a truncation length for T-BPTT, we sample trajectories from our replay buffer of length truncation length, and roll our trajectories out and propagate gradients backwards across the sampled trajectory. This requires us store and sample hidden states from our LSTM in our replay buffer. Similarly to the Compass World LSTM agent, this agent also conditions both the observation and action at every time step.
D.3 Ablation Studies
Here we list the ablation studies we performed over both our environments and select algorithms.
D.3.1 RockSample(7, 8) Half Efficiency Distance Experiment
We perform a small experiment to see the effect of the half efficiency distance dned on the performance of our algorithms in Figures 11a and 11b From these learning curves accurate sensor over distance to rocks) does not significantly affect the performance of the particle filtering auxiliary input, nor does it affect the performance of the ground-truth agent. This parameter scems to affect the LSTM- agent and observalions-only-basse
conclude that lower óred (or less
agent the most, decreasing performance for both. The results presented in the main body of this work use dped =
D.3.2 LSTM Action Concatenation Ablation
We conduct another small ablation study on action concatenation with the LSTM agent on both Modified Compass World and RockSample(7,, 8). Results are shown in Figure 12 Action conditioning seems to generally help the LSTM agent both environments- is because we are conditioning on and providing more information than the LSTM agent not conditioned on actions. We note that the degree in which this action conditioning helps varies from environment lo environment. In Modified Compass World, action conditioning is vital to the performance of the LSTM agent, whereas in RockSample(7, 8) we observe only potential marginal performance increases (potential since we have overlapping standard error bars). This reveals a further point with regards to action conditioning for RNNs: actioning conditioning will help depending on how much knowledge of the action resolves partial observability. In Modified Compass World, where the state is extremely partially observable, knowing the previous action resolves a big portion of the partial obscrvability of the environment, whereas in RockSample, the previous action does not reveal too much about the environment state.
E
Fishing Environment and Experimental Details
E.1 Fishing Environment Details
In this section, we describe the specifies for both Fishing environments, as well as the environment parameters for both Fishing 1 and 2, including the rates of change for currents and the rates of regeneration for rewards.
There are actions in both environments, each corresponding to moving in one of the cardinal directions. Both environments are represented by an x 11 image with locations that emit non-zero rewards, as denoted by the green points. Like in the Lobster environment (c.f. Section 3.1,, in these environments the four rewards regenerate stochastically after being collected. Once collected, the rewards stay and do not disappear without being collected, and the agent receives reward of +1. Currents in these environments also change stochastically over time. The multi-directional arrows represent these shifting currents for cach given position, and the directions in which they might be in. In addition to these sources of stochasticily, the agent also has a chance of "slipping" at every time step and remaining in the same position. Beyond this, there are also walls throughout the environment, denoted by the dark blue tiles, that block the agent from traversing to or through. Bumping into a wall results in a no-op. Finally, Fishing 2 in Figure 4c also has a glass wall denoted by the blue tiles, which the agent can see through, but cannot traverse through. We now
26

---

Published in Transactions on Machine Learning Research (05/2023)
Discounted returns (10 run avgerage)
Ground-truth
0.4
0.3
Auxiliary Inputs
0.2
LSTM (with cond)
0.1
LSTM (no cond)
Observations
0.0
0.0
0.2
0.4
0.6
0.8
1.0 1e6
Environment steps
Modified Compass World LSTM action conditioning ablation.
30
Auxiliary Inputs
30
Discounted returns (10 run avgerage)
Discounted returns
(10 run avgerage)
LSTM (with cond)
20
20
10
10
LSTM (no cond)
Observations
0
Ground-truth
-10-
-10
0.0
0.5
1.0
1.5
0.0
1.0 Environment steps
0.5
1.5
Environment steps
1e6
1e6
(b) RockSample(7, 8) LSTM action conditioning ablation.
(c) Fig. bb. but comparing only action conditioning vs action conditioning.
Action conditioning ablation for LSTM. Results are over 10 seeds for both Modified Compass World and RockSample(7, 8)-
Figure 12:
27

---

Published in Transactions on Machine Learning Research (05/2023)
describe the observations that the agent sees at every step. As for starting position, the agent starts in the (x, v) position (5.5) in both environments.
E.1.1 Mapping and Observations
One part of the partial observability of this environment is the limited observation an agent receives at every step. The agent receives 5 x 5 agent-centric observation veclor al every step. All agents view an accumulated (over time) agent-centric map of these observation vectors, which we call the agent map. For our 11 11 grid world, this amounts to a square observation of length 11+ the grid world account for the agent-centric view when the agent is at the edges of the grid world. At every step, the 5 observable area is updated on the agent map with the given coordinates of the agent. Hence, with every additional step, since rewards and currents change stochastically over time, previously observed (but currently moossevedd missing rewards and currents on the agent map are less and less accurate as time progresses.
21-the dimensions beyond the length 11 of
The agent's view is also obstructed by the walls in the environment. everything behind the wall is obstructed, even if the area was meant to be observable. This true for all walls except for glass obstacles, denoted in blue in Fishing 2. These glass obstacles act as walls, except that the agent is able to see through them, unlike normal walls. Glass walls are placed here so that the agent is able to the direction that the currents are facing within the tunnel to the reward.
an agent is next to a wall, then
The agent observes a lensor of shape 21 21 x 6, where the last dimension indicates the channels for different aspects of the environment/ patronmmnn//bbervatiinn /oservvtion.. The agent is able to see obstacles (walls and glass walls, 1st channel), currents and their direction (next channels), and finally the locations of rewards if they are present (last channel).
E.1.2 Stochasticity in the Environment
Both rewards and currents both environments are defined by Poisson processes, with potentially different. rates correspouding to each reward and each current. We now describe these rates for both environments. All currents depicted with a single-direction arrow denote a current that is static and does not change direction. Note for all currents, when a current is sampled to change, sample uniformly at random from the remaining current directions, excluding the original current direction.
Besides the stochasticity from the Poisson processes, the agent also has a 0.1 probability of "slipping" for a move, when the agent takes an action, there is a probability 0.1 the agent simply stays in place.
Fishing 1 Poisson Processes In Fishing 1, all our stochastic processes have equal rates of regeneration or current flipping. This rate is 60-on or on average, these Poisson processes will activate in expectation over after 60 stops. This larger rate is such that the agent is more incentivized to go collect other rewards, rather than staying at one particular reward and waiting for regeneration.
Fishing 2 Poisson Processes In Fishing 2, rates of reward generation are all set to 50, except for (v.x) coordinates (8,9), which has a rate of regeneration of 100. Note, from here onwards we will list positions as tuples coordinates of (z, y). As for our currents, we group our currents based on their reward regeneration rates (in order from left to right, top to bottom in the grid world):
10:
(0, 6), (0, 7), (2, 5), (2, 6), (4, 1), (5, 7), (5, 8), (5, 10), (8, 7), (9, 7).
20:
(1, 3), (1, 4), (7, 0), (7, 1), (8, 5), (9, 5), (10, 2), (10, 3), (10, 4).
30:
(5, 2), 3), (5, 4).
40:
(0, 2). (), (2, 1), (3, 9). (3, 10), (6, 2), (6, 3), (6, 4), (9, 8), (10, 8).
E.2 Sparing- Algorithmic Details and Hyperparameters
We now detail the algorithmic details and hyperparamoters swept for all agents on the Fishing environments. In both our Fishing experiments, we leverage a replay buffer (Lin 1992) for training. We list the hyperparameters swept for all our algorithms below:
28

---

Published in Transactions on Machine Learning Research (05/2023)
For the Fishing experiments, convolutional neural network to parse agent map tensor. We use and/ /or sweep the following hyperparameters:
Learning algorithm: Sarsa(O)
Discount rate: 7 = 0.99
Function approximation: Convolutional Neu-
Environment train steps: 2M for Fishing 1,
ral Network
12M for Fishing 2
Layers:
Max episode steps: 1000
Hidden size: 64
Step sizes: [10-4,10" 1000 100011107 10
Batch size: 64
Buffer size: 100K
Optimizer: Adam
Evaluation frequency: 2K for Fishing 1, 10K for Fishing 2
All hyperparameter sweeps were done over seeds, with the best hyperparameters decided for each algorithm based on mean discounted returns over the last 100 evaluation steps. Results in this section offline evaluation returns over environment steps. Offline evaluations conducted every evaluation frequency steps (as listed above). We run 5 test episodes per offline evaluation, and also average over these test episodes as well as seeds for a final average return for a given ovaluation step at a cortain training step. We then use these selected hyperparameters and this offiine evaluation to run experiments for 30 different seeds to obtain the results presented in Figures 4b and
E.2.1 Convolutional Neural Network Architecture
We now detail the architecture for our convolutional neural net work All our convolutional layers use a stride of and no padding:
Couppu channels 32, kernel size 10)
Relu activation
(oupput channels hidden size, kernel size 7)
Relu activation
(ouuuut channels = hiiden size, kernel size 1)
Linear layer with output Nlpctione
E.2.2
Convolutional Neural Network LSTM Architecture
Our LSTM implementation is convolutional neural network with an LSTM layer after the convolutional layers:
Cony2Douutput channels 32, kernel size 10)
Relu activation
(ouput channels hidden size, kernel size 7)
Relu activation
CouppD(output channels hidden size, kernel size 1)
Relu activation
Linear layer with output hidden size
LSTMehidden state size hidden size)
Linear layer with output Nlacioons
E.2.3 Exponential Trace Implementation Details
While we describe the approach to using exponential decaying traces for the Fishing environment in Section 5.2 we go into detail here with regards to implementation details and hyperparameters swept.
Our exponential decaying traces are simply another channel in our agent map tensor, with the same size of 15 x 15. It tensor of elements in the range of [0, 1], with each element denoting how long it has been
29

---

Published in Transactions on Machine Learning Research (05/2023)
since observing that particular position (where denotes the agent is currently observing this area, and 0 denoting it has never observed this position).
As for the decay rates, we swept the following rates: [1, 0.95, 0.85, 0.65].
E.2.4 Recurrent Neural Network Implementation Details
With our RNN implementation, we simply the same technique of training an LSTM with replay buffer as in Appendix D.2.2. where we sample trajectorics of length trumcation length for T-BPTT. We swept the following truncation lengths for both Fishing environment hyperparameter sweeps: [1, 5, 10].
30


Reviews:
[{"review_id": "17M9q3hvvL", "reviewer": "Reviewer_usya", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The writing quality is very high, which makes the paper easy to understand.\n- It is a good idea to unify some existing methods as a particular component of reinforcement learners in POMDPS for performing memory featurization, as it may guide other efforts to develop better implementations of this component.\n\n## Weaknesses\n- The results are limited to a small set of environments that appear to have been devised for use in this paper. No experimental results are presented on common RL benchmark environments. This significantly limits the practical takeaways for researchers who seek to improve performance on common benchmarks, or other, more general settings.\n- The results are limited to only using a single RL algorithm, so we cannot draw conclusions about the relative efficacies of different auxiliary input methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper contributes a framework and empirical analysis for RL methods in partially-observed environments that rely on inputs outside of the current observation and action. The paper shows how a few methods in the literature fit into this framework, and analyzes efficacy in a few environments. The results indicate that using exponential traces as input to LSTMs are the most performant in the experiments in the larger environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Section 3.2.1 has a typo / sentence fragment that needs fixed: \"For the Lobster environment, the preprocessing function g corresponds to a 1 if This particular form of history ...\". Presumably, the \"if\" and the \"This\" should belong to difference sentences, with the \"if\"'s continuation currently missing.\n- Eq. 10 seems incorrect. The $\\tau$ summation variable is not used in the equation, and it sums from $t$ to $t$ (i.e. sums once). Please clarify or fix.\n- Please change the labels of \"Auxiliary Inputs\" in Fig, 3 to something that clarifies their source (e.g. \"Particle-Filtering Aux. Inputs\"). In the submission, it is not clear from the Figure alone the method used to create auxiliary inputs.\n- Please justify the absence of particle filtering results in Section 5. These environments seem small enough such that it may be possible to run a particle filter (define the transition and emission distributions). Consider extending the results with a particle filtering input.\n- Please justify the limitation of the results to a single RL algorithm. Consider extending the results by using other RL algorithms, especially by those with existing open-source implementations and reference results on existing benchmarks.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TYymHEX6Jh", "reviewer": "Reviewer_Vzj8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem of how to augment the observation history is important as most real-world decision problems are either partially observable, or the observation dimension is too high to be efficiently modeled by the agent. Heuristic methods used in the augmentation are ad-hoc and lack principled analysis. This paper provides a unified formalism of input augmentation in this setting.\n2. This paper discussed the relationship between the proposed framework and many classical methods to solve partial observability. \n\nWeaknesses:\n\n1. As a new formalism, some definitions in the proposed framework is not clearly or concise enough.\n - 1.1 Is auxiliary input defined at the interface between agent and environment or it is a computation module inside of the agent. What is the difference between auxiliary input and observation and why one cannot be viewed as (part of) the other.\n - 1.2 It brings unnecessary notations and levels of concept to define $M_t$ as N different auxiliary input function $m_t^i$. Given each $m_t^i$ is already a vector function with m dimension, it is mathematically equally representative as the definition of $M_t$.\n - 1.3 It says \"we define auxiliary inputs to summarize the past, present and/or future.\". It need to be explained more how to summarize the future and compute the auxiliary inputs in the test time.\n2. In section 2 the authors define agent-state function as a recurrent function. In section 3, it says \"we investigate the use of auxiliary inputs as an input into the agent-state function.\" However, in later parts from section 3.1 and especially section 3.3, the authors study the use of  auxiliary inputs or observations as an input directly to the state-action value function. Please clarify that either my understanding is wrong or what's the relationship between the proposed framework and the recurrent agent-state function.\n3. The larger scale experiment in fishing domains only shows one form of auxiliary input.\n4. The experiment in fishing domains can be stronger by adding more discussion. There could be more discussion on the choice of preprocessing function and why LSTM + Exp Trace and LSTM learns differently. (LSTM with observation as input has the capacity to model a exponentially weighted summation of $g(O_t)$)\n\nMinor clarity issue:\n - Technically the auxiliary input is not actually another input which provide new information, but an aggregation function of observation-action history. It seems confusing to me why name it as input rather than aggregation or augmentation function.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper gives a new formalism on input augmentation in partially observable sequential decision making problems. They provide a set of rules to construct augmentation from interaction history, under which several common practices of augmenting the input or observations can be viewed as special case. They provide empirical evaluation in two finite-state and a larger pixel-based environment to show the efficacy of these approach. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Regarding the TMLR's evaluation criteria, the weakness point 1, 2 and 4 need to be clarified in the paper. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "858QEwqa8R", "reviewer": "Reviewer_pWPv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. I believe the fundamental framework to be both correct and useful in its generality. It seems to describe fairly well all input augmentation systems I could come up with and recall whilst reading the manuscript. Thus I think it is a valuable contribution, and I welcome efforts to more formally describe some of the commonly-thought-to-be-\"hacky\" steps we apply to our environment-agent problems as RL researchers when attempting to lower the cost of partial observability.\n\n2. The manuscript is well thorough in delineating constraints and structure in the framework, and makes a reasonable case for the presented evaluation settings. It is clear that the authors are attempting to develop an empirical argument bottom up using the presented toy problems such that later work might attempt to reproduce relatively simple (to describe, at least) dynamics in more complex environment-agent tuples.\n\n3. Overall __I__ found the paper and its arguments relatively clear, precise, and not surprising (which IMO in this case is a good property). I reckon an expert would probably not struggle to reproduce the results.\n\n## Weaknesses\n\n1. The paper is not an easy read, which is a shame considering the relatively straightforward contributions and findings. The main culprit seems to be the excessive use of formalism in key sections (2 and 3), which I don't believe strikes the right balance between correctness and clarity; but the later sections that serve to evaluate the framework using some common algorithmic instances are also complex in such a way that makes it hard to map my mental model of the discussed common techniques (which I hope is relatively representative).\n\n2. I found it strange how (a) the manuscript presents the framework as general wrt. (partially observable) agent-environment tuples, but (b) constrains the evaluation to a relatively small model-free agent class (value-based), and as such single agent (SARSA) within it. Given that the experiments seem to be relatively cheap, and that the manuscript's primary narrative seems to be one of analysis, I was expecting a broader empirical discussion, and generally found these choices to make for underwhelming data.\n\n3. Follow up from (2), the experimental setup is just too toy-ish. I strongly believe that use of toy / constrained environments is reasonable and absolute fair game when attempting to exactly study and demonstrate properties, tradeoffs, and emergent behavior of proposed methods in an empirical fashion. However, this particular topic is interesting to analyze and formalize primarily because it tackles the \"hacky\" methods that we as a community kinda winged usage across wildly diverse domains and agents, diverse along a large set of dimensions (complexity, observation types, stochasticity, structure, etc). The lack of empirical results in at least slight-above-toy domains (e.g. ALE) was surprising, and from my perspective reduced the impact of the empirical comparison of types of auxiliary inputs.\n\n4. I think the empirircal results are not really novel. Plenty of previous work has used these auxiliary inputs to effectively make popular problems tractable from an observability perspective, so the results don't seem to be new in any meaningful way. What new information are they trying to communicate?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript presents a framework for describing and categorizing techniques to regress information about the underlying state of the environment in partially observable RL settings in a model-free fashion (i.e. at the agent input level). The manuscript presents a short survey of techniques for augmenting standard agent observations to tackle temporal and dynamical partial observability.\n\nThe authors analyze a few classes of techniques by picking common but representative instantiations of such techniques (decaying traces, particle filters, GVs) to represent cases where such input augmentations may contain -- potentially learnt / estimated -- data about the past, present, or the future wrt. a policy rollout in an environment instance. They evaluate these in a few low dimensional toy environments.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the manuscript would be particularly improved by:\n\n1. Drastically simplifying the description of in the main manuscript, moving some of the verbosity (that to be fair does aid correctness) to the appendix.  Further improvement could also include adding visualizations of the representation of the state as used in some of the more realistic environments for each of the techniques studied.\n\n2. Questioning whether the claim that trace features can work generally work as well or better than recurrent models is sound given the complexity of the presented empirical settings. Would this claim hold with higher dimensional (or even just somewhat larger) input spaces. FWIW I believe the claim to be correct, but that's my own personal bias, rather than being something that I can gather directly from the manuscript as-is :-) Thus, I would consider attempting to compare these methods in \"harder\" (but standard) environments, to strengthen this part.\n\n3. Attempting to use the framework to learn _more_ about these methods, rather than simply reformalizing them into -- what might appear to the inexperienced reader  -- unnecessarily complex descriptions. As it stands, I don't understand the point of the evaluation setting. What are the authors attempting to communicate to the readers? Can the implied expectations (and payoff!) be made clearer in writing? What do we gain from using the framework instead of just hacking away? (Perhaps types of information gain?)\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a united framework for how to construct auxiliary inputs for partially observed RL, connects this to the state construction literature, and shows useful empirical findings that could guide practitioners' choices. 
Decision: Accept (poster)