Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
Authors: Matthew Kyle Schlegel
Volodymyr Tkachuk
Adam M White
Martha White
Venue: Accepted by TMLR ()

Abstract: Building and maintaining state to learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become a key point of interest for the state-building problem, and several large-scale reinforcement learning agents incorporate recurrent networks. While RNNs have become a mainstay in many RL applications, many key design choices and implementation details responsible for performance improvements are often not reported. In this work, we discuss one axis on which RNN architectures can be (and have been) modified for use in RL. Specifically, we look at how action information can be incorporated into the state update function of a recurrent cell. We discuss several choices in using action information and empirically evaluate the resulting architectures on a set of illustrative domains. Finally, we discuss future work in developing recurrent cells and discuss challenges specific to the RL setting.

Published in Transactions on Machine Learning Research (12/2022)
Investigating Action Encodings in Recurrent Neural Networks in Reinforcement Learning
Matthew Schlegel University of Alberta
mischlen@rualberta.ca
Volodymyr Tkachuk University of Alberta
wkachubb@ualberta.cc
Adam White University of Alberta
amuS@ualberta.ccu
Martha White University of Alberta
whitem@uullertto.co
Reviewed on OpenReview: https: // openreview. net/ forum? id= M6
Abstract
Building and maintaining state learn policies and value functions is critical for deploying reinforcement learning (RL) agents in the real world. Recurrent neural networks (RNNs) have become key point of interest for the state-building problem, and several large-scale reinforcement. learning agents incorporate recurrent networks. While RNNs have become a mainstay in many RL applications, many key design choices and implementation details responsible for performance improvements are often not reported. In this work, we discuss one axis on which RNN architectures can be (and have been) modified for use in RL. Spocifically, we look how action information can be incorporated into the state update function of recurrent cell. We discuss several choices in using action information and empirically evaluate the resulting architectures on set of illustrative domains. Finally, we discuss future work in developing recurrent cells and discuss challenges specific to the RL setting.
1 Introduction
Learning to behave and predict using partial information about the world is critical for applying reinforcement learning (RT.) algorithms to large complex domains. For example, a deployed automated spacecraft with a faulty sensor that is only able to read signals intermittently. For the spacecraft to stay in service it noods to deploy a learning algorithm
maintain helpful information (or state) about the history of intermittent sensor readings it relates the other sensors and how the spacecraft is behaving. A game playing system such StarCraft (Vinyals et al. 2019) provides another good example. An agent who plays StarCraft must build a working representation of the map, it's base and strategy, and any information about its rival's base and strategy as focuses its observations on specific locations to perform actions.
Deep reinforcement learning has expanded the types of problems reinforcement learning can be applied to, specifically those with complex observations from the environment (Mnih al. 2015 Vinyals al. 2019 Significant work has gone into engineering non-recurrent networks (Heseel et al. 2018 Espeholt et al. 2018

---

Published in Transactions on Machine Learning Research (12/2022)
while several challenges remain for recurrent architectures in reinforcement learning (Hausknecht and Stone 2015 Zhu et al. 2017 Tgl et al. 2018 Rafiee et al. 2022 Schlegel et al. 2021). There are many design and algorithricc decisions required when applying recurrent architecture to a reinforcement learning problem. We have a more detailed discussion on the open-problems for recurrent agents in Section G
Recurrent neural networks (RNNs) have been established as an important tool for modeling data with temporal dependencies. They have been primarily used in language and video prediction (Mikolov et all 2010 Wang and Cho 2016 Saon et al.) 2017: Wang et al. 2018 Oh et al., 2015). but have also been
used in traditional time-series forecasting (Bianchi et al.) 2017) and RL (Onat et al. 1998 Bakker 2002
Wierstra et al. 2007 Hausknocht and Stone, 2015) Heess et al. 2015). Many specialized architectures have been developed to improve learning with recurrence. These architectures are designed to better model long temporal dependence and avoid saturation including, Long-short Term Memory units (LSTMs) (Hochreiter
and Urgen Schmidhuber 1997). Gated Recurrent Units (GRUs) (Cho et al.) 2014; Chung et al. 2014), Non- saturating Rocurrent Units (NRUs) (Chandar et al. 2019), and others. Most modern RNN architectures integrate information through additive operations.
However, some work has also examined multiplicative which follows from what were known as Second-order
updating (Sutskever et al.] 2011 Wu et al. 2016)
RNNs (Goudreau et al. 1994).
One important design decision is the strategy used to incorporate action in the state update function which can have a large impact on the agent's ability to predict and control (see Figure D This has been noted before, Zhhu et al. (0177) provides discussion the importance of these choices developing ar- chitecture which encodes the action through several layers before concatenating with the observation en- coding. Other types of action encodings have been used for the state update in RNNs for RL (Schaefer ltt al., 2007 Zhu al. 2017 Schlegel et al.) 2021), but without in-depth discussion or focus on the ramifications of the particular choice of architecture. In other cases, action has seemingly been omitted (Oh et al.) 2015 Hausknecht and Stone 2015 Es- peholt et al. 2018). Other state construction ap- proaches also see action as a primary component, predictive representations of state encode predic- tions as the likelihood of seeing action-observation pairs given a history (Littman and Sutton 2002).
0.3
Without action input
Root Mean Squared Value Error
Concatenate previous action to observation
0.1
Separate weight matrix according to previous action
0.0
200 Steps (x1000)
100
300
Figure 1: Learning Curves for various RNN cells in Ring World using experience replay and three strate- gies to incorporate action into an RNN. The agent learns 20 GVF predictions for 300k steps and we re- port root mean squared value error averaged over 50 runs with 95% confidence intervals with window aver- aging over 1000 steps. See Section 5.1 for full details.
Action plays an important role in perception in cog-
nitive sciences. Noë (2004) proposed that perception is dependent on the actions we can take and have taken on the world around us. In effect, one can look at the objective of a reinforcement learning agent as the desire to control and predict the experience (or data) stream, which inevitably means we must model our agency on the data stream. Action has also played an important part in understanding representations (or codings) in the brain through common coding (Prinz) 1990), and in the larger interplay between prediction and action in the brain (Clark 2013). While the RNN architecture is not exactly reminiscent of these cognitive models, the role of action in perception further motivates the need to study the role action plays in an RL agent's 's perceptual system more in-depth.
In this paper, we focus on several architectures for incorporating action into the state function of an RNN in partially observable RL settings. Many of these architectures have been proposed previously for recurrent architectures (i.e. Zhu et al. (017);: Schlegel al. (2021)), and others are either related to or obvious extensions of those architectures. We perform an in-depth empirical evaluation on several illustrative domains, and outline the relationship between the domain and architectures. Finally, discuss future work in developing recurrent architectures designed for the RL problem and discuss challenges specific to the RL setting needing investigation in the future.
2

---

Published in Transactions on Machine Learning Research (12/2022)
2 Problem Setting
We formalize the agent-nnvironment interaction (POMDP). The underlying dynamics are defined by a tuple (S,A, R). Given a state v S and a E A the environment transitions to new state v E S according to the state transition probability ma- trix S A - [0, oo) with reward given by R : A R. The agent observes the sequence O,, d 1, 0++11 d++1.... where the observations
partially observable markov decision processes
lossy function over the state o, d fa(()) R", the actions are selected by the agent's current policy de ~ T(-100, 40, ........................... a,,1, [0, oo), and the reward is rt de (1 11....,./4 1/) E TR.
In this paper we perform experiments in two settings: prediction and control. For prediction, general value functions (GVFs) define the targets (Sutton et al. 2011 White 2015). A GVF is tuple containing cumulant C++1 = fe(or, dz. 0,+1, (+41) E R, continuation function 7++1 = [o Oe, o++1) [0, 1], and a history h = |ao, 01, 01, 02,....... Or] conditioned policy [ar[hh) E [0, oo). The goal of the agent is to learn a value function which estimates the expected cumulative return under π,
E [. ht]
where G dd C441 + 1441
To estimate the value function we use off-policy semi-gradient TD(0) Sutton 1988 Tesauro el al. 1995) For the control sctting learn policy which maximizes the discounted sum of rewards return G2 Ei 7'r++441 In this paper, use Q-learning [Watkins and Dayan 1992 to construct an action-value function and take actions according to an desilon-greedy strategy.
3 Constructing State with Recurrent Networks
For effective prediction and control, the agent requires state representation S, E R that is sufficient statistic of the past: E [G]]s E [Gi]s,, h.]. When the agent learns such state, it can build policies and value functions without the need to store any history. For example, for prediction, it can learn V(s.)
E [e]] In this section, describe the strategies used in this work to learn state.
An RNN provides one such solution to learning and associated state update function. The simplest RNN is one which learns the parameters E IRd recursively
a0xx +b)
where x; [o,, st-1] and a any non-linear transfer function (typically tanh). While concatenating infor- mation (or doing additive operations) has become standard in RNNs, multiplicative operations have also been explored
M
N OUR (o:) + b;
(sr);
Σ
where 0 e [
Using this type of operation was initially called second-order RNNs GGoudrea al. 1994 and was explored in landmark success of RNNs (Sutskever et al. 2011) in a character-level language modeling task.
RNNs are typically trained through the use of back-propagation through time (BPTT) (Mozer 1995). This am effectively unrolls the network through the sequence and calculates the gradient as it was one large network with shared weights. This unrolling is often truncated at some number of steps т. While this alleviates computational-cost concerns, the learning performance can sensitive to the truncation parameter (Pascanu et al. 2013). When calculating the gradients through time for a specific sample, we follow (Schlegel et al. 2021 and define loss
N C.(0) = Σ (u. - )1
3

---

Published in Transactions on Machine Learning Research (12/2022)
Update
Additive
at
W
St
P [|(a|+++n)xxn
z
01+1
t
a O2
St-1
Time
State Update
Multiplicative
W (lcxxin
Zt
at-
R
St-1
a
0+
Ot St-1
Figure 2: Visualizations of the multiplicative and additive RNNs. The dimensions of the weight matrices use the size of the RNN's state |st-1| and the size of the observation |o,1 m.
where is the size of the batch, and y is the target defined by the specific algorithm. This effectively means we are calculating the loss for single step and calculating the gradients from that step only.
There are several known problems with simple recurrent units (and to lesser extent other recurrent cells). The first is known the vanishing and exploding gradient problem (Pascanu et al.) 2013). In this, gradients are multiplied together (via the chain rule in BPTT) the gradient can either become very large vanish into nothing. In either case, the learned networks often cannot perform well and number of practical tricks are applied to stabilize learning (Bengio et al.) 2013). The second problem is called saturation. This occurs when the weights become large and the activations the hidden units are at the extremes of the transfer function. While not problematic for lcarning stability, this can limit the capacity of the network and make tracking changes in the environment dynamics more difficult (Chandar et al. 2019). Because of these issues, several variations the simple recurrent cell have been developed including the LSTMs, GRUs, and NSRUs. We focus experiments around the simple recurrent cells (RNNs) and GRUs.
Finally, to improve sample efficiency we incorporate experience replay (ER), critical part of a deep (recur- rent) system in RT (Mnih et al. 2015 Hausknecht and Stone 2015). There are two key choices here: how states are stored and updated in the buffer and how sequences are sampled (Kapturowski et al. 2018 We store the hidden state of the cell in the experience replay buffer as apart of the experience tuple. This is then used initialize the state when we sample from the buffer for both the target and uon-target networks. We pass back gradients to the stored state to update them along with
model parameters, see full discussion in Section D We also stored separate initial state for the beginning of episodes, which updated with gradients. We slightly differ from the approach taken by Kapturowski et al. (2018), but expect this architectural choice to have little impact on our discussion in this paper. If we sampled the beginning of an episode from the replay we used the most up to date version of this vector to initialize the hidden stale. For sampling, we allowed the agent. sample states across the episode. For samples at the end of the episode, we simply use shorter sequence length than T.
4 Architectural Designs for Incorporating Action
In this paper, we define two broad categories for incorporating action into the state update function of an RNN, and discuss various variations on these idcas (see Figure 2 for visualization of two main architectures).
4.1 Additive
The first category use an additive operation. The core concept of additive action recurrent networks is concatenating action embedding input into the recurrent. cell (Schaefer al.] 2007 Zhu et al.

---

Published in Transactions on Machine Learning Research (12/2022)
2017 For example, the update becomes
= (W*x, + a; +b)
(Additive)
where W and Wa are appropriatcly sized weight matrices. This requires no changes to the recurrent cell if the action embedding a E R concatenated to the observation vector. In the empirical experiments, the additive update cells use a hand-designed one-hot encoding function as all our domains have discrete actions.
A variant of the additive approach was explored in Zhu et al. (2117). where they modified the archilecture slightly to learn function of the action imput a, = fa(a)). In this paper, we use the label Deep Additive for the architecture, where the action encoding function fa is foed-forward neural network. As in their architecture, we concatenate the action embedding with the observation encodings right before the recurrent network. This enables us to focus on the changes in the basic operation rather than enumerating all possible places the action can be concatenated before the recurrent operation.
4.2 Multiplicative
The second category is inspired by socond-order RNNs (Goudrean et al. 1994) and first appeared as a part of a state update function in Rafols et al. (2006), where the observation, hidden state, and action embedding are integrated using a multiplicative operation:
S. = ( X2 Xx X3 a)-
(Multiplicative)
where W E /| al and Xn is the n-mode product, which we detail in Appendix D This type of operation is known to expand the types of functions learnable by a single layer RNN (Goudreau et al. 1994 Sutskever al., 2011). and decreases the networks sensitivity to truncation (Schlegel et al., 2021))
While this type of update has very clear advantages, there is also tradeoff in terms of mumber of parameters and potential re-learning depending on the gramularity of the action representation. For example, in the Ring World experiment above the RNN cell with additíve used 285 paramcters with hidden state size of 15. The multiplicative version would have used 510 parameters with the same hidden state size. While this doesn't 't seem like a lot, if we compare what it would be a domain like Atari (with 18 actions, 1024 inputs, and |s,| = 1024) the number of parameters would be 2 million vs 38 million respectively. As shown below in the empirical study, the size of the state can be significantly reduced when using a multiplicative update. In any case, would be worthwhile to develop strategies to reduce the number of parameters, which discuss next.
4.3 Reducing parameters of the Multiplicative
The first way we can reduce the number of parameters is by using low-rank approximation of the tensor operations. Like matrices, lensors have number of decompositions which can prove useful. For example, every tensor can be factorized using canonical polyadic decomposition, which decomposes an order-N tensor W E R/xxxxx...xx Iv into n matrices as follows
M W Wi.. = Σ
X (L) (22 D1s i...... iy.r
W(N)
where W E R A E R the weighting for factor т, and M is the rank of the tensor. This is a generalization of matrix rank decomposition and exists for all tensors with finite dimensions, see Appendix D for more details. We make several simplifications using the properties of n-mode products. Using the definition of the multiplicative RNN update,
W X3 a RJ Woott (x, Win Darii T Ar. = A,
(Factored)
Previous work explored using a low-rank approximation of a multiplicative operation. A multiplicative update was used to make action-conditional video predictions in Atari (Oh al. 2015). This operation also
5

---

Published in Transactions on Machine Learning Research (12/2022)
Ring World
TMaze
Figure The illustrative environments used in Sec- tion 5.1 and Section 5.2 respectively. (left) The Ring World environment with 6 states depicted, where the observation the agent receives is denoted in each of the circles, available actions denoted by the red ar- rows, and the agent's current location denoted by a double line. (right) The base TMaze environments are depicted with the available actions denoted below and labeled according the Bakker's TMaee and Di- rectional TMaze used in Section 5.2
0
Actions:
+4
Tile 90
0
0
-00
O
Bakker's
Directional
appears in Predietive State RNN hidden state update (Downey et al. 2017 albeit it never performed well as the full rank version. Our low rank approximation is also similar to the network used in [Sutskever et al. (2011), where they mention optimization issues (which were overcome through the use of quasi-secone order methods).
Another approach to reducing the number of parameters required and to reduce redundant learning by using an action embedding rather than a one-hot encoding, For example, in Pong it known that only 5 actions matter. By taking advantage of the structure of the action space we could potentially further reduce the number of parameters required to get these benefits. We explore this architecture briefly in Section E.5 While this is important piece of the puzzle, we do not. focus learning good action embeddings in this paper and leave it to future work.
5 Experiments
In the following sections, we set out to empirically evaluate the three operations for incorporating action into the state update function: No Action input (*NA"), Additive Action ("AA"), Multiplicative Action ("MA"), Factored ("Fac"), Deep Additive Action ("DAA"). We explore all the variants using both stan- dard RNNs and a GRU cell. Our experiments are primarily driven by the main hypothesis that the multi- plicative will strictly outperform the other variants, as suggested by Schlegel et al. (2021). To explore this hypothesis focus two main empirical questions:
How do the different cells affect the properties of the learned value function and internal state of the agent?
Are there examples where the other variants outperform the multiplicative variant?
Question 1:
There are several propertics we are interested in when analyzing the learning capabilitics of our agent. First, and most obvious, prediction error (calculated using root mean squared value error). While error is a reasonable method compare different architectures, Kearney al. (2020) argue only inspecting error can be misleading in the quality of the prediction. To account for this in analysis visually inspect the predictions well to confirm they reasonably modeling the target returns. With respect to the internal state, are primarily interested in understanding if there are qualitative differences which lead to differences in prediction quality.
Question 2:
The second question is more straightforward than the first, and requires a complete empirical investigation of all the variants on set of problems with diverse set of underlying dynamics and characteristics. You can see this question
extension of the hypothesis implied by Figure and Schlegel et al. (2021):
The multiplicative update outperforms the other variants in the reinforcement learning setting for both control and prediction.

---

Published in Transactions on Machine Learning Research (12/2022)
0.3
RMSVE
RNN D DARNN
GRU
0.2
FacRNN AARNN
DAAGRU FacGRU
0.1
AAGR
MAGRU
MARNN
0.0
2
6
10 (T)
14
2
6
10
14
Truncation
Truncation (T)
Figure Ring World sensitivity curves of RMSVE over the final 50k steps for CELL (hidden size) (left)
RNN (15), AARNN (15), MARNN (12), FacRNN (12 [solid] and 15 [dashed]), DARNN (12, |a| 2). and
(right) GRU (12), AAGRU (12), MAGRU (9), FacGRU (9 [solid] and 12 [dashed]), DAGRU (9, |a| 10). Reported results averaged over 50 runs with a 95% confidence interval. FacRNN used factors M {12, respectively, and FacGRU used M = {14, 12). All agents were trained over 300k steps.
While we cannot confirm the above hypothosis empirically, if question 2 is affirmed the hypothesis is false. Counter examples for the hypothesis will also lead to more intuitive knowledge about when to apply one of the above variants.
Other details:
In all control experiments, we e-greedy policy with 0.1. All networks are initialized using uniform Xavier strategy (Glorot and Bengio 2010), with the multiplicative operation independently normalizing across the action dimension (i.e. each matrix associated with an action in the tensor is independently sampled using the Xavier distribution). Unless otherwise stated, we performed a hyperparameter search for all models using a grid scarch over various parameters (listed appropriately in the Appendix(F) F. To bost to our ability we kept the number of hyperparameter settings to be equivalent across all models, except the factored variants which use several combinations of hidden state size and number of factors. The best settings were selected and reported using independent runs with sceds different from those used in the hyperparamcter search, unless otherwise specified. We controlled all the network sizes such that they had an approximately equal number of free parameters. All final network sizes can be found in Appendix
All experiments were run using an off-site cluster. In total, for all sweeps and final experiments we used ~ 20 cpu years, which was approximated based off the logging information used by the off-site cluster. All code for the following experiments can be found at https::/// //gith/btt.t..t. com con/mksscheeg/ wat.le/g/cciooNsss...o and is written in Julia (Bezanson et al. 2017), and we use Flux and Zygote as our deep learning and auto-diff backend [Innes 2018b a).
5.1 Investigating Properties of Predictions and State
We explore the first empirical qeustion by revisiting the Ring World environment, specifically to test model performance with various truncations, and compare the architecture's learned state. The Ring World, depicted in Figure 3 consists of cycle of states with single state containing and other states having an inactive observation bit. The agent can take actions moving either clockwise counter clockwise in the cycle of states. The agent must keep track of how far it has moved from the active bit. For all experiments we use a Ring World with 10 underlying states.
active observation bit,
The agent's objective
learn a total of 20 GVFs with state-tormination continuation functions of γ E (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). When the agent observes the active bit in Ring World (i.e. enters the first state) the predictions terminated (i.e. 7 0.0). The GVFs the observed bit cumulant.
7

---

Published in Transactions on Machine Learning Research (12/2022)
Additive
Multiplicative
Root Mean
Squared
Value Error
Time
Time
Figure 6: Individual learning curves for the additive (hidden size of 15) and multiplicative (hidden size 12) RNNs in Ring World with truncation = 6. The plots are smoothed with a moving average with 1000 step window sizes. The gray box denotes the seed used in Figures 5 and Overall, we see the multiplicative is quite resilient to initialization, but the distance from zero error in Figure D can be explained by a few bad initializations.
Half follow a persistent policy of going clockwise and the other follow the opposite direction persistently. The agent follows an cquiprobable random behavior policy. The agent updates its weights on every step following off-policy semi-gradient TD update with truncation values denoted. We train the agent for 300000 steps and averaged over 50 independent runs. We use root mean squared value error (RMSVE) as the core error metric, which is RMSVE, (V(W)] the true value function.
1 (h+) - Voracle (1x1)12, where Voracle is a known oracle for
Results:
We start with a survey over truncation values for all the archi- tectures in Figure ! For both the RNN and GRU cells the MA variant performs the best, while the additive performs the worst of the cells which include action information. Interestingly, the factored variants for the GRU perform almost identically, while the FacRNN with smaller hidden state perform marginally better. All factored variants straddled the performance of the additive and multiplicative updates. The DAAGRU performs similarly to the AAGRU, while the DAARNN fails to learn in this setting. Finally, the MARNN performs the best overall, only needing a truncation value of r = to learn, which is shorter than the Ring World. We conclude that with the same number of parameters, the operation used to update the state can have significant effect on the required sequence length and final performance.
0.0
Value
Actual MARNN AARNN
0.0
Avey
0.9
1.0
Value
Mww
0.6
To ground the prediction error reported, we present two repre- sentative examples of the learned predictions for the additive and multiplicative RNNs in Figure 5 These plots show a single seed (selected as the best for the additive) over a small snippet of time, but are representative of our observations of the gen- eral performance for both colls. The multiplicative follows the actual prediction within a small delta being as close to zero er- ror as we should expect, while the additive has many artifacts and other miss-predictions for both the myopic (y = 0.0) and long-horizion (y = 0.9) predictions. In Figure 0 we report all the individual learning curves for the additive and multiplicative.
0.4
Time
Figure 5: Ring World predictions of seed 62 for the multiplicative and additive RNNs. Discounts listed with the target pol- icy persistently going cunter-clockwise.
8

---

Published in Transactions on Machine Learning Research (12/2022)
Seed = 62
Truncation 1
Truncation 6
States
Additive
Actions
Truncation 1
Truncation 6
States
Multiplicative
Actions
Figure 7: TSNE plots for the additive and multiplicative RNNs for truncation (1, Given the learning objective (described in Section 5.1 we would want the state to have 10 distinct clusters for each state of the underlying environment. We should expect the truncation = to not be able to produce this kind of state for either cell variant. The learning curves correspond to a single seed (seed=62 which best for the Additive update). The top scatter plots are colored on the underlying state the agent currently in, the bottom scatter plots colored based the previous action the agent took. We initialized TSNE with the same random seed, with max iterations set to 1000, and perplexity set to 30. We present (top) additive and (bottom) multiplicative update functions.

---

Published in Transactions on Machine Learning Research (12/2022)
Median Seed
Truncation 1
Truncation 6
States
Additive
Actions
Truncation 1
Truncation 6
States
Multiplicative
Actions
Figure 8: TSNE plots for the additive and multiplicative RNNs for truncation (1,6). Given the learning objective (described in Section 5.1). we would want the state to have 10 distinct clusters for each state of the underlying environment. We should expect the truncation 7 1 to not be able to produce this kind of state for either cell variant. The learning curves correspond to single seed. The top scatter plots are colored on the underlying state the agent currently in, the bottom scatter plots are colored based on the previous action the agent took. We initialized TSNE with the same random seed, with max iterations set. to 1000, and perplexity set to 30. We present the median seeds for both cells (top) additive uses seed=55 and (bottom) multiplicative uses seed-67.
10

---

Published in Transactions on Machine Learning Research (12/2022)
Looking beyond performance:
A natural question is why might the multiplicative cell perform significantly better than the other cells in this simple setting? One hypothesis is that the multiplicative cell does a better job at separating the histories on action sequence as compared to the additive operation. While this question is difficult to test, we can peer into the learned state of each cell and see if there are qualitative features that appear to help explain the better performance. To do this we take learned agents over different truncation values started using the same seed. After learning (using the same parameters as in Figure
collect another 1000 steps of hidden states. With these hidden states we use TSNE (Van der Maaten and Hinton 2008) to reduce the space of hidden states to two dimensions. The resulting scatter plots for the additive and multiplicative simple RNNs can be seen in Figure 7
Overall, we observe the additive and multiplicative separate on the previous action equally well, matching our initial hypothesis. While action is important, the additive scems to be hyper-focused on action even as the cell is able to partition on environment state. The multiplicative, on the other hand, is able to cluster the hidden states for various environment states together with only minor separation on action as seen in states and 7. It is possible this is natural part of th learning process for both the cells, but the multiplicative is able to cluster the states in less samples. If we look at the median performer (sced=55 and seed=67 for the additive and multiplicative respectively) the additive fails to separate on environment state, while the multiplicative looks similarly to the previous seed.
5.2 Understanding when Action Encoding Does and Does Not Matter
In this section, we investigate learning behavior in two environments with slightly differing properties. The first domains is called TMaze [Bakker 2002), depicted in Figure 3 with size of 10, which was initially proposed to test the capabilities of LSTMs in RL using Q-Learning. The environment is long hallway with T-junction at the end. The agent receives an observation indicating whether the goal state is in the north position or south position at the T-junction (which is randomly chosen at the start of the episode). The agent can take actions in the compass directions. On each step the agent receives a reward of -0.1 and in the final transition receives a reward of 4 or -1 depending if the agent was able to remember which direction the goal was in. The agent deterministically starts at the beginning of the hallway. The observation in the first state is [1, 1] if the goal state is located above the agent and 1, if the goal state is below the agent. In the final state of the hallway the agent reccives [0, 1, as an observation, and everywhere else the observation is [1, 0. 1].
Our control agents are constructed similarly to those used in the Ring World environment. The agent's network is a single recurrent layer followed by a lincar layer. We perform a sweep over the size of the hidden state and learning rales, and selected all variants of a cell type to have the same value. We train our network over 300000 steps with further details reported in appendix F.2] We report the learned policy's performance over the final 10% of episodes by averaging the agent success in reaching the correct goal. We report our results using box and whisker plot with the distribution. The upper and lower edges of the box represent the upper and lower quartiles respectively, with the median denoted by line. The whiskers denote the maximum and minimum values, excluding outliers which are marked.
Shown in Figure (left), all the cells have similar median performance with the GRU (with no action input) performing the best with the least amount of spread. This conclusion is the same across the size of the hidden state, where the multiplicative and factored variants performed poorly (see Appendix for factored results). While this initially suggests the action embedding is not important beyond our simple Ring World experiment, notice the difference in how the environment's dynamics interact with the agent's action. In the TMaze, the underlying position of the agent is affected by only two of the actions (the East and West action), while the North and South actions only transition to a different state at the very end of the maze. Also, the agent's s actions do not affect needs to be remembered, no matter what trajectory the agent sees the meaning of the first observation is always the same. Thus, these results are much less surprising. For example, the multiplicative variants will have to learn the update dynamics multiple times for the North and South actions
11

---

Published in Transactions on Machine Learning Research (12/2022)
GRU
RNN
GRU
RNN
1.0
1.0
% %ouccess
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
NA AA
MA DAA
NA AA MA
DAA
NA AA MA DAA
NA AA MA DAA
Figure 9: (left) Bakker's TMaze box plots and violin plots over the performance averaged over the final 10% with 50 independent runs. Trained over 300k steps with t = 10. All GRUs use state size 6, while RNNs use a state size 20, The deep additive used an action encoding of |a| = 4. (right) Directional TMaze comparison over the performance averaged over the final 10% of episodes with 100 independent runs trained over 300k steps with 12 for CELL (hidden size): RNN (30), AARNN (30), MARNN (18), DARNN (25, a 15), GRU (17), AAGRU (17), MÁGRU (10), DAGRU (15, |a| 8).
To better replicate these dynamics in TMaze we add direc- tion component to the underlying state. For example, many robotics systems must be able to orient and turn to progress in a maze, which we hypothesize actions will be critical for modeling the state. The agent can take an action moving for- ward, turning clockwise, or turning counter-clockwise Instead of the observations only being a function of the position, the agents direction plays critical role. In the first state, the agent receives the goal observation 1, when facing the wall responding to the goal's direction. All other walls have the observation [0, 0J, and when not facing a wall the agent re- ceives the observation [0, 0, 1]. In Directionall the agent is forced to contextualize its observation by the action it takes before or after seeing the observation. We evaluate the stale updates using the same settings as in the TMae with results reported in Figure g (right).
RNN
MA
0.70
Fac
0.55
AA
0.50
15
20
40
89
160
GRU
MA
0.75
Fac
0.60
0.55
122
40
60
80
100
Factors
Figure 10: Sensitivily curves over number of factors AM with standard error for the (top) FacRNN (30) and (bottom) Fac- GRU (17). All agents were trained over 300k steps. See Appendix F.3 for sweeps over different state sizes. We use the data generated by a sweep over the learning rate with 40 runs and compare to the data in fig- urc M The red labels on the x-axis indicate when the network has the same number of parameters the multiplicative.
Now that the agent must. be mindful of its orientation, the ac- tion again becomes a critical component in learning. We see the multiplicative variants outperforming all other variants in this domain. Without action, the GRU and RNN are unable to learn, and even the additive and deep additive versions are un- able to learn in 300000 steps. We also sweep over the number of factors and report the performance compared to the mul- tiplicative and additive variants shown in Figure I0 We found that as the factors increase, generally the performance increases as well. This matches our expectations, as with increased factors the factored variants should better approximate the multiplicative variances. But there is a tradeoff when adding too many factors, causing performance decrease substantially. While the factored variant has some interesting properties, we docide focus the remaining experiments using the base architectures (NA, MA, AA, DA) and report full results with the factored variant in Appendix
12

---

Published in Transactions on Machine Learning Research (12/2022)
TMaze
Directional Taze
Figure 11: Two variants of combining cells. State size chosen based on procodures of pre- vious environments. (top) Performance of success rates (left) TMaze with same basic parameters above for CELL (hidden size); Softmax GRU (6), Cat GRU (6), Softmax RNN (20), Cat RNN (20). (right) Direc- tional TMaze with same parameters as above for CELL (hidden size): Softmax GRU (8); Cat GRU (12), Softmax RNN (15), Cat RNN (22). (Bottom) Average softmax weights of cells over training with standard error over runs.
GRL
RNN
GRU
RNN
0.9
%Success
0.5
0.5
SM
SM
RNNs
RNNS
0.01
-0m
Average Weight
GRU
GRUS
8.05
8.00
.0.2
1009
2000
3000
1080
2000
3000
Step (x100)
Step (x100)
5.2.1
Combining Cell Architectures
In this section, we consider the effects of combining the additive and multiplicative cells through two types of combination techniques. We see these architectures as a minor step toward building an architecture which learns the structural bias currently hand designed.
We combine the hidden state between an additive and multiplicative operation through two techniques. The first is through an element-wie softmax. Both the additive and multiplicative have the same size hidden state (s and 5" respectively), and each element of the hidden states weighted by
o s + e S 8; 0 + TT
where 09 0Th E R". This should learn which cell to use depending on the structure of the problem. The second combination is through concatenating the two hidden state together s = cat(s", ss"). This gives more room for experts to add more state to the different architectures, but in this work we fix the two architectures to have the same stale size.
We compare these combinations to the original architectures in TMaze and Directional TMaze following the same procedure as above. We expect these cells to perform as well as either the additive or the multiplicative (which ever is doing the best in the specific domain). The results can be seen in Figure II Overall, the softmax combination performs similarly or slightly better than the multiplicative version except in the Directional TMaee for the GRUs. In TMaze, concatenating the two states together performed better than the additive and multiplicative cells, but this operation worked slightly worse than the multiplicative in the Directional TMaze. To test the hypothesis that the softmax weighting should emphasize the better cell given domain we show the softmax weighting over the training period. For the TMaze the weightings end being approximately equivalent while the Directional TMaze shows very distinct separation where the multiplicative is weighted significantly more and the additive is continually down- weighted..
5.3 Learning State Representations from Pixels
Finally, we perform an empirical study in two environments with non-binary obsorvations. We are partic- ularly interested in whether the recurrent architectures perform comparably when the observation needs to be transformed by fully connected layers, when the observation is an image. We only use the GRU cells in these experiments. Full details can be found in Appendix F
The first domain we consider is a version of DirectionalTMaeee which uses images instead of bit observations. The agent receives a gray scale image observation on every step size 28 x 28. The agent sees a fully black screen when looking down the hallway, and a half white half black screen when looking at a wall. The agent observes an even (or odd) number sampled from the MNIST (LeCun et al. (010)) dataset when facing the direction of (or opposite of) the goal. The rewards are -1 every step and or -4 for entering the correct
13

---

Published in Transactions on Machine Learning Research (12/2022)
Image Directional TMaze
Average Return
Lunar Lander
Return
%Success
100
-20
0
0.7
60
-100
-200
0.5
-300
0.4
AA MA
DAA NA
AA
MA
DAA
0
100
200
300
400
Steps (10k)
Figure 12: (Ieft) Image Directional TMaze percent success over the final 10% of episodes for 20 runs for CELL (hidden size): AAGRU (70), MAGRU (32), DAGRU (45, |a| = 128). Using ADAM trained over 400k steps, (r) 20. GRU omitted due to prior performance. (center) Lamar Lander average reward over all episodes for CELL (hidden size): GRU (154), AAGRU (152), MAGRU (64), DAGRU (152, |a| = 64) and (r) = 16. (right) Lunar Lander learning curves over total reward. Ribbons show standard error and a window averaging over 100k steps was used. Lunar Lander agents were trained for 20 independent runs for 4M steps.
and incorrect goal position respectively. We report the same statistic in the prior TMaze enviromments, with the environment size set to 6. Notice the hallway size is smaller and the negative reward is larger, this was to speed up learning for all architectures.
Results for the Image DirectionalTMaze can be seen in Figure 12 In this domains, the multiplicative performs quite well, although not as well as in the simple version. The AAGRU is unable to learn in this setting, and the deep additive variant performs slightly better than the additive.
5.4 Learning State Representations from Agent-Centric Sensors
The second domain is partially observable version of the LamarLander-r2 environment from OpenAI Gym Brockman et al. (2016). The goal is to land lander the moon within landing area. Further details and results can be found in Appendix F.5 To make the observation partially we remove the anglular speed, and we filter the angle such that it is 1 if 7.5 < 0 < 7.5 and O otherwise. We report the average reward obtained over all episodes, and learning curves.
As seen in Figure 12 our findings generalize this domain as well. The multiplicative variant improves over the factored (see Appendix F additive, and deep additive variants significantly. In the LunarLander environment the multiplicative learns faster, reaching a policy which receives average 100 total reward per episode. Both the additive and factored eventually learn similar policies, while the standard GRU seems to perform less well (although not statistically significant from the additive variant). The average return is 100 less than some of the best agents on this domains. When we look at the individual median curves we see the agent does this well 50% of the time (see Appendix F). This difference can be explained by the failure start states being more frequent than in the fully observed case.
6 Open Problems for Recurrent Architectures in RL
Recurrent architectures are often taken off-the-shelf from the supervised learning setting for use in rein- forcement learning. While this has been moderately successful, the RL problem poses challenges not often considered by supervised learning. Below we discuss three interesting properties of an RL system, and how they affect learning using recurrent networks.
Practical Online Recurrent Learning:
In reinforcement learning, desirable learn as much about the most recent experience before selecting an action (i.e. to learn online and incrementally). Learning efficiently online enables adapting behavior in real time and scaling to massive data-streams and architectures. This puts pressure the learning
14

---

Published in Transactions on Machine Learning Research (12/2022)
system to update the weights within set amount of time so the system can act (Sutton et al. 2011 White 2015), which is often not a concern in the supervised setting. In settings where an agent must move around its environment independently, the on- computational system can be heavily constrained by the power of the processor as well as limited energy from the battery. An algorithm whose computational and memory complexity scales independently of the sequence length (without the quadratic complexity on size of the network as real time recurrent learning (Williams and Zipser 1989h) and could be applied online- incrementally would be a major breakthrough in using recurrent architectures for RL and computationally constrained systems generally. A detailed discussion on relevant literature is in Appendix A
Active Data Collection Matters:
Imagine an agent in a hallway with recognizable observations only at the beginning and end of the hallway, much like our TMaze environments. The agent must learn a state update which spans at least the length of the hallway. But this is in the best case scenario when the agent prioritizes making it to the end of the hallway. In reality, our agents will randomly explore the hallway until the end, often exlending the length of the sequence the agent needs to learn over. The interaction between the agent's behavior (or exploration) and the difficulty of training under partial observability with recurrent agent is currently unexplored. Active data collection strategies could mitigate the length of long-temporal dependencies, which would show massive improvements in our agent's 's learning efficiency and ability.
To show the potential of active data collection, we will briefly revisit the Directional TMaze. Start with an agent who has learned the base task of the Directional TMaee environment. If we force the agent to take specific actions and to start in specific orientation at the beginning of the episode, we could
1.0
GRU
RNN
0.9
%oucccss
0.8
feasibly teach the agent to artificially extend the horizon of its policy without increasing the length of the training sequence.
0.7-
See Figure [3 for preliminary results of how behavior can ex-
0.6-
tend the horizon of the agent's policy. In this experiment, trained multiplicative agents are paced through a set of inter- ventions. The naive strategy uses epsilon greed after forcing the agent to step forward twice down the hallway. The hand designed sequence, instead guides the agent through series of forced actions to build to the final desired policy. This sim- ple experiment shows the potential for slowly extending the temporal horizon policy without adjusting the truncation value by intervening on the agent's 's behavior.
0.5-
Nalve Strategy
Hand Designed
Naive Hand Strategy
Designed
Figure 13: Average success over the in- tervention taking the go forward activa- tion and starting in the eastward posi- tion. (Naive Strategy) Using the eval- uated intervention over 60k steps for train- ing, (Hand Designed) sequence of hand designed interventions to build up to the fi- nal evaluation intervention over 60k steps.
Insight Beyond Learning Curves:
Learning curves provide little understanding of an agent's learning process and this likely limits algorithmic progress in partially observable settings. Unfortumately, such metrics can't be used to address more complicated ques- tions about the agent and its behavior. While searching for SÔTA is admirable, deeper questions about the internal Iearned structures and behaviors of our agents are often, but not always, ignored. Analyzing the internal dynamics of an agent with recurrent architectures is uniquely challenging in reinforcement learning. Some challenges include (see Appendix B for details):
Generating data for evaluating and analyzing representation learning is especially difficult prob- lem for agents with recurrence. The data generated must be coherent trajectories the agent may potentially experience in the environment, meaning data generating policies must be selected to provide coverage over the space of egvnt-enviromment interactions.
Current tools for analyzing state representations are designed for NLP KKrpathh et al. 2015 Ming cet al. (017)) and are ill-suited for analyzing the link between the environment, the agent's state, and the behavior policy.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Analyzing the behavior of our agents through performance metrics leaves many questions unan- swered: How is the agent behaving to solve the task? When does the agent make long-term decision? In what circumstances might the agent's policy fail?
7 Conclusion
In this paper, we empirically evaluated several strategies for incorporating the previous action into the state update of & recurrent neural network. We demonstrated in several environments from several observation types that. this choice can have a large impact on an RL agent's 's performance for both prediction and control. Our empirical results suggest that the multiplicative operation performs the best even when using a smaller state vector, and the factored and the deep additive versions perform marginally better than the additive versions in most domains.
While the multiplicative seems to be the clear winner on the tested domains, it is important to note not all domains require this architecture. One interesting strategy could be to use the softmax combined colls to decide which cell to use in your final architecture (by looking at the sofumax weighting). One could also imagine an architecture which is able to learn which cell to use conditioned on the history of the agent (see Section E.Ip. Until better architectures for RL are defined this choice is left to system designers. While the additive and deep additive versions under-performed compared to the other encodings, it still out-performed naively using RNNs without action input.
What is apparent in our experiments here and empirical evidence recently gathered the performance of recurrent architectures in the online setting (Rafice et al. 2022, Schlegel et al. 2021 is that the methods and architectures developed and utilized by supervised learning might not be suitable for the reinforcement lcarning problem. This paper uncovered a simple choice can have a large impact, and provides some evidence that the assumptions made in supervised learning might be holding back recurrent architectures in the reinforcement learning setting (see Appendix C for details). Small, focused studies using recurrent agents in controlled experiments will continue to produce insights the limitations of the base algorithms and continue to inspire future algorithm developments.
Acknowledgments
We would like thank the Alberta Machine Intelligence Institute, IVADO, NSERC and the Canada CIFAR AT Chairs Program for the funding for this research, well Compute Canada for the computing resources used for this work. We would also like to acknowledge the anonymous reviewers whose comments have made the paper more clear, and Khurram Javed for his suggestions which lead to the Image Directional TMaze enviromment.
References
B. Bakker. Reinforcement Learning with Long Short-Term Memory. In T. G. Dietterich, S. Becker, and
Z. Ghahramani, editors, Adeances in Neural Information Processing Systems 14, 2002.
R. D. Beer. The dynamics of active categorical perception in an evolved model agent. Adaptive Behavior, 2003.
Y. Bengio, N. Boulanger-Levandowsti, and R. Pascamu. Advances in optimizing recurrent networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2013.
J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1), 2017.
F. M. Bianchi, F. Maiorino, M. C. Kampffmeyer, A. Rizzi, and R. Jenssen. An overview and comparative analysis of recurrent neural networks for short term load forecasting. Sefi:1775504377 2017.
G. Brockman, V. Cheung, Pettersson, Schneider, Schulman, Tang, and W. Zaremba. Openai gym. 1rX1::666.00154. 2016.
16

---

Published in Transactions on Machine Learning Research (12/2022)
S. Chandar, C. Sankar, E. Vorontsov, S. E. Kahou, and Y. Bengio. Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies. Proceedings of the Association for the Adouncement of AT Conference on Artificial Intelligence, 2019.
L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems, 34:15084 15097, 2021.
K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. On the Properties of Neural Machine Translation: -ncoder-Decoder Approaches. In Fighth Workshop on Syntaz, Semantics and Structure in Statistical Translation, 2014.
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation gated recurrent neural networks on sequence modeling. In NewrIPS 2014 Workshop on Deep Learning, 2014.
A. Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 2013.
S. Dohare, A. R. Mahmood, and R. S. Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprinl Xix::22008 08.06325, 2021.
C. Downey, A. Hefny, B. Boots, G. J. Gordon, and B. Li. Predictive State Recurrent Neural Networks. In Advances in Newral Information Processing Systems 30. 2017.
L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, T. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, 2018.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
M. Goudreau, C. Giles, S. Chakradhar, and D. Chen. First-order versus second-order single-layer recurrent neural networks. IEEE Transactions on Neural Networks, 1994.
M. Hausknecht and P. Stone. Deep Recurrent Q-Learning for Partially Observable MDPs. In Association for the Advancement of AT Fall Symposium Series, 2015.
N. Heess, J. J. Hunt, T. P. Lillicrap, and D. Silver. Haser control with recurrent neural networks. arXiv:1512.04: 04455, 2015.
M. Hessel, J. Modayil, H. Van Hasselt, Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Association for the Advancement of AI Conference Artificial Intelligence, 2018.
S. Hochreiter and J. Urgen Schmidluber. Long Short-Term Memcory. Neural Computation, 1997.
M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pages 2117-21 2126. PMLR, 2018.
M. Innes. Don't unroll adjoint: Differentiating ssa-form programs. arXiv:1810.0795.. 2018a.
M. Innes. Flux: Elegant machine learning with julia. Journal of Open Source Software, 2018b.
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukeuoglu. Reinforce- ment Learning with Unsupervised Auxiliary Tasks. International Conference on Representation Learning, 2017.
Jaeger. Adaptive nonlinear system identification with echo state networks. Advances in Neural Information Processing Systems, 15, 2002.
S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
17

---

Published in Transactions on Machine Learning Research (12/2022)
A. Karpathy, J. Johnson, and L. Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint ar wrXiv:1506.0201 2015.
N. R. Ke, A. Goyal, O. Bilaniuk, J. Binas, M. C. Mozer, C. Pal, and Y. Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding. Advances in Neural Information Processing Systems, 31, 2018.
A. Kearney, A. J. Koop, and P. M. Pilarski. What's sa good prediction? challenges in evaluating an agent's knowledge. Adaptive Behavior, page
2020.
D. Kudenko and H. Hirsh. Feature generation for sequence categorization. In Proceedings of the Association for the Advancement of Al Conference on Artificial Intelligence, 1998.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Onlinej. Available: http://yann.lcum cataun.co//edlll/ //miii,,,,, 2, 2010.
M. L. Litiman and R. S. Sutton. Predictive Representations of State. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, 2002.
A. R. Mahmood and R. S. Sutton. Representation search through generate and test. In AAAI Workshop: Learning Rich Representations from Low- Level Sensors, volume 10, 2013.
J. Menick, E. Elsen, U. Evci, Osindero, K. Simonyan, and A. Graves, A practical sparse approximation for real time recurrent learning. arXiv preprint arXi::2006.009 07232, 2020.
T. Mikolov, M. Karafíát, L. Burget, J. Cernocký, and S. Khudanpur. Recurrent neural network based language model. In Eleventh Annuul Conference of the International Speech Communication Association, 2010.
Y. Ming, S. Cao, R. Zhang, Z. Li, Y. Chen, Y. Song, and H. Qu. Understanding hidden memories of recurrent neural networks. In IEEE Conference on Visual Analytics Science and Technology, 2017.
N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. In International Conference Learning Representations, 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540), 2015.
J. Modayil, A. White, and R. S. Sutton. Multi-timescale nexting in a reinforcement learning robot. Adaptive Behavior, 2014.
M. C. Mozer. Induction of multiscale temporal structure. Advances in Neural Information Processing Systems, 4, 1991.
M. C. Mozer. A focused backpropagation algorithm for temporal. Backpropogation: Theory, archileclures, and applications, 137, 1995.
A. Mujika, F. Meier, and A. Steger. Approximating real-time recurrent learning with random kronecker factors. Advances in Newral Information Processing Systems 31, 2018.
A. Noë. Action in Perception. 2004.
J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Conditionnii Video Prediction using Deep Networks in Atari Games. Tn C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28. 2015.
A. Onat, H. Kita, and Y. Nishikawa. Recurrent neural networks for reinforcement learning: architecture, learning algorithms and internal representation. In IEFF International Joint Conference on Neural Net- works, volume 3, 1998.
18

---

Published in Transactions on Machine Learning Research (12/2022)
E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, 2020.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In Interna- tional Conference on Machine Learning, 2013.
W. Prinz. A Common Coding Approach to Perception and Action. In Relationships Between Perception and Action: Current Approaches. 1990.
B. Rafice, Z. Abbas, S. Ghiassian, R. Kumaraswany, R. S. Sutton, E. A. Ludvig, and A. White. From eye- blinks to state construction: Diagnostic benchmarks for online representation learning, Adaplive Behavior, 2022.
E. Rafols, A. Koop, and R. S. Sutton. Temporal Abstraction Temporal-difference Networks. In Y. Weiss, B. Schölkopf, and J. C. Platt, editors, Advances in Neural Informalion Processing Systems 18, 2006.
A. Samani and R. S. Sutton. Learning agent state online with recurrent generate-and-test. orXiv preprint arXiv:2112.0 15296, 2021.
G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall. English conversational telephone speech recognition by humans and machines. In Proceedings of Interspeech, 2017.
A. M. Schaefer, S. Udluft, and H.-G. Zämmermann. A recurrent control neural network for data efficient reinforcement learning. In IEEE International Symposium on Approzimate Dynamic Programming and Reinforcement Learning, 2007.
M. Schlegel, A. Jacobsen, Z. Abbas, A. Patterson, A. White, and M. White. General value function networks. Journal of Artificial Intelligence Research, 2021.
S. Sodhani, S. Chandar, and Y. Bengio. Toward training recurrent neural networks for lifelong learning. Neurul Computation, 32(1):1 35, 2019.
I. Sutskever, J. Martens, and G. Hinton. Generating text with recurrent neural networks. In International Conference on Machine Learning, 2011.
R. S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9 44, 1988.
R. Sutton, Td models: Modeling the world at mixture of time scales. In Machine Learning. Elsevier, 1995.
R. S. Sutton and B. Tanner. Temporal-Differenes Networks. In L. K. Saul, Y. Weiss, and L. Bottou, editors, Advances in Newral Information Processing Systems 17, 2005.
R. S. Sutton, J. Modayil, M. Delp, T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: A Scal- able Real-time urchitecture for Learning Knowledge from Unsupervised Sensorimotor Interaction. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
C. Talloc and Y. Ollivier. Unbiased Online Recurrent Optimizatin. In International Conference on Learning Representations, 2018.
G. Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58 68, 1995.
L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 (11), 2008
H. van Hasselt and R. S. Sutton. Learning to predict independent of span. arXiv preprint arXiv:1508.04553 2015.
H. van Hasselt, S. Madjiheurem, M. Hessel, D. Silver, A. Barreto, and D. Borsa. Expected eligibility traces.
In Proceedings of the Association for the Advancement of Al Conference on Artificial Intelligence, 2021.
19

---

Published in Transactions on Machine Learning Research (12/2022)
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and L. Polosukhin. Attention is all you need. Advonces in Neural Information Processing Systems 30, 2017.
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782), 2019.
T. Wang and K. Cho. Larger-context language modelling with recurrent neural network. In 54th Annual Meeting of the Association for Computational Linguistics, 2016.
Y. Wang, L. Jiang, M.-H. Yang, L.-J. Li, M. Long, and L. Fei-Fei. Eidetic 3d lstm: A model for video prediction and beyond. In International Conference on Learning Represenlations, 2018.
C. J. Watkins and P. Dayan. Q-learning, Machine learning, 8(3-4), 1992.
A. White. Developing predictive approach to knowledge. PhD thesis, University of Alberta, 2015.
D. Wierstra, A. Foerster, J. Peters, and J. Schmidhuber. Solving deep memory pomdps with recurrent policy gradients. In International Conference on Artificial Neural Networks, 2007.
R. J. Williams and D. Zipser. A learning algorithm for continually rumning fully recurrent neural networks. Newral computation, 1989.
Y. Wu, S. Zhang, Y. Zhang, Y. Bengio, and R. R. Salakhutdinov. On Multiplicative Integration with Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29, 2016.
P. Zhu, X. Li, P. Poupart, and G. Miao. On improving deep reinforcement learning for pomdps. arXiv preprinl ar Xiv: 1704. 07978, 2017.
20

---

Published in Transactions on Machine Learning Research (12/2022)
A Learning TongTaampoaa Dependencies from Online Data
Learning long-temporal dependencies is the primary concern of both RL and SL applications of recurrent networks. While great work has been done to coalesce around few potential architectures and algorithms for SL settings, these are often found lacking in the vnline-incremental RL context (Sodhani et al. 2019; Rafice et al. 2022 Schlegel et al. 2021) discussed in section G Not only do agents need to learn from the currently stored data (i.e. in an experience replay buffer), they must also continually incorporate the newest information into their decisions (i.e. update online and incrementally). The importance of learning state from an online stream of data has been heavily emphasized in the past through predictive representations of state (Littman and Sutton 2002), tcmporal-difforeee networks (Sutton and Tanner 2005 and GVF networks (Schlegel et al. 2021). and in modeling trace patterning systems (Rafiee et al. 2022 From
supervised learning perspective, several problems like saturating capacity and catastrophic forgetting are cited as the most pressing for any parametric continual learning system (Sodhani et al. 2019). Below we
suggest a few alternative directions needing further exploration in the RL context.
The current standard in training recurrent architectures in RL is truncated BPTT. This algorithm trades off the ability to learn long-temporal dependencies with computation and memory complexity. Currently, the system designer must set the length of temporal sequences the agent needs to model (as would be needed for truncated BPTT to be effective [Mozer 1995 Ke et al., 2018 Tallec and Ollivier 2018 Rafiee et al, 2022 ). Sctting this length is difficult task, as it interacts with the underlying environment and the agent's 's exploration strategy (see section 6 for more details). As the truncation parameter increases it is known that
the gradient estimates become wildly variant (Pascanu et al. 2013 Sodhani et al. 2019). which can make learning slow.
An alternative to (Iruncated) BPTT is real time recurrent learning (RTRL) Williams and Zipser 1989) Unfortunately RTRL is known to suffer high computational costs for large networks. Several approximations have been developed to alleviate these costs (Tallec and Ollivier 2018 Mujika et al. 2018). but these algorithms often struggle from high variance updates making learning slow. The approximation to the RTRL influence matrix proposed by [Mnnckk et al.] (2020) shows significant promise in sparse recurrent networks, even outperforming BPTT when trained fully online. Ke et al. (2018) propose a sparse attentive backtracking credit assignment algorithm inspired by hippocampal replay, showing evidence the algorithm has beneficial properties of both BPTT and truncated BPTT. The focused architecture was often able to compete with the fully connected architecture on length of learned temporal sequence and prodiction error several benchmark tasks. Another line of search/crede assigument algorithms is generate and test (Kudenko and Hirsh 1998. Mahmood and Sutton 2013: Dohare et al. 2021 Samani and Sutton 2021). These search algorithrs aren't as tied to their initialization as other systems as they intermittently inject randomness into their search to jump out of local minima. Many of these approaches combine both gradient descent and generale and test lo gain the benefits of both. While a full generate and test solution is possible, finding the right heuristics to generate useful state objects quickly could be problem dependent.
Learning long-temporal dependencies through regularizing objectives on the state has shown promise in alleviating the need for unrolling the network over long-temporal sequences. Schlegel et al. (022] use GVFs to make the hidden state of simple RNN predictions about the observations showing potential in lightening the need for BPTT. This approach is sensitive the GVF parameters to use as targets on the state of the network. Predictive state recurrent neural networks (Downey et al.) 2017) combine the benefits of RNNs and predictive representations of state (Littman and Sutton 2002) in single architecture. They show improvement in several settings, but don't explore the model when starved for temporal information in the update. Another approach is through stimulating traces, shown by RRafce et al. (2022), where traces of observations are used to bridge the gap between different stimuli. Instead of traces, an objective which learns the expected trace (van Hasselt et al. 2021) of the trajectory could provide similar benefits as predictive objective. One can even change the requirements on the architecture in lerms of final objectives. [Mzerr [1991] propose to predict only the contour or general trends of a temporal sequence, reducing the resolution considerably. Value functions are another object which takes an infinite sequence and reduces resolution to make the target casier to predict (Sutton 1995
Sutton et al. 2011 Modayil et al. 2014 van Hasselt and
Sutton 2015
21

---

Published in Transactions on Machine Learning Research (12/2022)
It is also possible to reduce avoid the need for BPTT for modeling long-temporal sequences by adjusting the internal mechanisms of the recurrent architecture. Echo-state Networks [Jaeger 2002) are one possible direction. Related to the generate and test idea, echo-state networks rely on a random fixed "reservoir" net- work, where prodictions are made by only adjusting the outgoing weights. Because the recurrent architecture is fixed, no gradients flow through the recurrent connections meaning no BPTT is needed to estimate the gradients. Unfortunately, these networks are dependent on their initializations making them hard to deploy in practice. Mozerr (1995) propose a focused architecture design, where recurrent connections are made more sparsely (even just singular connections). This significantly reduces the computational complexity of RTRL and allows for focused version of BPTT.
Transformers (Vaswani et al. 2017 are a widely used alternative to recurrent architectures in natural language processing, Transformers have also shown some success in reinforcement learning but either require the full sequence of observations at inference and learning time (Mishra et al. 2018 Parisotto al.) 2020 or turn the RL problem into a supervised problem using the full return as the training signal (Chen et al., 2021). Because of these compromises, it is still unclear if transformers are a viable solution to the state construction problem in continual reinforcement learning.
B Insight Beyond Learning Curves
Learning curves showing the agent's performance, usually through episodic return or prediction error, over the agent's lifetime has been the primary method algorithms are compared. Unfortunately, such metrics can't be used to address more complicated questions about the agent and its behavior. While scarching for SOTA is admirable, deeper questions about the internal learned structures and behaviors of our agents are often, but not always, ignored. Analyzing the internal dynamics of an agent with recurrent architectures is uniquely challenging in reinforcement learning.
One challenge is how data is generated. Unlike SL whose data is usually a dataset designed ahead of time, RL generates data through interactions with an environment, whose underlying dynamics are likely unaccessible to the system designer. While randomly generated data, in combination with tools from NLP (Karpathy et al.] 2015 Ming et al. 2017). can give us some insight into how our agent's perform, see section 5.1 extending the analysis to larger domains could leave large parts of the agent-environment interactions unseen.
While we provide some representational analysis in the prediction setting, further results in the control setting would be even more beneficial. Unfortunately, many of the analysis tools we considered require the "correct " target at a given time. In the control sctting, even when the underlying dynamics of an environment can be fully specified (say in lunar lander) a notion of what the right action is at a given time can be extremely difficult. discover. Future work should go into analyzing the link between histories, agent state, environment state, and behavior.
Even when analyzing the behavior of our agent, using the performance metric as the primary measure is deeply flawed. This type of analysis fails to address questions such as: How the agent might be behaving? When does the agent make a long-term decision? In what circumstances might the agent's 's policy fail? Analyzing the agent as a non-linear coupled system with the environment through a series of dynamical equations could lead to further insight on conditions which lead the agent to behave in certain ways or when certain decisions are made by the agent. Beer (2003) develop a scries of questions and experiments to analyze an artificial agent from this perspective in a simple catcher like domain. While these tools would be difficult to apply to real-world problems, using them in simulations could provide useful insight into full description of the agent's 's learned policy.
Because of the above challenges there are several lingering questions about these types of agents left unan- swered in these domains. 1) Under what conditions will the agent's 's policy fail in an environment? 2) How robust are the policies to out-of-distribution events and how does this effect the hidden state? 3) What algorithms do the learning process discover solve the domains reliably? 4) Is the model stable over a long training period or in continual domain? 5) When does the agent. make a decision, and does the agent
22

---

Published in Transactions on Machine Learning Research (12/2022)
stick to this decision? We believe answering these questions and more can lead to better understanding of recurrent agents as well as pathways to better algorithms for training such agents.
C Architectural Choices
Below are several architectural choices we made which should be empirically explored in future work:
Cell architecture: As exemplified in this paper, the architectural choices made in the supervised learning setting may not be the best suited for the RL setting. Here we focused on one simple architectural choice, how to incorporate action into the state, but show sometimes massive improvements in the networks ability to predict and control. Scctions 5.2.1 and E.1 explore this further, but future work should investigate novel RL based cell types.
The woes of the experience replay buffer: Current deep learning, including recurrent architectures, in reinforcement learning include the need for an experience replay buffer. While a learning algorithm which overcomes this limitation would likely be preferable, in the short term cohesive strategies for combining an experience replay with recurrent architectures should be empirically explored. There are two major approaches currently: 1) using the stale traces, or 2) warming up the agent from the beginning (or some number of time stops prior) of an episode (Hausknecht and Stone 2015). We use a third strategy here (using gradient information to refresh the hidden state minimize the objectíve), but found little difference between this and the stale approach. For much more insight and discussion on this choice see [apturowski
ct al. 20180.
Target networks and state: How should we initialize the hidden state for a target network? In this paper, we used the state stored (in the experience replay buffer) for the main model.
To continually learning, or to not: The trade-offs between a continually learning and contiontiy learning agent is extremely important for both recurrent and feed-forward architectures. In this paper, we chose to report results when the agent is still learning, but only stationary domains. Any learning system should track the environment as it changes, have an added problem that the hidden state is an accumulation of the agent's 's entire history, meaning novel observations could irreparably harm the hidden state if it is not reset at regular intervals (i.e. in an episodic domain). This will potentially harm the agent's 's performance for its entire lifetime if it unable to adjust it's weights accordingly. This is not the case in a ferd-forward network, where novel observations will not effect the long-term behavior of an agent in known parts of the state space. This effect understudied in the literature for recurrent agents, but is an important aspect of deploying recurrent RL agents in real world systems.
the agent experiences novel states. Recurrent archilectures
Objectives matter: It is known that some objective functions are more learnable in both the fully and partially observable settings (Mozer 1991 van Hassolt and Sutton 2015). Auxiliary tasks are often also used to augment the networks objective function (Jaderberg et al. 2017). or to constrain the learned state (Schlegel et al. 2021). Which objective functions, auxiliary tasks, and learning algorithms are more best when applied to training recurrent networks?
D Some background on tensors
When introducing the multiplicative update and the factored update, we relied on the weight matrix being 3-tensor rather than a matrix of weights. While this make the notation convenient. in the main paper, it requires some atypical background information. This section provides some background tensors and some low-rank decompasitions of tensors.
The simplest, albeit slightly inaccurate, way to describe and use a tensor is as a multi-dimensional array of numbers (either real or complex) which transform under coordinate changes in predictable ways. In this paper, we consider tensors as multi-dimensional arrays using Einstein summation notation. The ith, jth, kth component of an order-3 tensor will be denoted with lower indices Wiyk e R with associated dimension size denoted with corresponding uppercase letters as W RR
23

---

Published in Transactions on Machine Learning Research (12/2022)
Like matrices, tensors have number of decompositions which prove useful. For example, every tensor can be factorized using canonical polyadic decomposition (CP decomposition), which decomposes order-N tensor W E 12xx1x...IIx into N matrices as follows
R Σ
W Wi
X W h.r
W f2.
W(A) fv.r
r=1
Ar
W
W(N) iN.r
Explicit summation over (1...., RJ.
where W E xR, and R the rank of the tensor. This is generalization of matrix rank decomposition, and exists for all tensors with finite dimensions.
Working with tensors takes bit more care in deciding which fibers (generalization of row and column) the product should be over. One type of product is known as the n-mode product which is defined as follows
(W Xn Viindannt 18i6 inndyy W Wig idhi bodd jj
where E R.l.a. An important property which will be used in this paper is some simplifications we can make when considering n-mode products with their rank decomposition. For simplicity we will simplify an order-3 tensor (W RIJK
Wijk = dr v = v(,M) E R RIXM),
K
J
(W X2 v X3 v
)
W
Wigavig
vik
k=1
K
R
bya
vi,
R
J
Xndir
Σ
bi
Σ
Ckr K 1k
j=1
Σ
Xpair B Ck
Wx2v X3 k AA (v'BoKKC)
>
Similarly to CP decomposition, Tucker rank decomposition can be used to create similar operation. Tucker rank decomposition decomposes order-N lensor W E IR 12x... into N matrices another order-N tensor G E IR Ri x R2.... Rx as follows
R1
R
R
W Wi
Σ
Σ
Σ
(1) (1)
2
(()) ix
ha fa.r2
11
m1m
With similar simplifications to CP decompasition,
24

---

Published in Transactions on Machine Learning Research (12/2022)
(W X222V v X3
W Wesuvis vik
111111
Q
R
Opp
vizviR
k=j==
1=1==1
R
K
Σ bjn biovi's
Σ
Spge Cip
Che 1k
k=1
Q
R
9pgr Cip (v'B)
19 (v
p= g=1=
W W22 X3 X = GXI A X2 (v'B)*
X3 c3(
= - [ x2 (v'B) (v*
One interesting property of this operation is now each of the dimensions can have separately tumed rank, giving the system designer more discrction on where to focus reprosentational resources.
Using a lower rank approximation of multiplicative operation has been derived before several times. A multiplicative update was used make action-conditional video predictions in Atari (Oh et al.) 2015). This operation also appears in lower-rank approximation defined by Predictive State RNN hidden state update (Downey el al. 2017), albeit never performed as well as the full rank version. We find similarly that both factorizations perform below the full tensor version (i.e. the multiplicative). We don't 't report results for the Tucker rank decomposition as it performed similarly to the CP decomposition.
E Further Investigations
E.1
Learning the structural bias
So far, we've focused on architectures which have static architectures, where the agent has no agency in learning the appropriate structure. While this strategy seems to be reasonable as a starting point, in the future an architecture which can learn these different networks would be more desirable. We propose one such architecture here and an initial empirical evaluation of this architecture, leading to discussion on the problematic properties such algorithm might have.
2 Supdate O0, Xx, ar-1)
1/1 [xN(Xi, ar-1)
= 2 psi..
Where fax - Ris a parameterized function which is used create a mixture over the experts state z E Rlel produced by state update function fupdase- Both the gating function and the expert rmn state update function can be arbitrarily constructed. In this section focus the simple RNN update and a feed forward ANN with relu activations and softmax activation on the final layer.
The results are presented in Figure 14 A sweep over various number of experts and simple gating network with a single layer and softmax activation. As compared to the additive and multiplicative the mixture of experts RNN network performs in-between the two networks. The GRU, on the other hand, fails perform well in this domain. This might be related the results seen in section 5.2.1 where both the combined GRUs failed to outperform the multiplicative.
25

---

Published in Transactions on Machine Learning Research (12/2022)
Average Successes
Experts=3
MARNN
Average Successes
0.75
Experts=6
Avarage Successes
0.8
Experts=3
MAGRU
Avarage Successes
Experts=6
0.70
0.63
0.7
0.7
MoE
0.45
0.60
0.5
14
AAGRU
0.6
0.50
AARNN
0.55
25
50
75
100
125
25
50
75
120
25
50
75
300
25
50
75
100
125
Gating Network Size
Gating Network Size
Gating Network Size
Gating Network Size
Average Successes
Experts=12
Average Successes
Experts=24
Average Successes
Experts=12
Experts=24
0.75
0.8
0.70
0.7
0.65
0.60
0.8
Average
0.6
0.35
0.50
50
75
100
125
25
50
75
130
50
75
100
125
25
50
100
125
Gating Network Size
Gating Network Size
Gating Network Size
Gating Network Size
Figure 14: Directional TMaze sweep over size of the gating network (i.e how many units in a single hidden layer with relu activations and an output network w/ softmax output) for (left) RNN and (right) GRU. All experiments follow the procodures from previous results, except the MoE networks use 20 runs.
E.2 TSNEs over Time
In section 5.1 we hypothesized the separation of action faced by the additive agent could have been an artifact of the learning dynamics. To test this hypothesis we created TSNEs for various number steps in the environment for both the additive and multiplicative. The results can be seen figure 15 For the multiplicative we choose [50000, 75000, 100000, 300000] which shows the major learning milestones of the network. For the additive we choose [50000, 150000, 200000, 500000] which goes beyond the original experiment's s time limit and shows the major milestones when the network separates the histories according to state. For 100000 steps of training for the multiplicative we can see similar properties where the actions taken to get to specific states are quite separated. As the number of samples grow, to 300000, we see the states converging to be mostly clustered together regardless of the action taken. The additive version never sees the states converging, where even after 500000 timesteps the actions are still regarded highly by the network.
E.3
Online Setting
In this section, we test to see if our conclusions from the previous sections generalize to the fully online setting. We report some results for Ring World and DirectionalTMarre here, with further results in appendix F.1 and F.3 respectively. For both environments, all applicable settings are the same as in the replay counter parts. The only difference in how the network updated. Instead of sampling from an experience replay, we store a history of the truncation length and update the network on every step using the same semi-gradient updates.
We present the results for the online setting in figure 16 Compared to the replay setting, we see all the variants performed worse across the board. For DirectionalTMaze the AAGRU and MAGRU have a reasonable median performance. The MARNN and FacGRU are the only other cells which have runs reaching good performance, but overall perform poorly. We expect initialization plays a large role in the networks performance and should be investigated. We also see similar trends in Ring World, except the RNN variants outperform the GRUs. Another interesting consequence in the online setting, is the need increase the truncation value and hidden state size to perform reasonably for both Directional and Ring World.
26

---

Published in Transactions on Machine Learning Research (12/2022)
50000
75000
100000
300000
States
Actions
(a) Multiplicative for t 6 and seed=67.
50000
150000
200000
500000
States
Actions
(b) Additive for 7 = 6 and seed=62.
Figure 15: TSNE plots for multiplicative and additive RNNs for various number of training samples.
27

---

Published in Transactions on Machine Learning Research (12/2022)
Average under end
Average under full curve
GRU
RNN
GRU
RNN
0.4
1.0
%6Success
NA (RNN/GRU)
RMSVE
0.8-
0.8
AA (RNN/GRU)
(RNNGGU
0.2
0.6
0.1
0.4
0.4
MA (RNN/GRU)
MA
NA MA Fac
NA AA MA Fac
NA MA Fac
0.0
0
100
200
300
Steps (1000)
Figure 16: Online: (left + middle) Directional TMaze percent success in reaching the goal over the final 10% of episodes with 100 independent runs for CELL (hidden size): RNN (46), AARNN (46), MARNN (27), FacRNN (46) M 24, GRU (26), AAGRU (26), MAGRU (15), FacGRU (26) M 21. (right) Ring World learning curves over RMSVE with 100 independent runs for: RNN (20), AARNN (20), MARNN (15), GRU (12), AAGRU (12), MAGRU (9). Ribbons show standard error and a window averaging over 10k steps was used. Factored variants were excluded for clarity, due to high variance results. All agents were trained over 300k steps.
E.4
Masked Grid World
While the TMaze and DirTMaze give some insight into when different encodings might be preferable, the DirTMaze and Ring World share similar dy- namics in how the actions effect the unobserved state of the MDP. Specifically, there are two actions which effect a state component symmetrically. This prompts the question driving the benefits of the multiplicative update's success, or whether there are other scenarios where the multiplicative does better. We propose a new en- vironment which is a simple grid world with border wrapping. The agent can take a step in all the cardi- nal directions, and observes when it enters a random subset of the states (all aliased together). The goal state is also randomly selected at the beginning of agent's life. This creates random action observa- tion patterns the agent must notice and act on to get to the goal. The border wrapping prevents the agent from moving to a corner of the environment and then going to the goal.
1000
Train
1750
Test
900
1500
800
1250
700
1000
600
7
750
500
MA
500
400
whether this property is
10
15
20
25
30
10
15
20
25
30
1750
1000
1500
800
NA
1250
1000
600
750
MA
500
400
10 15 20 30
20 25
Figure 17: Average number of steps to goal over trun- cation for Masked Grid World left. over the entire learning process right from set of representative states after training. Cell (state size) top RNN (24), AARNN (20), MARNN (10) bottom GRU (24), AA-- GRU (20), MAGRU (10).
confirm the hypothesis that the im- provement with multiplicative update can be meaningful even when the state-action sequences randomly placed in the environment. While the improvement is much less drast than the Ring World and Dir TMaee, the improvement is still significant with standard error bars. Another interesting observation is the difference matters much more for the simple recurrent update than the GRU.
In figure 17
E.5
Further Results for the Deep Additive and Deep Multiplicative Architectures
We provide two more experiments with results reported in figure 18 First we provide results over various action encoding sizes for the Directional TMaze environment using the Deep Additive network from the main paper. Overall, we found the size of the encoding network not make a large difference in the final performance. In effect, this result suggests there is still limitation with the deep additive operation
28

---

Published in Transactions on Machine Learning Research (12/2022)
Action Layers=1
Action Layers=2
Average Successes
Average Successes
UARM
DAHNN
0.9
0.330
0.523
0.8
0.500
0.500
NumHidden
NumHidden
Action L
Average Successes
0.63
BAGRU
0.6
0.00
0.55
0.4 AASRU
NumHidden
MAGRU
DMAGRU
Figure 18: (left) Resulting average success over the last 10% of episodes for various number of hidden units in the action encoding network for the deep additive networks with standard error intervals. Each layer (denoted by the title of the plot) contains the number of hidden denoted by the x-axis. (right) Comparing the deep multiplicative operation with the base cells used in the main text.
that can't be overcome by larger encoding networks. We also provide an experiment in the Directional TMaze for deep multiplicative update. The deep multiplicative update uses the multiplicative update as base cell but first passes the action through a feed forward network like the deep additive network. The results of this network were quite poor overall, likely than being given it as prior information. From these results we decided to abandon the deep multiplicative extension of Zhu et al. (2017). Future work should consider how to better learn action encodings for such a network.
result of having to learn the action encoding rather
29

---

Published in Transactions on Machine Learning Research (12/2022)
MANN
MARNN
TacMARMV: Params AARNN
TacMMRNN: Params MAWW
0.2
0.1
0:1
011
0.1
0.0
0.0
0.0
FacMAGRU AAGRU
FacMAGRU
0.2
0.2
0.2
0.2
022
Figure 20: Truncation sensitivity curves for the Experience Replay sctting in ring world. Results RMSVE and error bars are 95% confidence, as in the main paper.)
F Further Empirical Details
In this section we discuss the experiments from the paper in greater detail. In the following tables the common programming notation (x y: 2) is used to denote an array of elements starting from æ increasing by y until z. For example 2 5) [1,3, 5]. When an operation is performed on an array is done element wise. For example ((1:2:5) [2, 4, 4,3.. All hyperparameters are reported in agent steps (which are
the same environment steps for all domains).
F.1 Ring World
Parameler
Value
Table 19 gives the hyperparameters used in the ringworld ex- periments. We also provide full sensitivity curves over trunca- tion for all cell types and hidden state sizes tested in Figure
Steps
300,000 steps RMSprop 16:33-)
Optimizer
RMSProp 1)
0.1
1.6
20
RMSprop P
0.9
Buffer Size
1000
F.2 TMaze
Buffer Warmup
1000
Batch Size
4
In Figure 21 we provide details of the experience replay of the Bakker's TMaze experiments.
Update freq
4 steps 1000 steps
Target Network Freq Independent Runs
50
F.3
BirectionalTMaze
Figure 19: Ring World Hyperparameters
The experiments presented in the paper used experience replay buffer size of 10000 for all the cells. We also ran exper- iments using an experience replay size of 20000 with similar conclusions. These results (and the associated
parameters) can be found in Figure 22
F.4
Image Directional TMaze
We detail all hyperparameter settings, and give results for different network sizes and truncation values in Figure
F.5 Lunar Lander
We provide all hyperparameter settings (Figure 24h and further results (Figures 25 and 26
30

---

Published in Transactions on Machine Learning Research (12/2022)
Parameter
Value
Steps
300,000 steps
1.0
GRU
RNN
Optimizer
RMSprop
% %uuucess
RMSProp RNN: 7)
0.01 x (2.0( (1:0--1:2:-2))
RMSProp GRU: 7)
0.01 x (2)0(-11:2-0))
0.8
RMSprop
0.99
0.7
Discount y
0.99
Truncation
12
0.6
Buffer Size
10000
0.5
Batch Size
Update freq
4 steps
NA AA MADAA
NA AA MADAA Fac
Target Network Freq
1000 steps
Independent Runs
50
Cell
RMSprop Learning Rate
Hidden State Size
Number of Model Parameters
GRU
0.005
6
214
AAGRU
0.0003125
6
286
MAGRU
0.0003125
6
754
FacGRU
0.0003125
6, MM
21
757
DAGRU
0.00125
4
306
RNN
7.8125e-5
20
584
AARNN
7.8125e-5
20
664
MARNN
7.8125e-5
20
2024
FacRNN
0.0003125
20, M = 40
2064
DARNN
7.8125e-5
20, =
684
Figure 21: TMaae Experience Replay experiments: (top left.) The hyperparameters used across all cells
(bottom) The cell specific hyperparameters (top right) Percent success over the final 10% of episodes.
Same Figure 9
31

---

Published in Transactions on Machine Learning Research (12/2022)
GRU
RNN
GRU
RNN
1.0-
1.0
VoSuccess
%Success
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
DAA
AA
MADAA Fac
NA
MA DAA Fac
AA
DAD Fac
Parameter
Value
Cell
Hidden State Size
Num Weights
Steps
300,000 steps
GRU
0.00125
17
1142
Optimizer
R.Mprrpp
RMSprop r
0.01 2.0 2.0(-11:::2)
AAGRU
0.0003125
17
1295
MAGRU
0.0003125
10
1303
RMSprop P
0.99
FacGRU
0.0003125
15, M 17
1320
Discount y
0.99
DAGRU
0.0003125
15,
1310
Truncation
12
RNN
0.00125
30
1143
Buffer Size
[10000, 20000]
AARNN
0.0003125
30
1233
Batch Size
8
MARNN
7.8125e-5
18
1263
Update freq
steps 1000 steps 100
FacRNN
0.0003125
25, M
1018
Target update freq Independent Runs
DARNN
0.0003125
25,
15
1263
Figure 22: Dircctional TMaze Experience Replay results: (top right) Percent success over the final 10% of episodes for buller size of 10000 (bottom right) for buller size of 20000. Learning rales chosen from best final performance the final 10% of episodes for 20 runs. Results for buffer size of 10000 are over 100 independent runs, with buffer size of 20000 in appendix only over the 20 sceds used for the sweep. (bottom left)) The hyperparameters used across all cells. (bottom right) The cell specific hyperparameters.
32

---

Published in Transactions on Machine Learning Research (12/2022)
Trune:
12
Trune: 12
Hidden Size:
70
32
60
55
45
32
70
132
64
122
112
100
64
132
0.9
0.9
% Success
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
AA
MA DAA
Fac
AA
MA
DAA
Fac
Trune: 20
Trunc: 20
% Success
0.
0.7
0.6
0.6
0.5
0.5
04
04
AA
MA DAA
Fac
AA
MA DAA
Fac
Cell
7 = 12(20)
|8|
+ M
[0]
Parameter
Value
MAGRU
9.76e-4 (9.76e-4)
32
154247
Steps
400,000
MAGRU
9.76e-4 (9.76e-4)
64
223175
Optimizer
R.Spprop
AAGRU
2.44c-4 (0.0625)
70
155201
ADAM
2.0(-20:20)
AAGRU
9.76e-4 (0.0156)
132
225323
ADAM B
0.9, 0.999
FacGRU
0.0625
(2.44c-4)
32, M-
154224
Discount Y
0.99
FacGRU
9.76e-4 (2.44e-4)
64, M-350
223009
Truncation т
12 (20)
FacGRU
9.76e-4 (2.44c-4)
70, 1M-144
155041
Warm up
1000 steps
FacGRU
9.76e-4 (2.44e-4)
132, M-208
225515
Replay Size
50,000
DAGRU
9.76e-4 (9.76e-4)
45, a
150838
Batch size
16
DAGRU
9.76e-4 (9.76e-4)
55, a-64
152022
Target update freq
10000
DAGRU
9.76e-4 (9.76e-4)
60, a=32
151399
Update freq
4
DAGRU
9.76c-4 (9.76e-4)
100, a-128
224263
Runs
20
DAGRU 9.76e-4 (9.76e-4)
112, 64
220935
DAGRU
9.76e-4 (9.76e-4)
122, a=32
223195
Figure 23: Image Directional TMaze: (top) Percent success over final 10% of episodes for the image tmaze for r = 12 and т 20 (labeled). See labels for size hidden state with left being small networks, and right being larger. (bottom left) The hyperparameters used across all cells in Image Directional TMaze (bottom right) The cell specific hyperparameters.
33

---

Published in Transactions on Machine Learning Research (12/2022)
Parameter 4,000,000 steps
Value
Steps
Lunar Lander
Steps before learning starts
1000 steps
Average Return
100
Optimizer
RMSprop
RMSprop
0.1 x 1.6 (0:06(2200:0000 20:2:-6)
64
152
-20
RMSprop
0.99
Discount y
0.99
-60
Truncation
16
Replay Size
100,000
Batch size
32
-100
Target update freq
1000 steps 8 steps
Update frequency
Hidden state learnable
True
NA AA MA DAA Fac
Independent Runs
20
Coll
RMSprop Learning Rate
Hidden State Size (Factors/Action Encoding)
Number of Model Parameters
GRU
0.0003553
154
156,414
AAGRU
0.0001387
152
155,004
MAGRU
0.0003553
64
153,732
FacGRU
0.0001387
152 (M=170)
152,668
FacGRU
0.0003553
100 (M-265)
153,808
FacGRU
0.0003553
64 (M=380)
153,710 716,
DAAGRU
0.000138778
152 (a=64)
182,684
Figure 24: Lunar Lander experimental details: (top left) The hyperparameters used across all cells in Lunar
Lander (bottom) The cell specific hyperparameters (top right) Average final reward over all episodes (same
as figure 12
34

---

Published in Transactions on Machine Learning Research (12/2022)
150
330
Episode Length
302
Episode Length
300
230
250
200
MAGRU GRU DAAGRU
150
MAGRI
Factore-
150
kumkloden- (6, Ractors-265
kumHioden- Pacioes-170
100
1.00x10'
30x11d
5,50x10
Episode
Episode
400
cell CAU (nh 1521
Pog Return (last 10% episodes)
cell AACRU (Ah) 152)
100
coll FACMAGRU (HR trst, Nh: 100. 265)
cell FACMAGRU (irt: Sear, Ah: 152, fac: 175)
Return
-200
200
MAGRU
Numtti deen hactors=2bs
10
10
10"40
36
10
30
10
20
Learning Rate
Steps(x100000)
Figure 25: Lunar Lander further results: (top left) Average final reward over the final 10% of episodes for 20 runs (top middle) Total steps per episode for non-factored cells for 20 runs (top right) Total steps per episode for factored cells for 20 runs (bottom left. learning rate sensitivity curves for 10 runs (bottom middle) Learning curves per episode for non fontoree cells over total reward for 20 runs (bottom right) Learning curves per episode for factored cells over total reward for 20 runs
35

---

Published in Transactions on Machine Learning Research (12/2022)
seed 24
seed 12
seed
seed
seed
seed 26
seed
seed 24
seed 21
seed 16
108
200
190
101
100
300
100
500
390
10
seed 11
seed 26
seed 20
seed
seed
seed
seed
seed
seed
seed 23
150
300
160
300
150
-108
-100
-00
-10
-00
-000
-10
-10
.2e
seed
seed
seed
seed
seed
seed
seed
seed
seed
seed
X0
200
200
200
200
200
200
200
230
203
168
500
163
130
-308
seed
19
seed 25
seed 17
seed 16
seed
seed
seed 19
seed 15
seed 13
seed 28
-106
(a) GRU
(b) MAGRU
seed
seed 18
seed
seed
seed
seed
seed
seed
seed
200
200
290
200
200
200
200
200
780
130
350
seed 25
seed 27
seed 21
seed 29
seed 28
seed 20
seed 24
seed 29
seed 30
seed 21
100
100
100
100
130
123
300
300
300
301
300
200
301
300
290
303
700
700
200
200
200
seed
12
seed 13
seed 26
seed 30
seed 17
seed 27
seed 23
seed
16
seed
12
seed
22
130
100
100
150
100
150
163
(c) AAGRU
(d) DAAGRU
Figure 26: Individual learning curves. Line is the median over 1000 episodes, with the shaded region as the 1st and 3rd quantile over the same window.
36


Reviews:
[{"review_id": "enuBU6O39t", "reviewer": "Reviewer_LrX8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Extensive experiments confirming the observation of Schlegel et al. (2021) of the strength of the multiplicative architecture\n2. Many open research questions identified in the appendix for future work. This is a nice contribution for motivating readers to continue this line of work.\n\n\n**Weaknesses**\n\n3. The core research question introduced in section 5 is informally described as exploring the effect of these architectures on \"learnability\". It is of particular importance to be precise when defining the core research question of the paper.\n\n4. Unclear conclusion on the deep action architecture. The main paper concludes \"deep action versions perform marginally better\" but in appendix E.5 \"From these results we decided to abandon this extension of Zhu et al. (2017)'s deep action.\"\n\n5. The deep action architecture is not explained in Section 4 (where the other architectures are introduced) but then referred to in Section 5.\n\n6. The dimensions m and n in Figure 2 are not defined.\n\n7. There are many instances of using acronyms (BPTT, RMSVE, ER, RTRL) before they are defined. Some are never defined.\n\n8. Environments are referred to before being defined. For example, in Section 5.1 the paper states we are \"revisiting the Ring World environment\" but we haven't yet been introduced to the environment.\n\n9. Figure 6 and 13 are hard to see the important details conveyed in colour due to the size of images. Both take up a whole page with significant white space surrounding the figures. The clarity could be improved by scaling up the images to use the full page.\n\n10. Figure 7 again refers to environments not yet defined, but more significantly I think the caption has the labels for left and right the wrong way around as the results are inconsistent with the text that summarizes them in the main body of the paper.\n\n11. The appendix is a disorganized collection of additional experiments. Some appear to add significant value but are not sufficiently described in context with the rest of the paper and, in the extreme example of the deep action architecture, appear to contradict the main paper. \n\n12. Appendix D appears almost entirely irrelevant, with only the last paragraph in a 2 page section adding some vague new insight in-line with the topic of the paper.\n\n13. Appendix E.1 and E.3 discuss results without referencing which figure they are referring to. Similarly, appendix E.5 refers to \"The results of this network\" ambiguously without clearly referring to (I assume from the following sentence) the deep action architecture.\n\n14. Figure 12 has no legend. What are the blue, purple and pink lines?\n\n15. Appendix E.4 criticizes running experiments in just TMaze and Ring World, but the main paper has results also in lunar lander. This seems like an unfair criticism of the main paper, and it is confusing that this result is pushed to the appendix when it is addressing a weakness the authors have identified in their own paper. \n\n16. The formatting of Figures 16 and 17's captions is unnecessarily confusing given the white space available on the page.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a wide range of experiments supporting the observations of Schlegel et al. (2021) on how to incorporate action into the update of recurrent networks for reinforcement learning.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I believe there are valuable insights amongst the experimental results gathered in this paper, but the current write up does not do them justice. I strongly encourage the authors to (1) consider a major revision of the structure of the paper to include results from the appendix into one clear long format paper; and (2) at least address all formatting issues raised in the weaknesses section of this review.\n\nAdditionally, in discussion with the authors, I am keen to hear:\n\na. A more formal description of the first research question addressed by this paper.\n\nb. A stronger case for the significance of confirming the previously published observation of Schlegel et al. (2021)\n\nc. How this work compliments the work of Kapturowski et al. (ICLR 2019) on \"RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNING.\"", "broader_impact_concerns": "The paper does not include a broader impact statement, but I have no major concerns on its absence in this instance. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sH4eeTDnzw", "reviewer": "Reviewer_hufT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Rigorous empirical evaluation, where many seeds are used and error bars are clearly identified\n- Inclusion of various baselines, such as different variants for each family of methods (additive and multiplicative)\n- Good effort in presenting results that pertain to different kind of environments, going from simpler domains to pixel observations\n\nWeaknesses:\n\n- The empirical evaluation relies on the off-policy setting, yet the algorithms used for all the experiments are not convergent or stable with function approximation. \n- The empirical setting is not very clear, much information about the environments is missing. Similarly information about the number of parameters used by each method is not clearly presented.\n- Related work is not complete and not well presented. It is not clear if previous has investigated a setting very similar to what is presented and what were their conclusions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "State construction is a fundamental problem in RL as the agent's behavior and predictions are based on these state features. Under the POMDP formulation, and more generally under function approximation in the standard MDP setting, high dimensional observations must be processed by the agent in order to make accurate decisions. A promising way to construct these state features is to leverage recurrent neural networks (RNNs) in order to encapsulate the agent's history and better situate itself within the world. Many design choices can be made when considering RNNs, in particular the authors study how incorporate action information, where the action correspond to the one taken by the agent in the previous state. There are essentially two families of methods: one using additive operations and another using multiplicative operations. The authors study these two families, and some variants/approximations of them in the prediction and control settings in smaller environments and larger ones. The results point towards multiplicative methods being more efficient across most settings.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Algorithm choices: It is not clear why the authors decided to use algorithms that are not guaranteed to converge with function approximation when performing their rigorous analysis. The algorithms I could understand that were being used is off-policy semi gradient TD and Q-Learning, which both are unstable due to the deadly triad (function approximation, off-policy and bootstrapping). Given that most environments seem to use function approximation, this really puts a doubt in terms of what we can learn from these experiments, as the usefulness of state construction is mixed with the unstability of the learning algorithm. There exists many algorithms that could be used instead of the two presented here, and for this reason I see this as a major issue. Moreover, why isn't the epsilon in Q-Learning decreased over time as is standard?\n\nThe empirical setting itself is not very clear. The authors assume that the reader will be familiar with the environments proposed, yet this shouldn't be taken for granted. For example, a nice description of Ring World or TMaze could be presented instead of Figure 5 or Figure 4 which honestly are really hard to get something meaningful out of them. In the TMaze and Ring World it is not clear what are the observations.  If the observations are Markov (i.e. not a POMDP), what is the exact role that the RNNs are achieving? \n\nOn another important point, the authors mention that additive methods and multiplicative methods have different numbers of parameters for the same number of hidden states. I would really appreciate a clear presentation of the number of parameters used by each of the methods for the presented experiments. An important question that lies within this is that perhaps a performance improvement is mostly due to the a larger network. It is also not clear what is the difference between TMaze and DirectionalTMaze, is it simply that the agent can now rotate? \n\nThere is also a proposition of combining additive and multiplicative methods, yet it is not clear why this is a proposed method. It seems like multiplicative methods are almost always better, so what can we expect to gain from this?\n\nResults on pixel observation and sensor observations are a good contribution. In the Image DirectionalTmaze it seems like MA is slightly better, but it also suprises me that the AA agent is not able to learn. Could the authors comment if this agent eventually learns? Also, what is the motivation behind the observations given to the agent? I wonder how realistic this scenario is, for example a foraging robot could reasonably have access to a camera which would give it much richer information than the pixels described in the paper. In the Lunar Lander experiments, the authors not that there is no statistical difference between some algorithms. However, look at the plot it seems like there also isn't one when comparing MA to the rest in terms of final performance. Could the authors comment on their choice of analysis?\n\nIn general the work is lacking clarity. A related works section would help a lot in terms of situating the current contribution to past attempts. Some baselines are not explained, that is the DA baseline (or at least I couldn't find an explanation). The presentation of the multiplicative method could be made more accessible. It is required that we look up what n-mode products are, and honestly I have never come across this term up until now. What are the lambdas in the section 4.3?\n\nThere are various typos across the paper that I do not hold against the authors:\nenvironment  section 2\nsignificantly section 4.2\neffect -> affect in the question of section 5, as well a various places later on\nIn this domain, we set the goal is to section 5.1\nsimilarly section 5.1", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vxaXcfzQpG", "reviewer": "Reviewer_7Jc4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have mixed feelings about this paper. In a nutshell, I like a lot the analytical approach taken assumed by the paper, however, in my view, at the current stage the paper may fall short of providing decisive evidence.  \n\nSpecific comments: \n- (again) I consider the approach of taking one axis and studying it in detail valuable and contrasting positively with many sota-obsessed papers\n- Overall, the quality of the text is very good; a nitpick suggestion for the intro is to defer some of the discussion to related work (especially the third paragraph).\n- I found it quite frustrating that the environments are not formally described. I was unsure about my understanding of the Ring World, until I checked the Internet. Even if these are generally known, it would be good to have the proper definition, even if only in the appendix.\n- The environments are my major concerns. I am unsure if the conclusions in the illustrative domains, like Ring World, will transfer to more complex settings. The Lunar Lander experiment indicates that this might be true, it is hard, however, to call it exhaustive. \n- Further, the learning curves in Figure 10, hint at some interesting phenomena. Some of the trainings are faster at the beginning, AA catches up at the end. It would be worth confirming whether these are merely random or general observations. \n- I find Figure 5 uninformative, it is not possible to compare these curves visually; the reader needs to believe the conclusions in the text. The box plots (e.g. Fig 7) are much easier to digest, but even a simple table would do.\n- Overall, I enjoy and consider valuable deeper analysis presented in the papers.\n\n\nI also have a tangential question if the authors have any thoughts on the topic of 'casual confusion', see e.g. [1], which might develop in memory-augmented architectures.  \n\n[1] Causal confusion in imitation learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper empirically studies a particular effect of architecture choices in RL training. More precisely, the question is how action should be embedded into the RNN-type networks in the POMDP setup.\n\nIn short, it advocates for using multiplicative solutions, as they provide faster and more stable training.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See above", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Being somewhat of an empirical study paper, the experimental method is particularly important, and reviewer hufT raises the valid concern that the choice of evaluating these methods with Q-learning and TD(0) may not have been optimal. However, I believe that these are reasonable choices, and improving this would require significant additional work, beyond the scope of a minor revision.

Clarity has been a problem with this paper through the review stage. However, through many revisions addressing the quite extensive feedback from reviewers, it has now improved sufficiently. 

As such, I am recommending "accept as is". If possible, and if the authors so choose, the authors may additionally acknowledge the efforts of the anonymous reviewers towards improving the manuscript.
Decision: Accept (poster)