AMRL: Aggregated Memory For Reinforcement Learning
Authors: Jacob Beck
Kamil Ciosek
Sam Devlin
Sebastian Tschiatschek
Cheng Zhang
Katja Hofmann
Venue:  (2020)
Keywords: deep learning
reinforcement learning
rl
memory
noise
machine learning

Abstract: In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy. We demonstrate that using techniques from NLP and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration. Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, a class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs. Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order. We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time. Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average return by 19% over a baseline that has the same number of parameters and by 9% over a stronger baseline that has far more parameters.

Published as a conference paper at ICLR 2020
AMRL: AGGREGATED MEMORY FOR REINFORCEMENT LEARNING
Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, Katja Hofmann
Microsoft Research, Cambridge, UK Jaccbb Beck@alus l www.wbowww........... Fi Firstname. Firstname.Tattaoe Iastnaneemicrooofft.ccco
ABSTRACT
In many partially observable scenarios, Reinforcement Learning (RL) agents must rely on long-term memory in order to learn an optimal policy. We demonstrate that using techniques from natural language processing and supervised learning fails at RL tasks due to stochasticity from the environment and from exploration. Utilizing our insights on the limitations of traditional memory methods in RL, we propose AMRL, class of models that can learn better policies with greater sample efficiency and are resilient to noisy inputs. Specifically, our models use a standard memory module to summarize short-term context, and then aggregate all prior states from the standard model without respect to order. We show that this provides advantages both in terms of gradient decay and signal-to-noise ratio over time. Evaluating in Minecraft and maze environments that test long-term memory, we find that our model improves average return by 19% over baseline that has the same number of parameters and by 9% over a stronger baseline that has far more parameters.
INTRODUCTION
We address the problem of reinforcement learning (RL) in tasks that require long-term memory. While many successes of Deep RL were achieved in settings that are (near) fully observable, such as Atari games (Mnih et al.] 2015), partial observability requires memory to recall prior observations that indicate the current state. Relying on full observability severely limits the applicability of such approaches. For example, many tasks in virtual and physical environments are naturally observed from a first-person perspective (Oh et al. 2016). which means that an agent may need to seek out and remember task-relevant information that is not immediately observable without directly observing the entire environment. Recent research has started to address this issue, but effective learning in RL settings with long sequential dependencies remains key challenge in Deep RL (Oh al. 2016 Stepleton et al.) 2018 Parisotto & Salakhutdinoy 2018
The currently most common approach to RL in partially observable settings relies models that use memory components that were originally developed for tasks like those that occur in natural language processing (NLP), e.g., LSTMS (Hochreiter & Schmidhuberi 1997) and GRUs (Cho al., 2014). Hausknecht & Stone (0155) first demonstrated benefits of LSTMs in RL tasks designed to test memory, and these and similar approaches have become common in Deep RL (Wang et al., 2016), including multi-agent RL (Rashid et al. 2018 Foerster et al. 2017).
In this work, demonstrate that the characteristics of RL severely impede learning in memory models that are not specifically designed for RL, and propose new models designed to tackle these challenges. For example, LSTMs excel in NLP tasks where the order of observations (characters or words) is crucial, and where influence between observations decays quickly with distance. Contrast this with a hypothetical RI. example where an agent must discover hidden passcode to escape locked dungeon. The order of observations is highly dependent on the agent's 's path through the dungcon, yet when it reaches the door, only its ability to recall the passcode is relevant to escaping the dungeon, irrespective of when the agent observed it and how many observations it has seen since.
Figure [ iillustrates the problem. When stochasticity is introduced to a memory task, even simply as observation noise, the sample efficiency of LSTMs decreases drastically. We show that this problem occurs not just for LSTMs, but also for stacked LSTMs and DNCs (Graves et al., 2016 Wayne et al.) 2018), which have been widely applied in RI., and we propose solutions that address this problem.

---

Published as a conference paper at ICLR 2020
4
4
Total Reward
3
Model
1
AMRL-Max
LSTM
50
100 150
200
250
0 50 100
150
200
250
Step (k)
Step (k)
Figure Our AMRL-Max model compared to standard LSTM memory module (Hochreiter & Schmidhuber 1997) trained on noise-free memory task (T-L, left) and the same task with observa- tional noise (T-LN, right). In both cases, the agent must recall signal from memory after navigating through a corridor. LSTM completely fails with the introduction of noise, while AMRL-Max learns rapidly. (68% confidence interval over runs, as for all plots.)
We make the following three contributions. First, in Section 3 we introduce our approach, AMRL. AMRL augments memory models like LSTMs with aggregators that are substantially more robust noise than previous approaches. Our models combine several innovations which jointly allow the model to ignore noise while maintaining order-variant models maintain informative gradients over very long horizons, which is crucial for sample-efficient learning in long-term memory tasks (Pascanu et al. 2012 Bakker 2001 Wierstra et al. 2009).
variant information as needed. Further, AMRL
Second, in 4 we systematically evaluate how the sources of noise that affect RL agents affect the sample efficiency of AMRL and bascline approaches. We devise series of experiments in two domains, (I) symbolic maze domain and (2) 3D mazes the game Minecraft. Our results show that AMRL can solve long-term memory tasks significantly faster than existing methods. Across tasks our best model achieves an increase final average return of 9% over baselines with far more parameters and 19% over LSTMs with the same number of parameters.
Third, Section 6 we analytically and empirically analyze the characteristics of our proposed and baseline models with the aim identify factors that affect performance. We empirically confirm that AMRL models are substantially less susceptible to vanishing gradients than previous models. We propose to additionally analyze memory models in terms of the signal-to-noise ratio achieved at increasing distances from a given signal, and show that AMRL models can maintain signals over many timesteps. Jointly, the results of our detailed analysis validate our modeling choices and show why AMRL models are able to effectively solve long-term memory tasks.
2 RELATED WORK
External Memory in RL. In the RL setting, work external memory models is most relevant to our own. Oh et al. (2016) introduce a memory network to store a fixed number of prior memories after encoding them in latent space and validate their approach in Minecraft, however the models are limited to fixed-length memories (i.e., the past 30 frames). The Neural Map of PParisotto & Salakhutdinov (2018) similar to our work that provides method in which past events are not significantly harder to learn than recent events. However, it is special-cased specifically for agents on 2D grid, which is more restrictive than our scope of assumptions. Finally, the Neural Turing Machine (NTM) (Graves et al., 2014) and its successor the Differentiable Neural Computer (DNC) Graves al. 2016) have been applied in RL settings. They use an LSTM controller and attention mechanisms to explicitly write chosen memories into external memory. Unlike the DNC which is designed for algorithmic tasks, intentionally stores the order of writes, and induces sparsity in memory to avoid collisions, write memories into order-invariant aggregation functions that provide benefits in noisy environments. We select the DNC, the most recent and competitive prior approach, for baseline comparisons.
Other Memory in RL. A second and orthogonal approach to memory in Deep RL to learn a separate policy network to act memory unit and decide which observations keep. These ap- proaches are generally trained via policy gradient instead of back propagaapaattion through time (BPTT) (Peshkin al. 1999 Zaremba & Sutskever 2015 Young et al. 2018 Zhang al. 2015 Han al.
2

---

Published as a conference paper at ICLR 2020
2019). These approaches are often difficult to train and are orthogonal to work which uses BPTT. The Low-Pass RNN (Stepleton et al. 2018) uses a running average similar to our models. However, they only propagate gradients through short BPTT truncation window lengths. In fact they show that LSTMs outperform their method when the window size is the whole episode. Since we are propa- gating gradients through the whole episode, use LSTMs
baseline instead. Li et al. (2015); Mirowski et al. (2017); Wayne et al. (2018) propose the use of self-supervised auxiliary losses, often relating prediction of state transitions, force historical data to recorded in memory. Along this line, model--ased RL has also made use of memory modules to learn useful transition dynamics instead of learning a policy gradient value function (Ke et al. 2019 Ha & Schmidhuber, 2018)) These are orthogonal and could be used in conjunction with our approach, which focuses on model architecture. Finally, several previous works focus on how to deal with storing initial states for truncated trajectories in replay buffer (Kapturowski et al. 2019 Hausknecht & Stone 2015).
Memory in Supervised Learning. In the supervised setting, there has also been significant work in memory beyond LSTM and GRUs. Similar to our work, Mikollv al. (2014), Oliva et al. (2017). Ostmeeer & Cowell (2019) use a running average over inputs of some variety. Mikolov et al. (2014) use RNN in conjunction with rumning averages. However, MMikolov al. (2014) use the average to provide context to the RNN (we do the inverse), and all use an exponential decay instead of non-decaying average. Additionally, there have been myriad approaches attempting extend the range of RNNs, that are orthogonal to our work, given that any could be used in conjunction with our method Krueger & Memisevic [2016] Belletti et al.] 2018 Trinh et al., 2018). Other approaches for
drop-in replacement for the LSTM component (Le et al.) 2015 Arjovsky al.
2016
de-noising and attention are proposed in (Kolbaek et al.) 2017 Wölmer et al.
2013 Vaswani al.,
2017 Lee et al.) 2018) but have runtime requirements that would be prohibitive in RL settings with
long horizons. Here, we limit ourselves to methods with (1) runtime per step.
METHODS
3.1 PROBLEM SETTING
We consider a learning agent situated in partially observable environment denoted as a Par- tially Observable Markov Decision Process (POMDP) (Kaelbling et al., [998). We specify this process as tuple of AS P,O,R, 7, At time-step I, the agent inhabits some state, 8t E S, not observable by the agent, and receives some observation as a function of the state or Ω ~ (o Ω x S Rzo- O is known as the observation function and is one source of stochasticity. The agent takes some action a, E A. The POMDP then tran- P(s++1ls;,a)) a;):: S x A x S - R2o. and the agent receives reward
sitions state 8141 r: R(s,, o:): x A - I and receives a next observation ot+1 ~ O upon entering 8111. The transition function P also introduces stochasticity. The sequence of prior observations forms an observation trajectory 7. Ω T. To maximize Eeo 7'r,, the agent chooses each discrete ar from a stochastic, learned policy conditioned on trajectories (a, IT):: T x A - [o, 1]. Given P, O, and π itself are stochastic, T; can be highly stochastic, which we show can prevent learning.
3.2 MODEL
In this section we introduce our model AMRI,, and detail our design choices. At high level, our model uses existing memory module to summarize context, and extends it to achieve desirable properties: robustness to noise and informative gradients.
LSTM base model. We start from standard memory model as shown in Figure 2a We use the base model to produce a contextual encoding of the observation o, that depends on the sequence of prior observations. Here, LSTM layer (defined A..
several feed-forward layers, FF, (defined in A.5). followed by an
e, (o.) h ((T The output of the LSTM
// The encoded observation
Previous work proposed a stacked approach that combines two (or more) LSTM layers (Figure 2b) with the goal of learning higher-level abstractions (Pascanu et al. 2013 Mirowski
al. 2017
3

---

Published as a conference paper at ICLR 2020
FF2
FF2
FF2
FF2
LSTM
LSTM
M
In
LSTM
LSTM
FF,
FF,
FF
FF,
b.
1.
(a) LSTM
(b) LSTM_STACK
(c) SET
(d) AMRL
Figure 2: Model Architectures. AMRL (d) extends LSTMs (a) with SET based aggregators (c).
Table I: Definition of our AVG, SUM, and MAX aggregators and their key properties (see text).
Aggregator
Definition
Jacobian
ST Jacobian
SNR
AVG
avg. 2i=0 2t
davg:
(1V
dæ.
SUM
sum, Σl= Dy
dsum: I
I
st/((Var(n))
da;
MAX
max; max
X;
E
1 dmax da;
( (1]mmmaa(()))))
A key limitation of both LSTM and LSTM STACK approaches is susceptibility to noise. Noise can be introduced in several ways, laid out in Section 3.1 First, observation noise introduces variance in the input oN. Second, as motivated by our introductory example of an agent exploring a dungeon, variance on the level of the trajectory Tr is introduced by the transition function and the agent's 's behavior policy. Recall that in our dungeon example, the agent encounters many irrelevant observations between finding the crucial passcode and arriving at the door where the passcode allows escape. This variance in generally produces variance in the output of the function conditioning on T.. Thus. although the first part of our model makes use of LSTM to encode previous inputs, we expect the output, h, to be sensitive to noise.
Aggregators. To address the issue of noise highlighted above, we introduce components designed to decrease noise by allowing it to cancel. We call these components aggregators, labeled M in Figures 2c and 2d] An aggregator a commutative function that combines all previous encodings h; in time-independent manner. Aggregators are computed dynamically from their previous value:
m. = :00=g(mm-- g(m.- he|: } The aggregated memory
where hr]: } denotes the first half of hr, and g() denotes the aggregator function, the choices of which we detail below. All proposed aggregators can be computed in constant time, which results in overall memory model that matches the computational complexity of LSTMs. This is crucial for RL tasks with long horizons.
In this work consider the SUM, AVG, and MAX aggregators defined in Table 1 All three easy to implement in standard deep learning frameworks. They also have desirable properties in terms of gradient flow (Jacobian in Table|aand signal-to-noise ratio (SNR) which we detail next.
Aggregator signal-to-noise ratio (SNR). Our primary design goal is to design aggregators that robust to noise from the POMDP, i.e., variation due to observation noise, behavior policy or environment dynamics. For example, consider the outputs from previous timesteps, h2 to be i.i.d. vectors. If we use the average of all h, our aggregators, then the variance will decrease linearly with t. To formally and empirically assess the behavior of memory models in noisy settings, we propose the use of the signal-to-noie for assessing how well system maintains a given signal, expressed as the ratio between the signal (the information stored regarding the relevant information) and noise. flow, we simply state the SNR that we have analytically derived for the proposed aggregators in Table [ We note that the SNR decays only linearly in time t for SUM and AVG aggregators, which empirically slower than the baselines. and has bound independent of time for the MAX aggregator. We come back to this topic in Section where we describe the derivation in detail, and provide further empirical results that allow comparison of all proposed and baseline methods.
ratio or SNR (Johnson) 2006). The SNR is standard tool
this section, to maintain
4

---

Published as a conference paper at ICLR 2020
Aggregator gradients. In addition to making our model robust to noise, our proposed aggregators can be used to tackle vamishing gradients. For example, the sum aggregator can be viewed as a residual skip connection across time. We find that several aggregators have this property: given that the aggregator does not depend on the order of inputs, the gradient does not decay into the past for fixed-length trajectory. We can show that for given input x; and given output o, the gradient da:;
dor (or expected gradient) of our proposed aggregators does not decay as moves away from t, for
given t. We manually derived the Jacobian column of Table I to show that the gradient does not depend on the index of a given past input. We see that the gradient decays only linearly in t, the current time-step, for the AVG and MAX aggregators. Given that the gradients do not vanish when used in conjunction with h2 as input, they provide an immediate path back to each h, through which the gradient can flow.
SET model. Using an aggregator to aggregate all previous O, yields novel memory model that we term SET (Figure[c).. This model has good properties in terms of SNR and gradient signal as shown above. However, it lacks the ability to maintain order-variant context. We address this limitation next, and include the SET model as an ablation baseline our experiments.
Combining LSTM and aggregator. In our AMRL models, we combine our proposed aggregators with an LSTM model that maintains order-dependent memories, with the goal obtain model that learns order-dependent and order- independnt information in a manner that is data efficient and robust noise. In order achieve this, we reserve certain neurons from h2, as indicated by '/' in Figure 2d We only apply our aggregators to one half of h. Given that our aggregators are commutative, they lose all information indicating the context for the current time-step. To remedy this, we concatenate the other half of h, onto the aggregated memory. The final action is then produced by several feed-foree forward layers, F F2 (defined in A.5):
C = [mm)) // The final contex
ac = FF2(c:) // The action output by the network
Straight-through connections. We showed above that our proposed aggregators provide advan- tages in terms of maintaining gradients. Here, we introduce a further modification that we term straight-through (ST) connections, designed further improve gradient flow. Given that our pro- posed aggregators are non parameetttiic and fairly simple functions, we find that we can deliberately modify their Jacobian decay. Thus, in our ST models, we modify the Jacobian of g and set it to be equal to the identity matrix (as is typical for oon-iifferentiabee functions). This prevents gradients from decaying at all, as seen in the ST Jacobian column of Table
follows. We pass the gradients straight through the model without any
Our proposed models combine the components introduced above: AMRL-Avg combines the AVG aggregator with an LSTM, and uses a straight-through connection. Similarly, AMRL-Max uses the MAX aggregator instead. Below we also report ablation results for SET which uses the AVG aggre- gator without an LSTM or straight-through connection. In Section we will see that all components of our model contribute to dramatically improved robustness to noise compared to previous models. We further validate our design choices by analyzing gradients and SNR of all models in Section 6
4 EXPERIMENTS
We examine the characteristics of all proposed and baseline models through a series of carefully constructed experiments. In all experiments, an RL agent (we use PPO (Schulman et al., 2017), see Appendix A.5 for wyper-parameter proposed by Bakker (200I) and Wierstra et al. (2009) in order to evaluate long term memory in RL agents. Visual Minecraft mazes have been used to evaluate fixed-length memory in Oh et al. (016))
interacts with maze-like environment. Similar mazes were
We compare agents that use one of our proposed AMRL models, or one of the baseline models, as drop-in replacement for the combined policy and value network. (A single network computes both by modifying the size of the output.) Baselines are detailed in Section 4.3
Note that fe is applied to the vector m.. Although the sum and max aggregator can be computed using the function ((z, y) sum(a, y) and g(z,y) y) maz(I, y) respectively, the average must divide by the time-step. Thus, we set fx)x) & for the average aggregator and fe( = x otherwise.
5

---

Published as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(f)
(e)
Figure 3: Overview of tasks used our experiments: (a) Length-10 10 variant of TMaz (the full task has length 100); (b) 3-room variant of MC-LS (Full Minecraft tasks have 10 or 16 rooms.); (c) MC-LSO; (d) original (MC-LS(O) and (e) noisy (MC-LSN) Minecraft observation; (e) sample optimal trajectory (left-right, top-down) through a 1-room variant of MC-LSO.
4.1 TMAZE TASKS
In this set of tasks, an agent placed in a T-shaped maze, at the beginning of a long corridor, as shown in Figure B(a) (here green indicates both the start state and the indicator color). The agent must navigate to the end of the corridor (purple) where it faces binary decision task. It must step left or right according to the start indicator it observed which requires memory to retain. At each time-step, the agent receives its observation as a vector, encoding whether the current state is at the start, in the corridor, or at the end. the start state, the color of the indicator is also observed.
Our experiments use the following variants of this task (see Appendix A. for additional detail);
TMaze Long (T-L) Our base task reduces to single decision task: the agent is deterministically stepped forward until it reaches the end of the corridor where it must make a decision based on the initial indicator. Corridor and episode length 100. Reward for the correct action, and -3 otherwise. This task eliminates exploration and other noise as a confounding factor and allows us to establish base performance for all algorithms.
TMaze Long Noise (T-LN) To test robustness to noise, observations are augmented by random variable n E {-1, sampled uniformly at random. The variable n is appended to the obervation, which is vector-valued. Other details remain as in T-L.
T'aze Long-Short (T-LS) Our hardest TMaze task evaluates whether additional short-term tasks interfere with memory model trying to solve long-tem memory task. We add intermediate task: we append n E [-1, }}, sampled uniformly at random, to the input and only allow the agent to progress forward if its discrete action {-1,1} matches. Corridor length is still 100. A maximum episode length of 150 is imposed given the introduction of exploration.
4.2 MINECRAFT MAZE TASKS
In order to test that our approach generalizes to high dimensional input, we create multiple Minecraft environments using the open-source Project Malmo (Johnson et al. 2016). Compared to the previous setup, we replace fully-connen colle-connected FF, layers with convolutional layers to allow efficient feature learning in the visual space (see Appendix A.5 for details). As in Oh et al. (2016), we use discrete actions. We allow movement in: (North, East, South West). We use the following task variants:
MC Long-Short (MC-LS) Our first Minecraft environment tests agents' ability to learn short and long-term tasks which adds the need to process video observations to the T-LS task. The agent encounters an indicator, then must navigate through a series of rooms (see Fig. ()). Each room contains either silver or blue column, indicating whether the agent must move around it to the left or right to get a reward. At the end, the agent must remember the indicator. There are 16 rooms total, each requiring at least 6 steps to solve. The episode timcout is 200 steps.
MC Saan--hhot-Oddreee (MC-LSO) This task tests whether models can learn policies condi- tioned on distant order-dependencies over two indicators. The two indicators each be green or red. Only a green followed by red indicates that the goal is to the right at the end. There are 10 rooms with timeout of 200 steps.
MC Long-SSort-Noon (MC-LSN) This task starts from MC-LS and adds observation noise to test robustness to noise while learning short and long-term task. For each visual observation we add

---

Published as conference paper at ICLR 2020
Tola Roward
Mudel AMRL Avg
Tosa Rooonng
AMGC -Mox
LSTH
Tooa
LSTNL STACK
DMC
(a) T-L
(b) T-LN
(c) T-LS
Figure 4: TMaze Results (5 seeds): AMRL-MMa and AMRL-Avg achieve superior performance
under observation noise, exploration, and interference short-term tasks. Best viewed in color.
Gaussian noise to each (RGB) channel. An example observation shown Figure B(e). There are 10 rooms with timeout of 200 steps.
4.3 RUNS AND BASELINES
We compare the following approaches: AMRL-Max. Our method with the MAX aggregator (Fig. 2d AMRL-Avg. Our method with the AVG aggregator (Fig. [dp. SET. Ablation: AMRL-Avg
without LSTM or straight-through connection (Fig. 2c). LSTM. The currently most common mem-
ory model (Hochreiter & Schmidhuber 1997) (Fig. LSTM Stacks LSTM cells
for temporal abstraction (Pascanu et al. 2013 Mirowski et al.) 2017) (Fig. [2b.. DNC. A highly
competitive existing baseline with more complex architecture (Graves al. 2016).
RESULTS
5.1
TMAZE TASKS
Our main results are provided in Figures and 5] We start by analyzing the T-L results. In this base task without noise or irrelevant features, expect all methods to perform well. Indeed, we observe that all approaches are able to solve this task within 50k environment interactions. Surprisingly, the L.STM and stacked LSTM learn significantly slower than alternative methods. We hypothesize that gradient information may be stronger for other methods, and expand on this in Section
We observe a dramatic deterioration learning speed the previous task in the additional noise features added to the state observations. LSTM and DNC are most strongly affected by observation noise, followed by the stacked LSTM.. In contrast, we confirm that proposed models robust observation noise and maintain near-identical learning speed compared to the T-L task, thus validating our modeling choices.
T-LN setting, which only differs from
Finally, we turn to T-LS, which encapsulates full RL task with observation features only relevant for the short-term task (i.e. long-term noise induced by short-term features), and noise due to ex- ploration. Our proposed models, AMRL-Avg and AMRL-Max able achieve returns near the optimal 13.9, while also learning fastest. All baseline models fail to learn the long-term memory task in this setting, achieving returns up 10.4.
5.2 MINECRAFT TASKS
The MC-LS task translates T-LS to the visual observation setting. The agent has to solve series of short term tasks while retaining information about the initial indicator. As before, we see AMRL- Max and AMRL-Avg learn the most rapidly. The DNC model learns significantly more slowly but eventually reaches optimal performance. Our SET ablation does not learn the task, demonstrating that both the order-invariant and order-dependent components crucial parts of our model.
The MC-LSO adds a strong order dependent task component. Our results show that the AMRL- Max model and DNC model perform best here far better than an LSTM aggregator alone. We note that this is the only experiment where DNC performs better than AMRL-Max or AMRL-Avg. Here the optimal return is 10.2 and the optimal memory-less policy return is 8.45. We speculate that DNC is able to achieve this performance given the shorter time dependency relative to MC-LS and the lower observation noise relative to MC-ISNN

---

Published as a conference paper at ICLR 2020
AMRL-Avg
AMRL--ha
Tooa
LSTM
LSTH STACK
UNC
200
360 Step (41
159
383
Diep
Step (0)
(a) MC-LS
(b) MC-LSO
(c) MC-LSN
Figure 5: Minecraft results (5 seeds): AMRL-Avg and AMRL-Max outperform alternatives in terms of learning speed and final performance.
Finally, MC-LSN adds noise to the visual observations. As expected, the LSTM and LTACK baselines completely fail in this setting. Again, AMRL-Max and AMRL-Avg learn fastest. In this task we see a large advantage for AMRL methods relative to methods with LSTMs alone, suggesting that AMRL has a particular advantage under observation noise. Moreover, note the strong performance A.RL-MMX despite the large state-space induced by noise, which affects the SNR bound. DNC is the baseline that learns best, catching up after 300k environment interactions.
ANALYSIS
Given the strong empirical performance of our proposed method, here analyze AMRL to under- stand its characteristics and validate model choices.
6.1 PRESERVING GRADIENT INFORMATION
Here we show that the proposed methods, which use different aggregators conjunction with LSTMs, do not suffer from vanishing gradients (Pascanu et al. 2012 Le et al. 2019). as discussed in Section 3 Our estimates are formed as follows. We set the model input to when and to for timesteps > O. We plot avg(d @ d), where = 1 da. steps, and we plot the average over three independent model initializations.
da; over t. Samples are taken every ten
Results over time, and for clarity, the final strength of the gradient, are summarized in Figure 6 We observe that the AMRL-Max and AMRL-AVG (and SUM) models have the same large gradi- ent. Our models are followed by DNC which in turn preserves gradient information better than LSTMs. The results obtained here help explain some of the empirical performance we observe in Section especially in that LSTMs are outperformed by DNC, with models having the greatest performance, However, the gradients are similar when noise is introduced (See Appendix (A.4), in- dicating that this does not fully explain the drop in performance noisy environments. Moreover, the gradients alone do not explain the superior performance of MAX relative to AVG and SUM. We suggest an additional analysis method in the next subsection.
6.2 LIGNAL-TO-NOISS RATIO (SNR)
Following the discussion in Section 3 we now quantify SNR empirically to enable comparison across all proposed and baseline models. We follow the canonical definition to define the SNR of function over time (Johnson 2006):
SNR(() = (N)((((s)))
= fe(s:)
(1)
where denotes time, Je is function of time, 8, is a constant signal observed at 4, and n, is a random variable representing noise time 1. Given this definition, our proposed aggregators (see Appendix A.3 for details).
derive the SNR analytically for
Analytical results are shown in the last column of Table I We see that the AVG and SUM aggrega- tors have the same SNR, and that both decay only linearly. (Empirically, we will see LSTMs induce
Guulcehre et al. 2017) analytically show that the gradients of DNC decay more slowly than LSTMss due in part the external memory writes acting as skip connections "wormholes".
8

---

Published as a conference paper at ICLR 2020
10
2
10"4
grad
10
grad
10
10
10-12
10-11
LSTM
LSTH STACK
DWC
AVG
SET
MAX
SUM
AMRL-Aug
10
40
60
Step
(a) Gradient Decay over Time
(b) Final Gradient Signal
Figure 6: Gradient signal over 0-100 steps. AMRL
models models and SUM maintain the strongest gradient.
exponential decay.) Moreover, the bound does depend on the size of the observation space, we observed superior performance even in large state spaces in the experiments (Section 5).
that Max has lower bound that is independent of Although
We now turn to an empirical estimate of SNR. In addition to the analytic results presented so far, em- pirical estimates allow assess SNR of our full AMRL models including the LSTM component, and compare to baselines.
Our empirical analysis compares model response under idealized signal that under idealized noise using the following procedure. The idealized signal input consists of a single 1 vector (the signal) followed by 0 vectors, and the noisy input sequence constructed by sampling from {0, 1} uniformly at random after the initial 1. Using these idealized sequences we compute the SNR as per Eq. We report the average SNR over each neuron in the output. We estimate E[sa] and E[n2]] over 20 input sequences, for each of model initializations.
The results show that AMRL-Max, Max, and the baseline DNC have the highest SNR. The lowest SNR is observed for LSTM and LSTM.STAK. The decay for both LSTM models is approximately exponential, compared to roughly linear decay observed for all other models. This empirical result matches our derivations in Table for proposed models. In Figure (a), observe that the SNR for LSTMs strongly depends on the time at which given signal occurred, while our Max models and DNC not susceptible to this issue.
10a
10-1
10-3
10-3
10-5
Shr
10
10-9
10
10 13
10-12
LSTM
LSTM STACK
AMRL-Aug
AVG
SET
AMRL-Mas Max
MAX
20
40
60
80
100
Step
(a) SNR Decay over Time
(b) Final SNR
Figure 7: MAX models and DNC have greatest SNR. LSTM and LSTM_STACK perform worst
with exponential decay. SUM and AVG have only linear decay, confirming our analytic finding.
6.3 DISCUSSION
The results in the previous section indicate that models that perform well on long-term memory tasks in noisy settings, such as those studied in Section 5 tend to have informative gradients and high SNR over long time horizons. In this section we further examine this relationship.

---

Published as a conference paper at ICLR 2020
10"
7.5
10
Maan Return
AMRL-AV SUM
JAAX
@ARL-MMax
7.0
10
6.5
AVG
8.5
Gradient
10"
DNG
7.5
8.0
10
10"10
LSTM
LSTM, STACK
SUM
DNC
AMRL Aug
10
STACK
10
10
10"
10"
10"
10
SNR
(a) Overall Performance
(b) Relation between Performance, SNR, Gradient
Figure 8: Overall Performance and Performance in relation to SNR and Gradient. Increasing either SNR or Gradient strength tends to increase performance. See text for details on the SUM model.
Figure 8 shows the aggregate performance achieved by each model across the experiments presented in Section 5 and in the appendix [A.2] We argue that these tasks capture key aspects of long-term memory tasks in noisy settings, We observe that our proposed AMRL-Avg and AMRL-Max ap- proaches outperform all other methods. Ablations Max and Avg are competitive with baselines, but our results demonstrate the value of the ST connection. AMRL-Max improves over the LSTM aver- age return by 19% with no additional parameters and outperforms the DNC average return by 9% with far fewer parameters. We have shown that AMRL models are not susceptible to the drastic per- formance decreases in noisy environments that LSTMs and DNCs are susceptible to, and we have shown that this generalizes to an ability to ignore irrelevant features in other tasks.
Figure b(b) relates overall model performance to the quantities analyzed above, SNR and gradient strength. We find SNR and gradient strength are both integral and complementary aspects needed for a successful model: DNC has a relatively large SNR, but does not match the empirical performance of AMRL likely due its decaying gradients. AMRL models achieve high SNR and maintain strong gradients, achieving the highest empirical performance. The reverse holds for LSTM models.
An outlier is the SUM model hypothesize that the growing sum creates issues when interpret- ing memories independent of the time-step at which they occur. The max aggregator may be less susceptible to growing activations given a bounded number of distinct observations, bounded input activation, or an analogously compact internal representation. That is, the max value may be low and reached quickly. Moreover, the ST connection will still prevent gradient decay in such case.
Overall, our analytical and empirical analysis in terms of SNR and gradient decay both validates our modeling choices in developing AMRL, and provides useful tool for understanding learning performance of memory models. By considering both empirical measurements of SNR and gra- dients we are able rank models closely in-line with empirical performance. We consider this a particularly valuable insight for future research seeking to improve long-term memory.
7 CONCLUSION
We have demonstrated that the performance of previous approaches to memory in RL can severely deteriorate under noise, including observation noise and noise introduced by an agents policy and environment dynamics. We proposed AMRL, a novel approach designed specifically to be robust to RL settings, by maintaining strong signal and gradients over time. Our empirical results confirmed that the proposed models outperform existing approaches, often dramatically. Finally, by analyzing gradient strength and aggnal-to-noise ratio of the considered models, we validated our model choices and showed that both aspects help explain the high empirical performance achieved by our models.
In future research, we believe our models and analysis will form the basis of further understanding, and improving performance of memory models in RL. An aspect that goes beyond the scope of the present paper is the question of how to prevent long-term memory tasks from interfering with shorter-term tasks issue highlighted in Appendix A.2.3 Additionally, integration of AMRL into models other than the standard LSTM could be explored. Overall, our work highlights the need and potential for approaches that specifically tackle long-term memory tasks from an RL perspective.
Appendix/A.22 A.223
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This work was done while Jacob Beck was research intern at Microsoft Research Cambridge. We would like to acknowledge Adrian O' ''addy, Yingzhen Li, Robert Loftin, Quan Vuong, Max Igl. and Game Intelligence team at Microsoft Research for their discussion, advice, and support.
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16. JMLR.org, 2016.
Bram Bakker. Reinforcement learning with Istm in non-markovian tasks with long-term dependen- cies. 2001.
Francois Belletti, Alex Beutel, Sagar Jain, and Ed Huai hsin Chi. Factorized recurrent neural archi- tectures for longer range dependence. AISTATS, 2018
Kyunghyun Cho, Bart Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the prop- erties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Synlax, Semantics and Structure in Statistical Translation, pp. 103-111, Doha, Qatar, October 2014. Association for Computational Linguistics.
Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI, 2017.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014. cite arxiv:1410.5401.
Alex Graves, Greg Wayne. Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwiska, Sergio Gmez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adri Puigdomnech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, He- len King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 538(7626):: 471 October 2016.
Caglar Gulcehre, Sarath Chandar, and Bengio. Memory augmented neural networks with worm- hole connections. 01 2017.
David Ha and Jürgen Schmidhuber. World models. CoRR, abs/1803.10012. 2018.
Moonsu Han, Minki Kang, Hyunwoo Jung, and Sung Ju Hwang. Episodic memory reader: Learning what to remember for question answering from streaming data. In ACL, 2019.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In AAAI Fall Symposia, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 983: 1780, November 1997.
D. H. Johnson. Signal-to-nol to-soiiee ratio. Scholarpedia, 1:2088, 2006. revision #126771.
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artifi- cial intelligence experimentation. In Proceedings of the Twenty-Fifth International Joint Confer- ence on Artificial Intelligence, IJCAI' 16, pp. 4246-4247. 4247. AAAI Press, 2016.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artif. Intell., 101()-2), May 1998.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent perience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
11

---

Published as a conference paper at ICLR 2020
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Modeling the long term future model-bbsee reinforcement learning. In International Conference Learning Representations, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. abs/1412.6980, 2015.
Morten Kolbaek. Dong Yu, Zheng-Huu Hua Tan, and Jesper Jensen. Joint separation and denoising of noisy multi-talker speech using recurrent neural networks and permutation invariant training. 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), pp- 1-6, 2017.
David Krueger and Roland Memisevic. Regularizing rnss by stabilizing activations. 2016.
Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less memorization. In International Conference on Learning Representations, 2019.
Quoe Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units. ArXiv, abs/1504.00941,, 2015.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer. CoRR, abs/1810.00825. 2018.
Xiujun Li, Lihong Li, Jianfeng Gao, Xiaodong He, Jianshu Chen, Li Deng, and Ji He. Recurrent reinforcement learning: A hybrid approach. CoRR, abs/1509.03044, 2015.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg. Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference Machine Learning (ICML), 2018.
Tomas Mikolov, Armand Joulin, Sumit Chopra, Michaël Mathicu, and Mare Aurelio Ranzato. Learning longer memory in recurrent neural networks. CoRR, abs/1412.7753, 2014.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifrre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning to navigate in complex environments. 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529533. February 2015.
Junhyuk Oh, Valliappa Chockalingam, Satinder P. Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. CoRR, abs/1605.009 2016.
Junier B. Oliva, Barnabás Póczos, and Jeff G. Schneider. The statistical recurrent unit. In ICML, 2017.
Jared Ostmeyer and Lindsay G. Cowell. Machine learning sequential data using recurrent weighted average. Neurocomputing, 2019.
Emilio Parisotto and Ruslan Salakhutdinov. Neural map: Structured memory for deep reinforcement learning. International Conference on Learning Representations, 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/1211.506l 2012.
Razvan Pascanu, aglar Gülehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recur- rent neural networks. International Conference on Learning Representations, 2013.
Leonid Peshkin, Nicolas Meuleau, and Leslie Pack Kaelbling, Learning policies with external mem- ory. In ICML,, 1999.
12

---

Published as a conference paper at ICLR 2020
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foer- ster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. CoRR, abs/1803. 1485, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0es//177.06344 2017.
Thomas Stepleton, Razvan Pascanu, Will Dabney, Siddhant M. Jayakumar, Hubert Soyer, and Rémi Munos. Low-pass recurrent neural networks memory architecture for longer-term correlation discovery. CoRR, abs/ I 005.04955, 2018.
Trieu H. Trinh, Andrew M. Dai, Thang Luong. and Quoe V. Le. Learning longer-term dependencies in mns with auxiliary losses. CoRR, abs/1803.00144, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs/161 1. 1.05763, 2016.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska. Jack W. Rae. Piotr Mirowski, Joel Z. Leibo, Adam Santoro, Mevlana Gemici, Mal- colm Reynolds, Tim Harley, Josh Abramson, Shakir Mohamed, Danilo Jimenez Rezende, David Saxton, Adam Cain, Chloe Hillier, David Silver, Koray Kavukcuoglu, Matthw Botvinick, Demis Hassabis, and Timothy P. Lillicrap. Unsupervised predictive memory in a goal-directed agent. CoRR, abs/1 10760, 2018.
Wierstra, Alexander Frster, Jan Peters, and J Schmidhuber. Recurrent policy gradients. Logic Journal of the IGPI, v. 18, 620-634 (2010), 01 2009.
Martin Wöllmer, Zixing Zhang, Felix Weninger, Bjöm W. Schuller, and Gerhard Rigoll. Feature enhancement by bidirectional Istm networks for conversational speech recogmition in highly non- stationary noise. 2013 IEEE International Conference Acoustics, Speech and Signal Process- ing, 2013.
Kenny J. Young, Richard S. Sutton, and Shuo Yang. Integrating episodic memory into a reinforce- ment learning agent using reservoir sampling. CoRR, abs/1806.00544 2018.
Wojciech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines. CoRR, abs/1505.00521, 2015.
Marvin Zhang, Zoe McCarthy, Chelsea Finn, Sergey Levine, and Pieter Abbeel. Learning deep neural network policies with continuous memory states. 20/6 IEEE International Conference on Robotics and Automation (ICRA), pp. 520-527, 2015.
13

---

Published as a conference paper at ICLR 2020
A
APPENDIX
A.1 ENVIRONMENT DETAILS
In this appendix we provide additional details of all tasks we report on in the main paper.
A.1.1 TMAZE LONG (T-L)
In these experiments, the agent is initially placed at one end of a corridor. the other end is a T-junction, where the agent can decide move left or right. The goal state cannot be observed and located in one of these two directions. Once the agent chooses to check a given direction, the episode terminates and the agent either receives success fail reward, determined by whether the agent picked the direction toward the goal. The position of the goal is randomized between episodes and is indicated only in the very first step by an observation called the indicator. T-L. is base task where noise is minimized. The agent is automatically moved to the next position in each step eliminate variation due an agent's exploratory policy. Thus, there single decision to learn whether to move left or right at the T-junction.
In all TMaze experiments, to avoid confounding factors such as changing the length of BPTT and changing the total number of timesteps per indicator observation, we fix the length of the maze and simply move the indicator. The indicator can be placed at the beginning of the maze or another location in the middle. The agent receives a reward of 4 for the correct action at the junction and -3 for an incorrect action at the junction. We encode observations as vectors of length 3, with the first dimension taking on the value of if the agent is at the start (0 otherwise), the second dimension taking on the value of 1 or -1 corresponding to the two values the indicator (when the agent is at the indicator, 0 otherwise), and the final dimension taking on the value of 1 if the agent is at the T-junction (0 otherwise). (For example, [1, -1, 0] encodes an observation for the agent at the start with the goal placed the right.). Unless otherwise stated, we timeout of 150 steps.
A.1.2 TMAZE LONG NOISE (T-LN)
Here we append a noise feature to the agent observation to test robustness to observation noise. Noise us sampled uniformly from {-1, 1}. This experiment is a variant of experiments proposed in Bakker (200T) where continuous valued noise was used. Here we choose discrete noise features as they allow us build up to the short-long decision task discussed next.
A.1.3 TMAZE LONG-SHIORT (T-LS)
The short-term task that we add is to "recreate" the noise observation. More precisely, we append dimension to the action space that allows for two actions: one representing the value and the other representing -1. If this action matches the noise observation, then the agent proceeds to the next step and received a reward of 0.1. Otherwise, the agent stays where it is and the observation is recreated with the noise dimension sampled from {-1, 1}.
1.1.4 MINECRAFT MAZE LONG-SHORT (MC-LS)
The agent starts on an elevated platform, facing block corresponding to a certain indicator: When on the platform, the agent must step the right to fall off the platform and into the maze. The agent is now positioned at the southern entrance to a room oriented on the north-south axis. Stepping forward, the agent enters the room. At this point, there are columns preventing the agent from moving cast or west. The agent has all of the actions (north, cast, west) available, but will remain in place if stepping into column. The agent's 's best choice to move forward onto a ledge. On this ledge, the agent faces column whose block type (diamond or iron) indicates safe direction (east or west) to fall down. the agent chooses correctly, gets a positive reward of 0. At this point, the agent must proceed north two steps (which are elevated), fall back to the center, then north to enter the next room. At the very end (purple), the agent must go right if the initial indicators were green (green then red in the multi-step case), and left otherwise. The agent receives a reward of for a correct action at the end and -3 otherwise. In addition, if the agent takes action that progresses it to the next step, it receives reward of 0.1 for that correct action.
the agent's left and right
14

---

In this version of the Minecraft Maze we start from the MC-LS task and add O-mean Gaussian noise to each channel in our RGB observation, clipping the range of the pixels after the noise to the original range of range [-1,1]. The noise has a standard deviation of 0.05. In addition to adding noise that could affect the learning of our models, this experiment tests learning in continuous observation Published as a conference paper at ICLR 2020
Unless otherwise stated, we use timeout of 200. Here 13.7 is optimal, and 10.2 is the best possible for a memory-less policy. There are 16 rooms total, each requiring at least steps each to solve.
A.1.5 MINECRAFT MAZE LONG-SHORT ORDERED (MC-LSO)
In this version of the Minecraft Maze, there is a second initial platform that the agent will land on after the first and must also step right off of to enter the maze. The options for the colors of the indicators are (green, red), (red, green), (green, green), (red, red), Of these, only the first indicates that the agent is take right the end of the maze. As in the T-Maze Long-Short Ordered environment, the goal here is to see if aggregating the LSTM outputs is capable of providing an advantage over an aggregator or LSTM alone. We use only 10 rooms given the time required to solve this environment. We speculate that this environment is the only one where DNC outperforms our ST models due to the shorter length, which gives our models less of an advantage.
A.1.6 MINECRAFT MAZE LONG-SHORT NOISE (MC-LSN)
space that could be problematic for the MAX aggregator. We use 10 rooms for this experiment with optimal return 10.1, and the optimal memory-less policy return is 6.6.
A.2 ADDITIONAL EXPERIMENTS
In addition our primary experiments presented in the main paper, we designed the following experiments to confirm that our proposed models retain the ability to learn order dependent infor- mation, using the path through the LSTM model. The expected result that learning speed and final outcome matches that of the baseline methods, and that the SET model cannot learn the order- dependent aspects of the tasks. This is indeed what we confirm.
A.2.1 LONG ORDER VARIANCE EXPERIMENT (T-LO)
This experiment modifies the TMaze such that there are two indicators at opposite ends of the hallway. We place indicator at position and N-2 (the ends of the corridor that are just adjacent two the start and T-junction respectively). The two indicators take on one the pairs of values with equal probability: [1, -1], [-1, 1J, [1, 1]. [1]. Only the first of these [1] corresponds to the goal being placed to the left at the end. In order to solve this environment optimally, the agent must remember both of the indicators in the correct order.
We expect that a single h2 would need to be used to encode both, due to the order- variance and that SET cannot perform this task. Given that the indicators on which the order depends span the length of the maze, we do not expect to see any performance differences between methods other than SET. Results are shown in Figure 9 (left) and confirm our hypothesis. SET is not able learn the task while all other approaches correctly learn order-dependent information and solve the task optimally within most 50k steps.
A.2.2 TMAZE LONG-SHOV ORDERED EXPERIMENT (T-LSO)
In order to see whether we can learn policies conditioned on distant order- aider-dependencies. along with irrelevant features, we extend the T-LS environment with an additional indicator, similar to that in T-LO. As above, the two indicators can take on values in the pairs: -1], [-1, 1], [1, 1], [-1, -1], with equal probability. Only the first of these [1. -1] corresponds to the goal being placed to the left at the end. In this experiment, the two indicators were placed at positions and 2, so that their observation does not include the start bit, which could be used to differentiate the two. Unlike in T-LO, our indicators appear at the start of the corridor and are adjacent. Here, we expect baseline methods to be less sample efficient than AMRL because gradients decay over long distances.
Our results in Figure Deneeeeoonfin confirm our hypothesis. We see only AMRL-Avg and AMRL-Max are able exceed the return of the best memory-less policy (12.15). We confirm by inspecting the individual
15

---

Published as a conference paper at ICLR 2020
Long
D50 ST
AMG
HALST
NA.
ESTW 5TACE
LIT0
SET
50
100 150
300
256
100
203 200
400
550
Figure 9: Results: T-LO (left) and T-LSO (right) experiments. These are included in overall perfor- mance but not discussed in the main body. Our results confirm that AMRL models maintain order dependent memories while the SET ablation does not. Note: "ST" here is short for "AMRL""
Chickin
cl
c2
Muldd
c3
1TT9
a
Stap
Figure 10: Top down map of Chicken environment (left) and a view from the agent near the start of the episode when the signal is green (center). Results shown the right.
learning curves, that both AMRL-Avg and AMRL-Max achieve reward near optimal. This is also the only setting where stacked LSTM performed worse than LSTM.
A.2.3 MINECRAFT CHICKEN
Our final environment is also constructed Minecraft and meant to more closely emulate the Strong Signal setting with recurring signal, defined in A.3 beloo. In this environment, the agent inhabits a by room, starting at the south end the middle (a on the map), and must collect as many chickens possible from positions cl, c2, c3 (see Figure [T0] left and center), receiving reward 0.1 per chicken. The chicken inhabits one of three locations. The chicken starts forward block and offset left or right by block (c1 or c2), placed uniformly random. The chicken is frozen in place and cannot move. Once collected, new chicken spawns behind the agent in the middle of the room just in front of the agent's original spawn location (c3). Once that chicken is collected, the next chicken will again spawn forward and offsse (cl or c2), and the cycle continues.
After 48 steps of the repeated signal, the floor changes from red or green grey. The agent then has another 96 timesteps to collect chickens. At the end, the agent must recall the initial floor color make decision (turn left/right). the agent is correct, receives a reward chickens, otherwise it receives reward of -3 and falls into lava. The agent has actions: forward, backward, left, right, collect. In the final step, when the agent is asked recall the room color, left corresponds to red, right corresponds to green, and all other actions are incorrect (reward -3).
and keeps all the
We see that models quickly plateau at return near 4, although 8.8 is optimal. Roll-outs indicate that all models learned to remember room color, but struggled to collect chickens. Training the best performing model, MAX, for 1.5 million steps, we saw still perfect memory in addition to good chicken collection (c.g. roll-out: https://youttube/CL.ti wtw.b//e/CHmmmSmSeSSmm from localization failure in an entirely grey room. Chicken collection can be learned the same extent, but is learned slightly faster, without the memory-dependant long-term objective.
with most mistakes coming
16

---

Published as a conference paper at ICLR 2020
A.3 SIGNAL-TO-NOIEE RATIO (SNR) ANALYTICAL DERIVATIONS
We derive the SNR analytically for our proposed aggregators in two different settings. We term the setting assumed in the main body of the paper the weak signal setting. In this setting we assume that the signal takes on some initial value so. followed by O's s. This simulates the setting where an initial impulse signal must be remembered. Additionally, we assume that each n, is O-mean and is sampled from the same distribution, which we call
The assumptions of the weak signal setting are motivated by our POMDP from the introduction in which an agent must remember the passcode to door. In this setting. the order of states in between the passcode (signal) and the door are irrelevant (noise). Moreover, all of our aggregators are commutative. Thus, instead of the ordered observations in trajectory, we can consider the set of states the agent encounters. If the episode forms an ergodic process, then as the episode continues, the distribution defines a marginal distribution over observations. Thus, it is useful to consider the case where each state is drawn not from ((o)(s)) (s)), but rather i.i.d. from p. for the purpose of analysis.
encountered observations will approach the stationary distribution p. which
In addition to the weak signal setting, it is worth considering a recurring signal. We term such a setting the strong signal setting. The strong signal setting assumes that the signal recurs and that the signal and noise are not present at the same time. Specifically, we assume that there is one signal observation Og, which can occur any time; that each observation or is drawn from pt that the signal is o, if o, Os. and 0 otherwise; and that the noise is 0 if or would be relevant, for example, for an agent engaging in random walks at the start of training. This agent may repeatedly pass by a signal observation that must be remembered.
and or otherwise. This setting
For the weak and strong signal settings, we now analytically derive the SNR of each of our proposed aggregators (summarised previously in Table
Average Aggregator Signal averaging is common way to increase the signal of repeated measure- ments. In the weak signal case, the signal decays, but only linearly. Writing 8 as a shorthand for Sg:
1 SN SN R(au((st), ovg(ne)) = (-50 + +))))/22 [)/
n4) (From Eq. !
5² /B/ n4
= 3² (EE
Σ
n= EE
n4 2 E
n2
82/(Var)) (Var(
m2) + ne
Given i.i.d.
2² (tVar(n) + =
Assuming O-mean noise:
x² /((aar(())
In the strong signal setting, we can actually see linear improvement. In this setting, given that the signal is also random variable, we use:
=N /
(2)
SNR( = (N)((( = f((s))
(3)
where t denotes time, fr is function dependent on ft-1, S; is a constant signal given time, and n a random variable representing noise at time f. Given this definition we have:
SNR = (gaog($>))/1
Σ n
17

---

Published as a conference paper at ICLR 2020
i=t
= p(s))> - p(s))
E
n
~ ))9l)
1=1
p(s)
(62 /BI
p)21)
p-)(s)
From above:
p(s) (TV//tVa((())
1- p(s)s
p(s)
p(s)
Note that if we do not assume that the noise is zero-mean, can derive the following:
p(s)
(s2 (tV + {
1 p()
This converges to:
p(8) p(s)
(5² /E(m1]]))
Sum Aggregator In the weak setting:
SN SN R(sum(ss), sum)(n)) = 2// [E|
nz)
SN SNR(agg(s) (v)g(()))))
In the strong signal setting:
SN R(sumsss) sum(n:)) =
~ p) = SN avg(n:)
1=1
Thus in both settings, the SNR is the same as that of the average.
find SN R(mar(s:)) mar(())) to be inadequate. The reason for this is that for 8, as we have previously defined it maar(60.....8 8t) = maxr(s 0) will equal o, if o, > O, else O. However, there is reason that the signal should vanish if the input is less than O. Instead, we define m, to represent the signal left in our max aggregator. We define me be o, if Vo-cis(o, 2 o;) else O. That is, if the signal "wins" the max far, my is o, s, else O, representing no signal. We define 2 to be maFueiss(ot) if Bocegi(o; > o,) else 0. In the continuous setting (weak setting with no repeats of any observation signal noise) we have:
Max Aggregator For the
aggregator,
32
SN R(m,, 21)
2+1
5² (1 < SN SN
0
E
2+1
In the discrete setting, we can derive the bound:
E Emm²
2²
SN R
E[
(2) (1 - 1))az((1)²
which conveniently has no dependence on t and is very reasonable given small observation space.
A.4 STRONG SNR AND NOISY GRADIENTS
In this section, we empirically measure the gradient under observation noise and empirically mea- sure the SNR in the strong signal setting. We see that the gradient the noisy setting is similar to the gradient in the non-noisy setting, We also that when the signal is attenuated in the strong setting, the SNR of LSTM--based methods drops substantially. This is due to the SNR the LSTMs being dependent on recency of the signal, and also results in greater variance in the SNR. Our models, on the other hand, have a SNR with low variance and an SNR that resilient to signal attenuation. Our models have stronger dependence the number of prior signals than on the recency of the signal.
18

---

Published as a conference paper at ICLR 2020
Gradient (Signal)
Gradient (hoise)
307
40'1
40-1
Maad
Miai
10
30
Figure 11: (Left:) Decay of gradient in strong signal setting, (Right:) Decay of gradient when the input is sampled from U (1, -1). Results are surprisingly similar to non-noisy setting.
Strong
10-
10
10
LTT
STAOK
mm
Ste
Figure 12: (Left:) SNR of various models in the weak signal setting. (Right:) SNR of models in strong signal setting defined in A.3 LSTM and LSTM_STACK fail when the signal is removed, since they are very dependent on being temporally close the signal, instead of simply depending on the number of prior signals. Again, MAX and DNC perform the best.
A.5 HYPER-PARAMETER
Learning Rate For learning rates, the best out of 5e-3, 5e-4, and 5e-5 are reported in each exper- iment, over 5 initialization each.
Recurrent Architecture All LSTT sizes were size 256. DNC memory size 16 (slots), word size 16 (16 floats), 4 read heads, write head, and 256-LSTM controller.
Feed-forward Architecture We use ReLU activations. Our last feed-forward layer after the mem- ory module size 256 (2))). Before memory module we have two feed forward layers both size 256 (F).. For models with image input, FF, consists of 2 convolutions layers: (4.4) kernel with stride and output channels 16; (4.4) kernel with stride 2 and output-channels 32. Then this is flattened and fully-connected layer of size 256 follows.
we use the adam optimizer (Kingma & Ba 2015), and a PPO agent (Schulman et al.) 2017]. For training our PPO agent: mini-batch size 200, train batch size 4,000, num sgd iter 30, gamma .8.
Optimizer
Software We used Ray version 0.6.2 (Liang et al. 2018 We used python 3.5.2 for all experi- ments not in the appendix, while python 3.5.5 used for some earlier experiments in the appendix. Python 3,6.8 was used for some empirical analysis.
A.6 LSTM DEFINITION
The LSTM maintains its own hidden state from the previous time-step. h, and outputs ht We the following definition of the L.STM (Hochreiter & Schmidhuber 1997:
19

---

Published as a conference paper at ICLR 2020
4 = (Wi. + W + b,)
f: a (W J. + 5 + by)
Z: - a (W o W. Wo bo)
é tanh(W.... c, + We,hh + be)
C₁ = c @ f: + c ① iz
h: = tanh(c;) 0 zt
20


Reviews:
[{"review_id": "ryxqgvvRtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# UPDATE after rebuttal\n\nI have changed my score to 8 to reflect the clarifications in the new draft.\n\nSummary:\n\nThis paper presents a family of architectural variants for the recurrent part of an RL controller. The primary claim is that simple components which compute elementwise sum/average/max over the activations seen over time are highly robust to noisy observations (as encountered with many RL environments), as detailed with various empirical and theoretical analyses. By themselves these aggregators are incapable of storing order dependent information, but by combining an LSTM with an aggregator, and pushing half the LSTM activations through the aggregator, and concatenating the the other half of the activations with the aggregator output, the resulting output contains order dependent and order independent content.\n\nThe motivation is very clear (many of the most challenging modern video games used as RL environments clearly have noisy observations, and many timesteps for which no new useful information is observed) and the related work is comprehensive. An increase in training speed and stability, apparently without any major caveats, would be of great interest to any practitioner of Deep Reinforcement Learning.\n\nThe experiments provided are good, and vary nicely between actual RL runs and theoretical analysis, all of which convinces me that this could well become a standard Deep RL component. I do have a range of questions & requests for clarification (see below) but I believe the experiments as presented, plus some additions before camera ready, will make for a good paper of wide interest.\n\n\nDesicion: Accept. I would give this a 7 if I was able to. The idea is very simple (indeed I find it slightly hard to believe no one has tried this before, but I don't know of any references myself) and the results, particularly Figure 4, are compelling. It's nice to see a very approachable more mathematical analysis as well, it would be good to see more papers proposing new neural components with this kind of rigour. I look forward to trying this approach myself, post publication.\n\n\nDiscussion:\n\nAMRL-Avg coming out the best in Figure 8 makes a lot of sense to me, as I can see how average provides a stable signal of unordered information. One thing that really doesn't make sense to me is why Max and Sum would also be good - obviously their SNR / Jacobians are quite similar, but fundamentally there is a risk of huge values being built up in activations (especially with Sum), which at least have the potential to cause numerical instability, provide weird gradient magnitudes (which then mess up moment tracking optimizers like Adam, etc). Thre does not seem to be any mention of numerical stability, or whether any considerations ned to be taken for this. Maybe we could hope that 'well behaved' internal neron activations are zero mean, so the average aggregator will never get too big - but is this always the case, at all points in training, in every episode? I appreciate the straight through estimator might ameliorate this, but it is not made entirely clear to me in the text that this is the reason for using it. Addressing this point would increase the strength of the argument.\n\nGiven that DNC was determined to be the strongest baseline in a few of the metrics, and that AMRL combines these aggregators with LSTM, an experiment that I'm surprised is missing would be to make AMRL containing a DNC instead of an LSTM. Is there a reason why this wasn't attempted?\n\n\nIt would have been good to see a wider set of RL experiments - the Atari suite is well studied, easily available, and there are many open source implementations to which AMRL could easily be slotted into (eg IMPALA, openai baselines, etc).\n\nThe action space for Minecraft is not spelled out clearly - at first I assumed this was the recent project Malmo release, which I assume to have continuous actions (or at least, the ability to move in directions that are not compass points), but while Malmo is mentioned in the paper, the appendix implies that the action space is (north, east, west) etc. I'm aware that there is precedent in the literature (Oh et al 2016) for 'gridworld minecraft', but I think it would improve the paper to at least acknowledge this in the main text, as I feel even most researchers in 2019 would read the text and assume the game to be analogous to VizDoom / DeepMind Lab, when it really isn't. Note that most likely the proposed method would give an even bigger boost with \"real\" minecraft, as there are even more non-informative observations between the interesting ones, and furthermore I think the environment choice made in this paper is fine, as it still demonstrates the point. A single additional sentence should suffice.\n\nMinor points:\n\nNTM / DNC were not \"designed with RL in mind\" per se, the original NTM paper had no RL and the DNC paper contained both supervised and RL results. Potentially the statement at the bottom of page 1 was supposed to refer mainly to stacked LSTMs - either way I feel it would be better to slightly soften the statement to \"...but also for stacked LSTMs and DNCs (cite), which have been widely applied in RL.\"\n\nIn the RL problem definition, the observation function and the state transition function are stated as (presumably?) probability distributions with the range [0,1], but for continuous state / observation spaces it is entirely possible for the probability density at a point to exceed 1. \n\nAlso in the RL section - the notation of $\\tau_t \\in \\Omega^t$ should probably also contain the sequence of $t-1$ actions taken throughout the trajectory - this is made clear in the text, but not in the set notation.\n\nThe square bracket \"slicing\" notation used for slicing and concatenating is neat, but even though I spend all day writing python it still took me a while to realise what this meant, as I haven't seen this used in typeset maths before. Introducing this notation (as well as the pipe for concatenation) at first usage would help avoid tripping up readers unnecessarily.\n\n\n\nNote that this pdf caused several office printers I tried it on to choke on later pages, and I get slow scrolling in Chrome on figures such as figure 11 - possibly the graphics are too high resolution (?)\n\n\ntypos: A5, feed forward section: \"later\" should be \"layer\" I believe.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rJxMhbcAtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: The response helped address my questions. I've raised the score to 6.\n\nThis paper studies reinforcement learning for settings where the observations contain noise and where observations have long-range dependencies with the past. The proposed approach builds on the LSTM model and adds an aggregated memory cell, which decreases noise by allowing them to cancel at a high level.\n\nExtensive experiments are provided on two sets of tasks, the TMaze and the Minecraft tasks. The experimental results look convincing to me. However, as someone who is outside the area, it is difficult to understand the details of the paper. There are numerous observations and experimental design choices which are not clearly explained I think.\n- The current version (~ 10 pages) is significantly over length.\n- LSTMs are sensitive to noise: Is there an explanation for this observation? Also, is the claim still true by suitably regularizing the LSTM, etc?\n- TMaze Long-Short: What's the intuition for why this setting requires learning over long-term memory tasks?\n\nMore detailed comments/questions:\n- Intro P1: You start by talking about tasks that require long-term memory. Then you talk about full vs partial observations. What's the connection between these two?\n- Intro P4: This observation is interesting -- is there an explanation or intuition for what's happening here?\n- Figure 1: Why 68% confidence interval?\n- Aggregators: The 1/2 notation in the definition of m_t looks very confusing.\n- Definition of a_t, P5: what is FF_2?\n- TMaze Long Noise: By adding noise, do you mean that the observations are simply randomly sampled from {-1, +1}?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gKWjh15H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to add one more memory layer on top of LSTM that offers more direct integration of information over time. The goal is to demonstrate that it is very useful to aggregate previous information in a RNN-based reinforcement learning settings. The set-based aggregation improves the gradient over time and maintain good signal-to-noise ratio. The model is evaluated on Tmazes and Minecraft.\n\nOverall it is an interesting paper in the sense that the introduced aggregation layer is so simple but effective in noisy RL. The explanation using arguments of gradient decay and SNR decay seems to be convincing. \n\nIt is interesting to know if stacked LSTMs with vertical skip connection as in ARML work because the upper LSTM seems to have the same design goal of integrating information over time.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a way to augment memory in recurrent neural networks with order-independent aggregators. In noisy environments this results in an increase in training speed and stability. The reviewers considered this to be a strong paper with potential for impact, and were satisfied with the author response to their questions and concerns.
Decision: Accept (Poster)