Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
Authors: Johan Ferret
Raphaël Marinier
Matthieu Geist
Olivier Pietquin
Venue:  (2020)
Keywords: reinforcement learning
transfer learning
credit assignment

Abstract: The ability to transfer knowledge to novel environments and tasks is a sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is Secret, a novel approach to transfer learning for RL that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as a separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.

Under review as a conference paper at ICLR 2020
SELF-ATTENTIONS CREDIT ASSIGNMENT FOR TRANSFER IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to transfer knowledge novel environments and tasks is sensible desiderata for general learning agents. Despite the apparent promises, transfer in RL is still an open and little exploited research area. In this paper, we take a brand-new perspective about transfer: we suggest that the ability to assign credit unveils structural invariants in the tasks that can be transferred to make RL more sample efficient. Our main contribution is SECRET, a novel approach to transfer learning for RI. that uses a backward-view credit assignment mechanism based on a self-attentive architecture. Two aspects are key to its generality: it learns to assign credit as separate offline supervised process and exclusively modifies the reward function. Consequently, it can be supplemented by transfer methods that do not modify the reward function and it can be plugged on top of any RL algorithm.
INTRODUCTION
To some, intelligence is measured the capability of transferring knowledge to unprecedented situations. While the notion of intellect itself is hard to define, the ability to reuse learned information a desirable trait for learning agents. The coffee test (Goertzel al., 22012), presented as a way to assess general intelligence, suggests the task of making coffee in a completely unfamiliar kitchen. It requires a combination of advanced features (planning, control and exploration) that would make the task very difficult if not out of scope for the current saate-of-the-all Reinforcement Learning (RL) agents learn. On the other hand, is solved trivially by humans, who exploit the universally invariant structure of coffee-making: one needs to fetch mug, find coffee, power the coffee machine, add water and launch the brewing process by pushing the adequate buttons. Thus, to solve the coffee test, transfer learning appears necessary. Were we to possess a random kitchen simulator and a lot of compute, current transfer methods would still fall short of consistently reusing structural information about the task, hence also falling short of efficient adaptation.
Credit assignment, which in RL refers to measuring the individual contribution of actions to future rewards, is by definition about understanding the structure of the task. By structure, mean the relations between elements of the states, actions and environment rewards. In this work, we investigate what credit assignment can bring to transfer. Encouraged by recent successes in transfer based on supervised methods, we propose to learn to assign credit through a separate supervised problem and transfer credit assignment capabilities to new environments. By doing so, we aim at recycling structural information about the underlying task.
To this end, we introduce SECRET (SElf-attentioaal CREdit assignment for Transfer), a transferable credit assignment mechanism consisting of self-attentive sequence-to-sequeee to-sequence model whose role to reconstruct the sequence of rewards from a trajectory of egvet-environment interactions. assigns credit for future reward proportionally to the magnitude of attention paid to past state-action pairs. SECRET can be used to incorporate structural knowledge in the reward function without modifying optimal behavior, the structure of the task.
show in various generalization and transfer scenarios that preserve
Existing fackward-view credit assignment methods (Arjona-Medina et al. 2019 Hung al. 2018) require to add auxiliary terms to the loss function used to train agents, which can have detrimental external memory, which hinder
effects to the learning process (de Bruin al. 20188), and rely the generality of their approach. SECRET does neither. Also, as we show in Sec. Se.. the architecture consider for SECRET has interesting properties for credit assignment. We elaborate about

---

Under review as a conference paper at ICLR 2020
novelty with respect to prior work in Sec. We insist on the fact that the focus of our work is on transfer and that it is not our point to compete on credit assignment capabilities.
We would like to emphasize several aspects about the generality of SECRET: 1) our method does not require any modification to the RL algorithm used to solve the tasks considered, 2) it does not require any modification the agent architecture either and 3) it does not alter the set of optimal policies we wish to attain. Moreover, our method for credit assignment is offline, and as a result, it can use interaction data collected by any mean (expert demonstrations, replay memories (Lin) [992), backup agent trajectories.... ). We believe that this feature is of importance for real-world use cases where high number of online interactions is unrealistic but datasets of interactions exist as a byproduct of experimentation.
Background We place ourselves in the classical Markov Decision Process (MDP) formalism (Put- erman 1994). An MDP is tuple (S, A, ,, R, P) where S is state space, A is an action space, is discount factor (7 E [0, 1)), R: x x + R is a bounded reward function that maps state-action pairs to the expected reward for taking such an action in such a state. Note that choose form that includes the resulting state in the definition of the reward function over the typical R This is for consistency with objects defined later on. Finally, P S x A - As is Markovian transition kernel that maps state- state-action pairs to a probability distribution over resulting states, As denoting the simplex over S.
A + I.
An RL agent interacts with an MDP at given timestep / by choosing an action 0₂ E A and receiving resulting state St-1 ~ P(-|s,, du) and a reward re R(s,, at 8(+1) from the environment. A trajectory = (s;, 0z Ti... is set of state-action pairs and resulting rewards accumulated in an episode. A subtrajectory is portion of trajectory that starts at the beginning of the episode. The performance of an agent is evaluated by its expected discounted cumulative reward E| 00 r-re]- In a partially observable MDP (POMDP), the agent receives at each timestep t an observation or ~ O-(ss;) that contains partial information about the underlying state of the environment.
2 SECRET: SELF-ATTENTIONAL CREDIT ASSIGNMENT FOR TRANSFER
SECRET uses previously collected trajectories from environments in a source distribution. A self- attentive sequence model is trained to predict the final reward in subtrajectories from the sequence of observation-action pairs. The distribution of attention weights from correctly predicted nonzero rewards is viewed as credit assignment. In target environments, the model gets applied to small set of trajectories. We use the credit assigned to build a denser and more informative reward function that reflects the structure of the (PO)MDP, The case where the target distribution is identical to the source distribution (in which we use held-out environments to assess transfer) will be referred to as generalization in nn-domain transfer, opposed out-of of-domain transfer where the source and the target distributions differ.
2.1 SELF-ATTENTIOALL CREDIT ASSIGNMENT
Credit assignment as offline reward prediction We learn to assign credit through an offline reward prediction task, based on saved trajectories of engereenvironmmet interactions. We create sequence-to-sqquenee (seq2seq) model (Sutskever et al.) 2014) that takes as input the sequence of observation-actioon pairs and has to reconstruct the corresponding sequence of environment rewards. Being offline, the reward prediction task is learned separately from the RL task, and the reward prediction model does not share representations with the agent. This way, the representations learned for credit assignment do not affect or get mixed with the representations learned for control. Operating offline brings several advantages: one can directly interact with the replay memory of agents and even use expert demonstrations arbitrary saved transitions be useful in settings where on-policy interactions are costly, such as robotics. We equip our seq2seq model with an attention mechanism (Bahdanau et al. 2015) and view the attention weights of the reward reconstruction task as our primary source of assigned credit. The motivation to do so is that the seq2seq model looks into the past to find predictive signal in order to reconstruct the reward, so observation-action pairs it attends to should be those which reduce its uncertainty about the future, in other words those that explain future reward and should be credited.
source of supervision, which could
2

---

Under review as a conference paper at ICLR 2020
On the use of observations In MDPs, environment states follow the Markov property: they summarize the history of previous interactions and are sufficient to predict the future. As such, predictive models highly biased towards focusing the current sequence element, which hinders credit assignment. Under that consideration, when dealing with MDPs, we turn states into observations by applying transformations that hide certain amount of information from states and break the Markov assumption. For instance, in gridworlds with visual states, we crop the image and get a player centered image with a given window size. Doing so encourages the model to look into the past to find predictive signal, and allow us to track the relative importance given to each element to reconstruct the credit assigned. In POMDPs, this might be unnecessary depending on the amount of information shared between observations and true states.
Self-attention for credit assignment Unlike other seq2seq architectures, self-attentive models like Transformers (Vaswani et al. (017)) have direct computational paths between pairs of sequence elements, due to their representations that depend on projections of all sequence elements. This feature is key to long-term credit assignment. As an illustration, consider an RL task where the terminal reward depends only on the first observation, which is drawn randomly. Predicting the reward correctly requires to remember the first observation, which would be very challenging for a recurrent architecture whose memory goes through ((n) transformations, n being the size of the sequence. On the other hand, self-attentive model directly accesses the value of the initial observation, which makes credit assignment easier.
Reward prediction architecture We use Transformer decoder with a single self-attention layer (Lin et al. 2017) and a single attention head. The model input is a sequence of observation- action couples (or, ae to...... Each observation goes through series of convolutional layers (for visual inputs) followed by a series of vector in the discrete action case, is concatenated to the learned observation embedding. Those repre- sentations of dimensionality d, are combined with positional encoding (PE), fed to a self-attention layer and then to position-wise feedforward layer that outputs logits for reward prediction classes. PE encodes the relative positions of sequence elements, see Appendix A for details.
layers. Each action representation, a one-hot
Self-attention is an attention mechanism with parameterization (Wa, W;, W.). each matrix belonging to Rdd,xde that puts sequence elements in relation by computing non-linear similarity scores for all pairs of elements in the sequence. To do so, each sequence element is mapped to a query vector that matched against keys and values obtained from the previous elements. To be consistent with the goal of assigning credit, the model should not be able to peek into the future. Thus, we restrict the computational window of each sequence element to the information stored in representations of the previous elements in the sequence and its own by applying a causal mask M. to the result of the pairwise similarity computations, assigning a value of to masked elements after the softmax.
Let X = (#=):=o...., E Txx: denote the input sequence in matrix form, 2t being the result of internal computations of the model on its th input. In the same fashion, we note Z = (21)t=0..... E R T xd the sequence resulting from the application of self-attention. We then have
Mr @ (QKT) -C(1 -C(1-MM (()) MM)
7 softmax
V,
Vile
where Q XW, E Tx4x stores queries, K XW E Txx: keys. and V = XW, E XTxds values as linear projections of the input; d, stands for the dimension of the key vectors, Me E (0. [0,1)7xx TTT is a binary matrix that acts as a causal mask (a lower triangular matrix), O the Hadamard product and C' is a large constant (109 in practice).
Notably, the resulting observation-action representation can viewed as a linear combination of the values of previous elements: 2; = Ei=o Otgez where ax.-- = (ose-t)==1.... X exp( (ge. k:) Vdz). The vector a contains the normalized attention weights for the prediction at timestep and sums to 1. Since observations contain only a portion of their initial information, the fact that the model succeeds in the prediction task indicates that it reconstructed the missing information from its past. Therefore, attention weights themselves can be viewed as a form of credit assignment, and will be used such in what follows.
While performing regression on the rewards could also be an option, our experiments found that regression tends converge poor local optima. Consequently, we predict the sign of the experi-
3

---

Under review as a conference paper at ICLR 2020
enced rewards: g(r) sign(r) with sign(0) O. We chose the sign the classification target for its invariance to the scale of the rewards. We use a weighted sequential cross-entropy as the loss function over the class-wise model predictions fo.e. writing r(o, a) the subtrajectory of ending with the observation-action couple (o, a) to translate the effect of the binary mask:
w(c) |T|
Co(T) = =
E)t(()) - c} log (fe.c(T(o.a))).
0E(-1,0.1)
(o,a,))ET
We have found class weighting to be very important keep the variance of prediction metrics across a variety of datasets of sampled trajectories low.
Generating trajectories To train SECRET, we generate dataset of trajectories which contains a certain proportion of successful trajectories. If source environments are simple enough so that the task has sufficient chance be solved by acting randomly, we trajectories. For more complex distributions of environments, we use a RI. agent (either trained or in the learning phase) generate trajectories. We think purely exploratory methods like Ecoffett et al. (2019) could have advantages over using an RL agent and leave the study of their use for future work.
random policy generate
2.2 LEVERAGING CREDIT VIA REWARD SHAPING
In this subsection, we explain how we use credit assignment to make learning more sample-efficient
Reward shaping In RL, agents often deal with sparse rewards that make the learning process slow. Reward shaping Ng et al. (1999) is technique that aims at densifying the reward so as to improve sample efficiency. It defines class of reward functions that can be added to the original environment rewards without modifying the set of optimal policies. For given MDP M = (S, A, 7, R, P), we define a new MDP M' = (S, A, 7, R', P) where R' = R+ is the shaped reward and F the shaping. The reward shaping theorem states that there exists function e such that F (s, s") + "o(s') - ((s), then M and M' admit the same set of optimal policies. é is called a potential function. With domain knowledge, one can use reward shaping to design more informative reward functions without encouraging unwanted behavior. Nevertheless, shaping rewards requires good priors for the task and the potential function must often be engineered manually.
Since SECRET weighs the contribution of observation-action pairs to future reward, we use it to derive a shaped reward that corresponds to the sum of future reward reachable from the underlying state, weighted by the attention calculated by the model. We explain the process in the following.
Computing the potential function
We define the redistributed return R of trajectory 7 as:
T
T
1 (s, = Σ Hs+ s, ,, a} Σ
are az),
where Oriejj is the attention weight on (0i, when predicting the reward rý and 8; are environment states. Indeed, SECRET observations but we keep the states they constructed from compute the potential. In POMDPs, we recover approximate state from the observation, either manually or through inference. In this work, we use state constructed manually, see Appendix A for details.
To compute the potential function, we generate a set D of trajectories like described in Sec. 2. Since we operate on trajectories, the same state-action pair can appear twice in sequence and benefit from a different amount of attention, which is why we must include the first summation. In the reward shaping formalism, the potential function & depends only on the state. To stay within bounds, we define & as the forwarded redistributed return. It is computed as the following estimate:
((s)
1 D
Σ TED
I1s
R) (2/2, (5/2,a(2))
4

---

Under review as a conference paper at ICLR 2020
Note that in practice only redistribute individual rewards that successfully predicted. Also, some states are generally missing from the data distribution induced by the set of trajectories used. For those states, we set to potential to O, which results in a - -(s) additional reward when transitioning those from the state As result, it gives agents incentive stay on the support of the data distribution unless they encounter high-reward states.
Because it relies on reward shaping, SECRET conserves optimal policies. We empirically find that agents learn faster with the resulting augmented reward function. A way to look at it is that we densify the learning signal and bias the agent towards behaviors that encourage future rewards.
2.3 TRANSFERRING CREDIT ASSIGNMENT
We start by conveying intuition as to why SECRET should transfer to new environments. In fields other than RL, seq2seq models similar to that of SECRET have shown outstanding transfer capabilities (De-
Vlin et al. 2018 Peters et al. 2018 Howard and Ruder 2018), even in low-resource settings Zoph
al. 2016). transfer scenarios that preserve the structure of the MDP, the optimal finegrained control sequence can vary drastically from one environment to another. This is why credit assignment an interesting alternative to transfer of weights: given an underlying environment state and specific action, their contribution to future rewards is not fundamentally altered. Such scenarios include specific changes in the state (or observation) distribution and changes to the reward function that preserve the optimal policies. These also include changes in the dynamies of the environment, and though it affects credit assignment, we show later on that SECRET adapts surprisingly well to such scenarios. Another point that motivates the use of our method for transfer is the fact that we keep the representations learned for credit assignment separate from the control representations learned by agents. Indeed. de Bruin et al. (2018) showed that RL representations were not optimal for transfer.
Transfer setting We argue that transfer should be considered effective when agents learn solve target tasks efficiently because efficiency gains in the target domain compound while the cost of training in the source fixed. Hence, we use the Total Target Time Scenario metric (Taylor and Stone, 2009) to assess transfer. Nevertheless, collecting trajectories in the source domain can be costly. We report the number of trajectories used to train SECRET in each scenario.
As before, SECRET is trained episodes of interaction sampled from the source distribution. In each target environment, we sample multiple trajectories (see the following section for details about the policies used to generate the trajectories). We then compute the attentional potential function by calculating an estimate of the expected redistributed reward, as described Sec. 2.2
EXPERIMENTS
In this section, we aim to answer the following questions: can SECRET improve the sample efficiency of learning for RL agents? Does it generalize and/or transfer? How does it compare to transfer baselines? the credit assigned by SECRET interpretable? The results of complementary experiments are presented in Appendix B.
The Triggers environment We introduce Triggers, an in- terpretable and customizable environment that we use to assess the quality of the credit inferred with method. In Triggers, the agent is located in a two-dimensional bounded grid. Its actions consist solely of moving of one cell in one of the cardinal directions. Any action that would lead the agent outside the boundaries of the environment (as indi- cated by the walls in the figure) is ignored but still counted as an action taken by the agent. The goal of the agent (rep- resented as yellow square) is to activate all the switches (red squares) and then collect all the prizes (pink squares). Prizes are the only source of reward and give 11 penalty unless all switches are activated, in which case they give a +1 bonus. Both prizes and switches disappear once collected.
Figure Example of Triggers envi- ronment.
5

---

Under review as a conference paper at ICLR 2020
The main feature of Triggers is that every positive reward is conditional to the presence of a known subset of states in the agent history, and thus credit assignment can be assessed a rigorous way. Some instances of Triggers can prove challenging to solve optimally for traditional RL methods since agents have to activate every Triggers before experiencing rewards. Triggers environments being MDPs, turn their states into observations by cropping the view around the agent. We 3x3 windows in all our experiments. Trajectories are generated with random policies.
DMLab keys doors We use the _eys_ddoorss____eee_eeeoroaeerre 3D environment from DMLab (Beattie et al. (2016)) in which the agent must locate keys whose colors indicate the doors they open. It can only possess one key, therefore picking the wrong key prevents it from reaching further re- wards. The agent receives as imput what would correspond a first person view of what is in its line of sight. It can move forward, backward and rotate. Each key picked up grants +1 bonus, equally to each door opened. Independently, cake rewards the agent by +50 increase score when collected. We do not apply any transformation to the agent inputs. that setup, agents benefit from understanding the link between keys and doors. We hypothesized that credit assignment mechanism might identify this relation and reward the agent for picking up keys. To assert this, we modified the setting that picking up keys does not provide rewards. Additionally, the visual input is richer than the one from Triggers environments and the average number of steps per episode is extended. Finally, agents move and rotate across the room. Since picking up a key does not require to actually see the key, it can be hard know whether key was taken and predict further door opening rewards. Trajectories are generated with a trained agent.
doors puzzle
100
120
Figure 2: Observations from DMLLa are first person views.
Agents used We use tabular Q-learning (Watkins and Dayan 1992) for in-domain and out-of- domain experiments Triggers except for the transfer environments with modified dynamics where we use Deep Q-Networks (DQN) (Mnih et al.] 2015). We use Proximal Policy Optimization (PPO) (Schulman et 2017) agents for in-domain experiments DMLab.
3.1 CREDIT ASSIGNMENT
We provide analysis of the credit inferred by SECRET. The analysis is qualitative and quantitative, since we rely on both visual assessment and binary detection metrics.
The process of evaluating credit assignment in Triggers goes follows: first generate trajectories and train the model. We then compare the credit assigned by SECRET on trajectories sampled from held-out environments the exact knowledge of where triggers are. It is a vector that is 0 almost everywhere and 1 on state-action couples that precede the activation of Triggers. By doing so, we explicitly target the state-action couples whose resulting state is causally linked to the reward experienced later.
ground truth credit assignment. We build that ground truth by exploiting
We find the redistribution be near optimal in simple instances of Triggers (see Fig, Heft): attention concentrates quasi exclusively on aate-nccion pairs that enable the collection of future reward. This confirmed by precision-reed recall analysis: over the distribution of scenarios considered, a simple binarization heuristic over attention values yields an average precision of 0.96 for an average recall of 0.94. More information on the heuristic is in Appendix A.
In keys_dcors__p puzzze, we adopt the same set of experiments. Since the agent can move backward and spin, in some scenarios it takes key that not in is line sight. addition, the granularity of the state space is such that off-by-one prediction errors are common but do not hinder the credit mechanism: attributing credit to the state-action couple preceding the collection of a key or the previous leads to imperceptible changes in the resulting shaped rewards. Fig, right shows similar results as for Triggers. Appendix A also provides a heatmap for this task that shows that attention concentrates around the keys.

---

Under review as a conference paper at ICLR 2020
5.18
0.7-
0.14
0-6
2,012
0.4
0.100
0.5
MM
0.0
-100
-50 Laking bey spenting
Redrive step trajectory (0) corresponds to activating trigen
stop Inabecaory tngpends
Figure 3: On the left, the distribution of attention weights around triggers for correct positive reward predictions in a 8x8 Triggers maze with triggers and one reward. The x-axis denotes the signed number of steps between the state-action couple receiving attention and the closest actual moment the agent activated switch. On the right, the distribution of attention weights around keys for correct reward predictions for door traversals DMLab.
0.150
10
0.125
0.100
30
(avg on
0.075
20
0.025
10
0.000
vanilla RL Secret
Vanilla
Secret
1.4
Number of spisodes
Number of Interactions
1c6
Figure 4: On the left, in-doniin transfer results on 8x8 Triggers with 3 triggers and 1 reward. On the right, results in DMLab.
3.2 TRANSFER
We then study how we can leverage the inferred credit and transfer representations that are helpful new scenarios. show that agents train faster when using shaped rewards from SECRET. As before, the reward model is trained episodes of interaction in environments sampled from the source distribution. In transfer environments, we sample multiple trajectories, each using the same maze configuration. We then compute the attentional potential function by calculating estimate of the expected redistributed reward, as described in Sec. 2.2] To evaluate its effect, we compare agents trained from environment rewards agents that the resulting shaped reward.
In-domain transfer For in-domain transfer, we transfer the representations for credit assignment to held-out instances the same distribution over MDPs. For the Triggers environment, RL agents tabular Qllearners. For the DMLab environment, 2017) and modify the original task: we do not reward the agent for collecting keys but only to open doors that the attention focus on the key positions. Note that this makes the task harder.
PPO agents (Schulman et al.)
As we display in Fig. agents learn visibly faster to solve tasks when benefitting from SECRET in both environments.
Out-of-domain transfer For out-of-domail transfer we use the Triggers environment and consider two scenarios that are hard for standard agents: transfer to bigger environments (see Fig. 5 and transfer to environments with inverted dynamics (see Fig, 6). In the bigger setting, direct weight transfer cannot be used since the visual input has bigger spatial dimensions. On the other hand, SECRET be used since the transformation visual input dimensions. In the inverted dynamics setting, the effect the agent's 's actions are inverted, which makes the task hard for transfer methods. In that setting, compare the transferability
apply to turn states into observations conserves the
7

---

Under review as a conference paper at ICLR 2020
Out-of-domain transfer (bigger mates]
Out-of-domain transfer (modified dynamics)
De www.mee.............. t0 10
0.150
0.22
0100
0.6
0.073
0.4
0.059
0.2
-0.2
Oecounted reort ty- (V-08% 0e000
0175
Vinlla
Secret
0.22
0.4
0.100
0.2
0.000
0.0
0.023
-02
0.000
Semr
WT
Secret VT
3000 4000 Mumner 40000s tralied
3.0
Number iiforaations
Figure 5: Bigger environments we consider bigger mazes where the structure of the original task is conserved (number of triggers, number of prizes). Environments drawn are 12x12 grids with trigger and 1 prize for the top figure versus prizes for the bottom one. Environments from the training distribution are 8x8 grids.
Figure 6: The controls of the oft-o--domain distribution are inverted (up becomes down, right becomes left). Environments are 8x8 grids with 1 trigger and 1 prize (top figure) or 2 prizes (bottom figure). The effect of the shaping exclusively beneficial, while trans- ferring weights from the source task can be detrimental the learning process.
of our mechanism to that of the representations learned by an agent equipped with deep function approximation. To this end we use DQN agents and either train them from scratch in the target environments or start from the set of weights learned in the source environments (WT in Fig. 6
In both settings, shaping the rewards assists the agent in learning to solve the task. We display some results in Fig. and Fig. [ When transferring to bigger environments, the agent benefits very early on from the shaped reward, while also reaching better asymptotical performance.
RELATED WORK
Transfer in RL. While a lot of approaches exist in the transfer literature, to the best of our knowl- edge none explicitly transfer credit assignment capabilitics. Previous works aimed at making the training of an agent in the same task more sample-efficient by using a pretrained model teacher (Rusu et al. 6a Schmitt et al. 2018). We learn to assign credit as a parallel task that does not modify the representations of the RL agent. Others learn auxiliary reward functions in the hope that they will enable transfer by imposing consistency in the reward (Houthooft et al. 2018 Hessel et al. 2018 Agarwal et al. 2019). Although we also learn additional reward signal, it based on a redistribution of rewards from the environment, which ensures consistency with the original reward function. Transfer is also viewed learning tasks in a sequential way (Rusu al.) 2016b] Kirkpatrick et al. 2017) and this suggests to introduce inductive bias to the neural architectures of agents to reduce catastrophic forgetting. Our method does not require to alter the agent's architecture. Other explicitly address the problem of transfer through the lens of multitask learning (Parisotto) iet al. 2016, Teh et al. 2017) while we stick to learning from an initial distribution of environments. Meta-learning approaches aim to train agents on distribution of tasks environments so that their learned skills and representations work across the underlying continuum, and allow for fast adaptation of the agents (Duan et al. 20166] Wang et al.) 2016 Finn et al. 2017;] Mishra et al. 20188] Co-Reyes et al.) 2019 Zou et al. (2019). In contrast to meta- leatnnl methods, we do not modify the RI. algorithm used train the agent and SECRET is compatible with any core algorithm for RI..
8

---

Under review as a conference paper at ICLR 2020
Credit assignment Previous works investigated the role of attention mechanisms for credit signment. Ke et al. (2018) propose SAB, a sparse attention mechanism used to derive a modified backpropagation algorithm. We draw inspiration from SAB but operate in the RL context without sparsity assumptions about the attention weights. RUDDER (Arjona-Medina et al. 2019) proposes to equip RI agents with online method for credit assignment based on return decomposition. Our method operates offline and decomposes individual rewards. RUDDER requires a specific exploration scheme, an additional episodic replay buffer, a compute-heavy contribution analysis method and the addition of several auxiliary losses to the objective the RL agent optimizes. In comparison, SECRET a lightweight method that does deal with exploration. Crucially, the focus of Arrooa-- Medinal et al. (2019) is online credit assignment while ours is on transfer. Hung et al. (2018) provide an agent with an external memory and the unsupervised task of reconstructing its inputs (both states and rewards). The agent uses memory reads as a way to identify related elements in sequences, and uses those to transfer the value of states providing delayed rewards to the bootstrapping target of contributing elements. In contrast, SECRET makes of con-autoregressive architecture, does not reconstruct states, makes use of reward shaping instead of modifying the update function and most importantly does not rely model that goes backward from high-reward states and samples state-action pairs that could have led that state. SECRET also works backward from high-reward states but creates links previous states from existing trajectories instead of sampling them.
external memory. Recall Traces (Goyal et al. (0188)
generative
CONCLUSION
In this work, we investigated the role credit assignment could play in transfer learning and came up with SECRET, novel transfer learning method that takes advantage of the relational properties of self-attention and transfers credit assignment instead of policy weights. We showed that SECRET led to improved sample efficiency in generalization and transfer scenarios in non-trivial gridworlds and more complex 3D navigational task. To the best of our knowledge, this is the first line of work in the exciting direction of credit assignment for transfer. We think would be worth exploring how S SECRET could be incorporated into online reinforcement learning methods and leave this for future work.
REFERENCES
Agarwal, R., Liang, C., Schuurmans, D., and Norouzi, M. (2019). Learning to generalize from sparse and underspecified rewards. Proceedings of the International Conference on Machine Learning (ICML.201) 2019).
Arjona-Medina, J. A.. Gillhofer, M.. Widrich, M., Unterthiner, T., and Hochreiter, S. (2019). Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems (2019).
Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR 2015).
Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefraneq. A., Green, S., Valdés, V., Sadik, A., et al. (2016). Deepmind lab, arXiv preprint 1612.3:6
Co-Reyes, J. D., Gupta, A., Sanjeev, S., Altieri, N., DeNero. J., Abbeel, P., and Levine. S. (2019). Meta-learning language-guided policy learning. In Proceedings of the International Conference on Learning Representations (ICLR 2019).
de Bruin, T., Kober, J., Tuyls, K., and Babuška, R. (2018). Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters (2018), 333):194-11400
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the Annual Meeting of North American chapter of ACI. (NAACI.2019)

---

Under review as a conference paper at ICLR 2020
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. (2016). RIS Fast reinforcement learning via slow reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR 2017),
Ecoffet, A., Huizinga, Lehman, J., Stanley, K. O., and Clune, (2019). Go-explore: new approach for hard-exploration problems. arXiv preprint urXiv:1901.1000
Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017).
Goertzel, B., Iklé, M., and Wigmore, J. (2012). The architecture of human-like general intelligence. In Theoretical Foundations of Artificial General Intelligence, pages 123-144. Springer.
Goyal, A., Brakel, P., Fedus, W., Lillicrap, T., Levine, S., Larochelle, H., and Bengio, Y. (2018). Recall traces: Backtracking models for efficient reinforcement learning. Proceedings of the International Conference on Learning Representations (ICLR 2019).
Hessel, M.. Soyer, H., Espeholt, L., Czarnecki, W., Schmitt, S., and van Hasselt, H. (2018). Multi-task deep reinforcement learning with popart. arXiv preprint arXiv: 1809v 044744
Houthooft, R., Chen, Y., Isola, P., Stadie, B., Wolski, F., Ho, O. J., and Abbeel, P. (2018). Evolved policy gradients. In Advances in Neural Information Processing Systems (2019), pages 5400-5409,
Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In ACL 2018. Association for Computational Linguistics.
Hung. C., Lillicrap, T. P., Abramson, J., Wu, Y., Mirza, M., Carnevale, F, Ahuja, A., and Wayne, G. (2018). Optimizing agent behavior long time scales by transporting value. CoRR, abs/1810.06722.
Ke, N. R., Goyal, A., Bilaniuk, O., Binas, J., Mozer, M. C., Pal, C., and Bengio, Y. (2018), Sparse attentive backtracking: Temporal credit assignment through reminding. In Advances in Neural Information Processing Systems (2018), pages 7650-7661.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Gtrabsa--Barwinnse A., et al. (2017). Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (2017), 11<(13)3521-3526
Lin, L.J. (1992). Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, 8(3-))223--32.
Lin, Z., Feng, M., dos Santos, C. N., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. (2017). A structured self-attentive sentence embedding. In Proceedings of the International Conference on Learning Representations (ICLR 2017).
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. (2018). A simple neural attentive meta-learner In Proceedings of the International Conference on Learning Representations (ICLR 2018).
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A.. Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540):529.
Ng. A. Y.. Harada, D., and Russell, S. (1999). Policy invariance under reward transformations: Theory and application reward shaping. In Proceedings of the International Conference on Machine Learning (ICML 1999), volume 99, pages 278-287.
Parisotto, E.,, Ba, J., and Salakhutdinov, R. (2016). Actor-mimic: Deep multitask and transfer rein- forcement learning. In Proceedings of the International Conference on Learning Representations (ICLR 2016).
Peters. M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer. L. (2018). Deep contextualized word representations. In Proceedings of the Annual Meeting of North American chapter of ACL (NAACL 20/8).
10

---

Under review as a conference paper at ICLR 2020
Puterman, M. L. (1994). Markov Decision Processes. Wiley.
Rusu, A. A., Colmenarejo, S. G., Gülçehre, Ç., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. (2016a). Policy distillation. In Proceedings of the International Conference on Learning Representations (ICLR 2016).
Rusu, A. A., Rabinowitz, C.,, Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. (2016b). Progressive neural networks. CoRR, abs/1606.0466)
Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., and Gelly, S. (2018). Episodic curiosity through reachability. Proceedings of the International Conference on Learning Representations (ICLR 2019).
Schmitt, S., Hudson, J., Zidek, A., Osindero, S., Doersch, C., Czarnecki, W. M., Leibo, J. Z., Küttler, H., Zisserman, A., Simonyan, K., and Eslami, S. M. A. (2018). Kickstarting deep reinforcement learning. CoRR, abs/1803.03835
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. (2016). High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR 2016).
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. CoRR, 1101177777.44.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, and Salakhutdinov, R. (2014). Dropout: simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15(1):1929-1955
Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks.
In Advances neural information processing systems (2014), pages 3104-3112.
Taylor, M. E. and Stone, P. (2009). Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633-15)
Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. (2017). Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems (2017), pages 4496-4506.
Tieleman, T. and Hinton, G. (2012). Lecture 6.5 RMSPropp: Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. (2017). Attention all you need. In Advances in Neural Information Processing Systems (2017), pages
Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. (2016). Learning to reinforcement learn. CoRR, 0aa//1611.0555
Watkins, C. J. and Dayan, P. (1992). Q-learning. Machine learning, 8(3-4)277-229.
Zoph, B., Yuret, D., May, J., and Knight, K. (2016). Transfer learning for low-resource neural machine translation. ACL 2016.
Zou, H., Ren, T., Yan, D., Su, H., and Zhu, J. (2019). Reward shaping via meta-learning. arXiv preprint arXiv: arXiv:1901.0e0 1901.09330.
11

---

Under review as a conference paper at ICLR 2020
sign(r+)
1
FF softmax
Transformer block
FF + ReLU
O(t-k-I)=t 1)-t
(t--
concat
Conv ReLU
St-k-1 at-k-1
St-k k
ai
St
at
Figure 7: The architecture used for SECRET. a. is the vector containing the attention weights of the model for its prediction step
A ADDITIONAL EXPERIMENT DETAILS
In this section we provide additional details about our experimental setup and the hyperparameters
A.1
TIME LIMITS
In Triggers, episodes stop after 50 timesteps in 8x8 grids and 100 timesteps in 12x12 grids. Those values were chosen large enough such that the policies used train the reward predictor have chance to gather informative rewards. In DMLab, episodes stop after 900 timesteps, as defined by the standard settings.
A.2 REWARD PREDICTION MODEL.
We use the same set of hyperparameters in all our experiments with few variation. In Triggers experiments, we use 128 units per dense layer, 32 convolutional filters and a single convolutional layer process partial states and actions. We apply dropout (Srivastava al. 2014) at several places in the model as a regularizer. We use a dropout rate of 0.1 after dense layers, a dropout rate of 0.2 in the self-attention mechanism, and dropout rate 0.2 in the normalization blocks of the Transformer architecture. Since Transformers create representations via pooling, they need additional information as take into account the relative positions of sequence elements. Hence, we the same positional encoding scheme as in Vaswani et al. (2017) 77,, that is we add sine and cosine signals of varying frequencies the sequence of input embeddings Xemb before the self-attention layer:
PE sin
PE.2111 = cosl
2223/d
12

---

Under review as a conference paper at ICLR 2020
X = Xemb + PE.
In these equations, t refers to the timestep in the sequence, to the dimension of the embedding considered, and has constant value of 10000.
In DMLab experiments, we use two convolutional layers, 16 filters for each, and otherwise identical
hyperparameters.
Typically, we set the class weights in the loss function to te(1) = wo(-1) 1) = 0.499, w()) = 0.02.
Fig. gives overview of the whole architecture.
A.3 HEURISTIC FOR PRECISION- ANALYSIS
In Sec. 3.1 compare the attention vectors get outputs from the reward prediction model to ideal credit assignment with binary metrics. The ground truth we use is a binary vector of the size of the attention vector. Its values are 0 everywhere and for timesteps that correspond to the activation of a Triggers. To do so, we introduce a simple heuristic to binarize the attention scalars: we consider all values above a threshold a to correspond to events to be credited. Then, we can measure precision and recall as in a binary classification paradigm. The precision and recall reported are the average precision and recall over 4 scenarios Triggers with 8x8 grid, trigger and reward; Triggers with a 8x8 grid, 1 trigger and 2 rewards: Triggers with a 8x8 grid, 2 triggers and 2 rewards; and Triggers with a 8x8 grid, 3 triggers and 1 reward. In each scenario, we train the model over a set of 40000 trajectories, each of which is drawn from a randomly sampled maze. Then, we apply it on 5000 trajectories from held-out environments and collect the attention weights corresponding to predictions on timesteps where the agent experiences positive reward. We use a fixed of 0.2.
A.4 TRANSFER IN TRIGGERS
We collect 40000 trajectories sampled from random policies to train the prediction reward model in all distributions of environments.
In experiments involving tabular Q-learning we use online Q-learning with learning rate of 0.1 and a constant greediness factor e also equal to 0.1.
For the out-of-domain transfer experiment with modified dynamics, use a smaller version of the DQN architecture in Mnih et al. (2015). The first convolutional layer has 8 filters, a 3x3 kernel size and stride of 2. The second and the third convolutional layers have both 16 filters, 3x3 kernel size and a stride of 1. Those are completed by a feed-forward layer with 64 units followed by another feedforward layer with many units as the number of available actions. The greediness factor is decayed linearly from to 0.01 over 250000 steps in the environment at train time and has a constant value of 0.001 test time. We use RMSProp (Tieleman and Hinton| 20T2) a base learning rate of 0.00025. We update the target network every 2000 steps and initially fill the replay buffer with 5000 transitions sampled following a random policy. The replay buffer has maximum size of 1000000.
optimizer with
A.5 IN-DOMAIN TRANSFER IN DMLAB
We provide additional details about this setup: we train SECRET using 10000 trajectories sampled from distribution of mazes that are generated randomly. These trajectories are sampled using an agent trained over the same distribution. We do so to increase the proportion of trajectories where rewards are experienced. Indeed, we found that using random policies yielded very few of these. Once the model is trained, we use it to compute the attentional potential function over fixed maze. 1000 trajectories are sampled on the fixed maze using the same policy as the one that generated the trajectories used to train the reward prediction model. Since consecutive frames can be very similar, we consider a positive reward prediction to be correct (and thus use the corresponding attention weights when estimating the potential) if it happens within 5 frames of reward actually experienced in the environment. We then compare the performance of agents trained with the original reward function to those trained with the shaped reward.
13

---

Under review as a conference paper at ICLR 2020
Figure 8: Attention heatmap in DMLab. Attention concentrates around the key which is necessary to unlock reward.
An important point is that we use the knowledge of the position and the keys the agent possesses to build the state used to compute the potential function. This information is not given to the agent. We create approximate state such: it is the concatenation of the discretized position and the identifier the key possessed. The discretized position is the result of the euclidean division by a cell size integer Cs and is necessary since the DML.a position is continuous. es is fixed and has the value of 50 in our experiments. We acknowledge that relying on manually constructed state limits the generality of our approach, but work by using an estimate of the true state.
confident that this limitation be addressed in future
All agents mentioned in that section are PPO learners with learning rate of 0.00019, an entropy coefficient of 0.0011, 12 actors, a discount factor estimation (Schulman et al. 2016) with À 0.95. All those hyperparameter values are taken as from the experiment section in Savinov et al. (2018), whose code is open-source.
0.99. They use generalized advantage
A.6 ATTENTION HEATMAP IN DMLAB
In Fig. we display the per-position average attention weight along trajectories starting in middle-left room that led to opening the blue doors. Attention (shown in shades of red) concentrates around the key position in the lower-left corner.
B ADDITIONAL EXPERIMENTS
In this section we provide the results of additional experiments that are useful to understand several aspects of SECRET.
B. INFLUENCE OF THE STATE TRANSFORMATION IN TRIGGERS
We study the influence of the size of the window in the transformation used to turn states into observations in Triggers. The goal of this experiment notably study the effect of the degree of partial observability to the assigned credit, and also the effect of transferring possibly badly assigned credit to an agent, through reward shaping.
We consider the following window types: 3x3, 5x5 and 7x7 (odd numbers because the window is centered on the agent), as well the full state. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution the source. Environments are 8x8 mazes with 3 triggers and 1 prize. We display reward prediction and credit assignment metrics for each window size in Table and the average discounted return of tabular Q-lcarning agents for each window size in Fig Da Notice that the considered reward prediction accuracy (Table I] is weighted, such that all classes have the same importance.
In this setting, we can observe that bigger window sizes are detrimental to the quality of the credit assigned, even if it helps to predict reward (Table This was be expected: the more "observable" the sequence of observations used to train the reward predictor, the easier is the reward prediction (in the full state case, it can be predicted solely from the current state-action couple). but also the less
14

---

Under review as a conference paper at ICLR 2020
Table Influence of the window size on the reward prediction and credit assignment quality.
Window size
Reward prediction accuracy
Credit precision
Credit recall
3x3
0.67 0.06
1.0 + 0.
1.0 1.0 0.
5x5
0.92 + 0.02
0.57 + 0.04
0.38 + 0.03
7x7
0.75 + 0.12
0.07 0.04
0.03 0.02
Full state
0.76 + 0.14
0.10 + 0.05
0.02 + 0.02
Table 2: Influence of the class weighting on the reward prediction and credit assignment quality,
w(O)
Reward prediction accuracy
Credit precision
Credit recall
1.0
0.72 + 0.15
0.83 + 0.34
0.59 + 0.27
0.72 0.09
1.0 + 0.
0.89 0.05
0.01
0.68 + 0.09
1.0 0.
0.93 + 0.05
0.001
0.65 = 0.04
1.0 + 0.
0.95 0.03
likely the assigned credit will be centered on the trigger. However, even with a low credit precision, the shaped reward still accelerates the learning process (Fig. 9a
B.2 INFLUENCE OF THE CLASS WEIGHTS IN THE REWARD PREDICTION LOSS
We stated that class weighting was important to keep the variance prediction metrics across a variety of datasets of sampled trajectories low, the classes (sign of the reward) being quite imbalanced. Here, we study the influence of the class weights considered in the loss of the reward prediction model in Triggers.
We consider the following class weights for the class corresponding to zero rewards: w(0) = 1, w()) 0.1. w0) 0.01 and w(0) 0.001. The other class weights are fixed and have the value w((1 = w( 1) 1) 1. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution the source. Environments 8x8 mazes with 3 triggers and 1 prize.
We display reward prediction and credit assignment metries for each window size in Table 2 and the average discounted return of tabular Q-learning agents for each window size in Fig 9b In that setting, putting heavier misclassification penalties for under-represented classes (-1 and 1) results in improved credit and RL performance.
B.3 INFLUENCE OF THE AMOUNT OF AVAILABLE DATA
We study the influence of the amount of data used by SECRET in Triggers task. The objective is to assess how data-demanding is the proposed approach, and the effect of lack data on the transfer. There are two types of data: the data from the souree domain (used to train its reward prediction component), and the data from the target domain (used to build the potential function).
B.3.1 SOURCE DISTRIBUTION
We consider the following number of trajectories to train the self-attentive reward predictor: 500, 1000, 5000, 10000, 25000 and 50000. We place ourselves in the in-domain setting and evaluate SECRET on held-out environments from the same distribution the source. Environments are 8x8 mazes with triggers and prize.
We display reward prediction and credit assignment metrics for each window size in Table 3 and the average discounted return of tabular Q- Q-learning agents for each number of trajectories in Fig. 9c We observe that increasing the size of the dataset improves the results, both for the efficiency of the reward predictor (Table and for the transfer (Fig. (9c). Having too few data can slow down learning compared to vanilla agent (Fig. 9c), but it does not prevent from learning solve the task.
15

---

Under review as a conference paper at ICLR 2020
Table 3: Influence of the number of episodes in the source domain on the reward prediction and credit assignment quality.
Number of source episodes Reward prediction accuracy
Credit precision
Credit recall
500
0.53 + 0.11
0.23 0.37
0.09 + 0.2
1000
0.65 + 0.07
0.49 + 0.33
0.52 + 0.42
5000
0.79 = 0.08
0.9 = 0.30
0.9 = 0.30
10000
0.84 = 0.07
0.9 = 0.30
0.9 = 0.30
25000
0.88 0.09
0.9 + 0.30
0.9 + 0.30
50000
0.83 = 0.11
0.8 + 0.4
0.8 + 0.4
B.3.2 TARGET DISTRIBUTION
We consider the following number of trajectories sampled from the target environment to build the potential function: 100, 500, 1000, 2000, 5000, 10000. We place ourselves the in-domain setting and evaluate SECRET on held-out environments from the same distribution as the source. Environments are 8x8 mazes with 3 triggers and prize.
We display the average discounted return of tabular Q-learning agents for each number of trajectories in the source domain in Fdg Here again, we observe that the number of episodes in the target distribution is an important parameter for SECRET, but it has less impact than the number of episodes in the source domain.
0.14
0.12
0.12
0.10
0:10
0.08
0.09
0.08
0.08
0.04
SECRET
(WT-))-1 w101- w11)-11
SECRET (265 window)
SECRET (7X7 window!
0.02
SECRET (1 m1ll-1.)
(w0-3)=11 -1000001110
SECRET (full state)
SECIET (x(-1)_1. w10)-3.000 m11)-111
0.00
2000
6080
5000
8000
10000
12000
2000
4000
5E00
5000
10000
13000
aphoddee
(a) Effect of the window size on SECRET.
(b) Effect of the class weighting on SECRET.
0.14
0.14
0.10
0.10
0.08
0.08
Vanila HL
Vanila HL
SECRET 1500 sploade))
SPCRIT 1100 episndes)
0.04
SECRET 11000 cpicodes)
0.04
SECRET (500 eploodes)
S2:3111 15000 upandee)
34:111 11:00) spasoddee)
0.02
SECRET 110000 epleades)
colsodes)
0.02
SECRET (2000 eprsades
15000
0.00
0.00
Number of sploodes
Number of eploodes
(c) Effect of source data on SECRET.
(d) Effect of target data SECRET.
Figure 9: Effect of various parameters on SECRET.
16

---

Under review as a conference paper at ICLR 2020
04
0.3
02
01
0.0
08
Rebrive step trajectory (0) corresponds to activating trigen
Relattee step trjecttry comesponde serkaating tagper)
Figure 10: On the left, the distribution of attention weights around triggers for correct positive reward predictions in a Triggers maze from an out-of-domain distribution (bigger mazes). The x-axis denotes the signed number of steps between the state-action couple receiving attention and the closest actual moment the agent activated switch. On the right, same figure for another out-of-doman distribution (inverted dynamies).
B.4 ATTENTION DISTRIBUTIONS OUT-OF-DOMAIN SCENARIOS
In the main paper, we have only shown the distribution of attention for an in-domain scenario. Here, following the same protocol as in Sec, 3.1 measure the distribution attention weights in the two out-of-domait scenarios considered in our experiments: bigger mazes and inverted dynamics. The results are reported in Fig. 10 We observe similar distribution of attention, peaked on the triggers.
17


Reviews:
[{"review_id": "Skx8qjXJcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel transfer learning mechanism through credit assignment, in which an offline supervised reward prediction model is learned from previously-generated trajectories, and is used to reshape the reward of the target task. The paper introduces an interesting new direction in transfer learning for reinforcement learning, that is robust to the differences in the environtment dynamics. \n\nI have the following questions/concerns.\n\n1. The authors insist that their fous is on transfer and not competing on credit assignment. If accurate credit assignment leads to better transfer, shouldn't achieving the best credit assignment model (thus competing in credit assignment) lead to better transfer results?\n\n2. What effect does the window size for transforming states to observations have on the performance of SECRET?\n\n3. On a high-level, how does SECRET compare to transfer through relational deep reinforcement learning: https://arxiv.org/abs/1806.01830? Relational models use self-attention mechanisms to extract and exploit relations between entities in the scenes for better generalization and transfer. Although SECRET intentionally avoids using relations, I think a discussion around relational models for RL is warranted. I'm curious what happens if SECRET is allowed to exploit relations in the environment.\n\n4. What happens if the reward model uses very few trajectories and is not able to predict good rewards? Does transfer through credit assignment become detrimental? In other words, in a real-world scenario, how I do know when to start using SECRET, or when am I better off learning from environment rewards alone? Especially given that SECRET requires 40000 trajectories in the source domain.\n\n5. Are the samples generated in the target domain for collecting attention weights included in the number of episodes when evaluating SECRET? For example, in Figure 4. I believe the number of episodes required to collect those target samples should be added to the number of episodes when using SECRET since the agent must interact with the environment in the target domain.\n\n6. On a lighter note, I don't believe using a coffe-brewing machine has a 'universally invariant structure' of coffee-making. That's a luxurious way of making coffee :) In the developing world, we still need to boil water, pour coffee powder in it, etc., all without a coffee-brewing machine.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gkk2lt9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to consider the problem of transfer in the context of sequential decision-making -- in particular reinforcement learning -- from the view-point of learning transferable credit assignment capability. They hypothesize that by learning how to assign credit, structural invariants can be learned which the agent can exploit to assign credit effectively and thus learn more efficiently in new environments (be it in-domain or out-of-domain). They pose the credit assignment problem as learning to predict (sparse) rewards at the end of sub-trajectories, finding the extent to which past state-action pairs appear to be responsible for these rewards (by means of the reward-prediction training), and creating a dense reward function via reward shaping (such that the set of optimal policies does not change). This is appealing as no modifications are needed to the RL algorithm/architecture. To examine their hypothesis, they created a method, called SECRET, based on self-attention and supervised learning to train credit assignment capability offline: sample many trajectories (often a mixture of expert and non-expert ones) from the source distribution, train a self-attentive seq2seq model to predict the rewards in these trajectories offline. Once this model is trained, they apply this model to a relatively small set of sampled trajectories from the target distribution and obtain the attention weights. Then, they use these attention weights as a proxy for credit assignment and, thus, use them to form a reward redistribution function. In their experiments, they show that the average attention weights actually signal the state-actions at which the future reward is triggered. They also show in their experiments that SECRET improves learning performance on in-domain transfer learning (larger mazes), as well as an out-of-domain case (with modified dynamics).             \n\nOverall, this paper proposes an interesting general avenue for research in transfer learning in RL. Regarding the proof-of-concept method and experiments, I need some clarifications. Given these clarifications in the authors' response, I would be willing to increase my score.\n\n1. Regarding this statement on breaking Markov property: \"hide a certain amount of information from states and break the Markov assumption\". \n(i) It is unclear to me what this \"certain amount\" would need to be in general. I believe this would require domain-specific knowledge to know what can be removed to break Markov-ness while not introducing significant state-aliasing (which could hinder the agent's learning). \n(ii) Does any extent of partial-observability warrant that the success in reward-prediction would mean that we have a valid credit assignment model? I feel like this is not generally true, in which case I question the statement on p.3: \"Note that in POMDPs, this is unnecessary since the observations agents get from the environment are incomplete by construction.\"      \n(iii) Regarding generality, the fact that states need to be (manually) preprocessed seems to me like a downside of this approach. Can you see any way around this?    \n\n2. In p.4, this is mentioned: \"In POMDPs, we recover an approximate state from the observation...\".\nI do not see how this is done in the DMLab experiments. If this is done manually, and not trained, then I think it should be clearly stated in the main text. I think the 2nd paragraph of Sec. A.4 is stating that extra information about the state was utilized, and not approximated via a trained model to recover the states (i.e., no s^=h^-1(o) was used)? \n\n3. What is the observation type of Vanilla RL in the out-of-domain experiments? Is it also observing its local-view (similar partial observability as SECRET) or does it have access to the full state? I would argue that it is important that the performance of Vanilla RL with partial observation is reported. Including both cases could also be beneficial. \n\n4. Fig.3 shows attention weights on held-out environments from an identical distribution as the source (i.e., in-domain). \nI would like to see how well the attention signal works when the target distribution differs from the source. Is there a reason why this is not demonstrated?  \n\n5. Not sure about specific definitions of sub-trajectory and trajectory in the paper: \n(i) What constitutes a sub-trajectory (as opposed to a trajectory) in the context of this paper?\n(ii) Are the lengths of the sub-trajectories or trajectories fixed?\n\n6. Why do the attention weights not sum to 1 in Fig.3?\n\n7. Could you clarify the role of positional encoding and how it is done?\n\n\nMinor comments:\n\n1. M is used to denote both MDP and causal map. \n2. Explicitly defining d_i in p.3 should improve clarity.  \n3. Using 40k and 10k trajectories of interactions to train the credit-assignment model (on Triggers and DMLab domains, respectively) seems quite demanding, which seems somewhat unrealistic to deem useful for application to robotics perhaps?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eNjMcpKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on credit assignment using a self-attention module for transfer RL problems. Specifically, the reward signals are assigned backward to previous states according to the attention weights. This can be helpful especially when the reward signal is sparse. Experiments on the newly proposed Triggers environment and the DMLlab keys & doors environment show that the proposed algorithm, SECRET, can speed up training in the transferred environment.\n\nPros:\n- The writing is mostly great.\n\nCons:\n- Some design choices are not well-motivated or even problematic.\n- Experiments are not sufficient.\n\n(1) On page 3, the definition of Z is problematic. The mask matrix M is applied *before* the softmax transformation, which means future values can have non-zero attention. This is because softmax will never produce zero probability. It would still be problematic even if M is applied after the softmax transformation because in this case, the attention for past elements could become very small and almost surely not sum to 1 (except for the last element). Therefore, regardless of the position of M, the attention will be questionable.\n\n(2) Page 4, the weight w(c) is not defined for the weighted cross-entropy. It is claimed that such weighting is essential, but no evidence is provided to support this.\n\n(3) The proposed potential function is not very well-motivated. It is not clear why it should be defined like this instead of other alternatives. Moreover, for never-visited states, the potential is set to 0, which seems to prevent exploration. This would potentially harm the performance in a new environment especially when the training trajectories are far from optimal.\n\n(4) Sec.2.3 says \"given an underlying environment state and a specific action, their contribution to future rewards is not fundamentally altered.\" Can you elaborate? Also what is \"the rank of the rewards\"?\n\n(5) Experiment:\n(5.1) Why Fig.5 and 6 do not have similar asymptotic returns? Given that they both correspond to 1 trigger and 1 (2) prize(s), the asymptotic return should be close.\n(5.2) As mentioned above, it would be interesting to see whether SECRET will prevent exploration if the behavior agent is (heavily) biased. The random agent in the Triggers environment provides sufficient support for the whole state space, while the PPO agent in the DMLab is well-focused on the \"good\" regions. If a \"bad\" agent (say, exploits some low reward regions) is used, SECRET may slow down instead of speed up training in the transfer environment. This is an important scenario to see whether SECRET will potentially create a negative transfer.\n(5.3) No other method from the literature is used for comparison. Several alternatives are discussed in the Related Work \"credit assignment\" section, but none is compared in the experiment.\n\nMinor comments\n- In other fields than RL -> in fields other than RL.\n- The caption of Fig.3-left: there is no \"key\" in the Triggers environment. It uses switches.\n- WT is not explained in Fig.6. \n- h(s) is defined as the observation given a state s, but it is not used in later discussion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach to transfer learning in RL based on credit assignment. The reviewers had quite diverse opinions on this paper. The strength of the paper is that it introduces an interesting new direction for transfer learning in RL. However, there are some questions regarding design choices and whether the experiments sufficiently validate the idea (i.e., the sensitivity to hyperparameters is a  question that is not sufficiently addressed). Overall, this research has great potential. However, a more extensive empirical study is necessary before it can be accepted.
Decision: Reject