Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes to consider the problem of transfer in the context of sequential decision-making -- in particular reinforcement learning -- from the view-point of learning transferable credit assignment capability. They hypothesize that by learning how to assign credit, structural invariants can be learned which the agent can exploit to assign credit effectively and thus learn more efficiently in new environments (be it in-domain or out-of-domain). They pose the credit assignment problem as learning to predict (sparse) rewards at the end of sub-trajectories, finding the extent to which past state-action pairs appear to be responsible for these rewards (by means of the reward-prediction training), and creating a dense reward function via reward shaping (such that the set of optimal policies does not change). This is appealing as no modifications are needed to the RL algorithm/architecture. To examine their hypothesis, they created a method, called SECRET, based on self-attention and supervised learning to train credit assignment capability offline: sample many trajectories (often a mixture of expert and non-expert ones) from the source distribution, train a self-attentive seq2seq model to predict the rewards in these trajectories offline. Once this model is trained, they apply this model to a relatively small set of sampled trajectories from the target distribution and obtain the attention weights. Then, they use these attention weights as a proxy for credit assignment and, thus, use them to form a reward redistribution function. In their experiments, they show that the average attention weights actually signal the state-actions at which the future reward is triggered. They also show in their experiments that SECRET improves learning performance on in-domain transfer learning (larger mazes), as well as an out-of-domain case (with modified dynamics).             

Overall, this paper proposes an interesting general avenue for research in transfer learning in RL. Regarding the proof-of-concept method and experiments, I need some clarifications. Given these clarifications in the authors' response, I would be willing to increase my score.

1. Regarding this statement on breaking Markov property: "hide a certain amount of information from states and break the Markov assumption". 
(i) It is unclear to me what this "certain amount" would need to be in general. I believe this would require domain-specific knowledge to know what can be removed to break Markov-ness while not introducing significant state-aliasing (which could hinder the agent's learning). 
(ii) Does any extent of partial-observability warrant that the success in reward-prediction would mean that we have a valid credit assignment model? I feel like this is not generally true, in which case I question the statement on p.3: "Note that in POMDPs, this is unnecessary since the observations agents get from the environment are incomplete by construction."      
(iii) Regarding generality, the fact that states need to be (manually) preprocessed seems to me like a downside of this approach. Can you see any way around this?    

2. In p.4, this is mentioned: "In POMDPs, we recover an approximate state from the observation...".
I do not see how this is done in the DMLab experiments. If this is done manually, and not trained, then I think it should be clearly stated in the main text. I think the 2nd paragraph of Sec. A.4 is stating that extra information about the state was utilized, and not approximated via a trained model to recover the states (i.e., no s^=h^-1(o) was used)? 

3. What is the observation type of Vanilla RL in the out-of-domain experiments? Is it also observing its local-view (similar partial observability as SECRET) or does it have access to the full state? I would argue that it is important that the performance of Vanilla RL with partial observation is reported. Including both cases could also be beneficial. 

4. Fig.3 shows attention weights on held-out environments from an identical distribution as the source (i.e., in-domain). 
I would like to see how well the attention signal works when the target distribution differs from the source. Is there a reason why this is not demonstrated?  

5. Not sure about specific definitions of sub-trajectory and trajectory in the paper: 
(i) What constitutes a sub-trajectory (as opposed to a trajectory) in the context of this paper?
(ii) Are the lengths of the sub-trajectories or trajectories fixed?

6. Why do the attention weights not sum to 1 in Fig.3?

7. Could you clarify the role of positional encoding and how it is done?


Minor comments:

1. M is used to denote both MDP and causal map. 
2. Explicitly defining d_i in p.3 should improve clarity.  
3. Using 40k and 10k trajectories of interactions to train the credit-assignment model (on Triggers and DMLab domains, respectively) seems quite demanding, which seems somewhat unrealistic to deem useful for application to robotics perhaps?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel transfer learning mechanism through credit assignment, in which an offline supervised reward prediction model is learned from previously-generated trajectories, and is used to reshape the reward of the target task. The paper introduces an interesting new direction in transfer learning for reinforcement learning, that is robust to the differences in the environtment dynamics. 

I have the following questions/concerns.

1. The authors insist that their fous is on transfer and not competing on credit assignment. If accurate credit assignment leads to better transfer, shouldn't achieving the best credit assignment model (thus competing in credit assignment) lead to better transfer results?

2. What effect does the window size for transforming states to observations have on the performance of SECRET?

3. On a high-level, how does SECRET compare to transfer through relational deep reinforcement learning: https://arxiv.org/abs/1806.01830? Relational models use self-attention mechanisms to extract and exploit relations between entities in the scenes for better generalization and transfer. Although SECRET intentionally avoids using relations, I think a discussion around relational models for RL is warranted. I'm curious what happens if SECRET is allowed to exploit relations in the environment.

4. What happens if the reward model uses very few trajectories and is not able to predict good rewards? Does transfer through credit assignment become detrimental? In other words, in a real-world scenario, how I do know when to start using SECRET, or when am I better off learning from environment rewards alone? Especially given that SECRET requires 40000 trajectories in the source domain.

5. Are the samples generated in the target domain for collecting attention weights included in the number of episodes when evaluating SECRET? For example, in Figure 4. I believe the number of episodes required to collect those target samples should be added to the number of episodes when using SECRET since the agent must interact with the environment in the target domain.

6. On a lighter note, I don't believe using a coffe-brewing machine has a 'universally invariant structure' of coffee-making. That's a luxurious way of making coffee :) In the developing world, we still need to boil water, pour coffee powder in it, etc., all without a coffee-brewing machine.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work focuses on credit assignment using a self-attention module for transfer RL problems. Specifically, the reward signals are assigned backward to previous states according to the attention weights. This can be helpful especially when the reward signal is sparse. Experiments on the newly proposed Triggers environment and the DMLlab keys & doors environment show that the proposed algorithm, SECRET, can speed up training in the transferred environment.

Pros:
- The writing is mostly great.

Cons:
- Some design choices are not well-motivated or even problematic.
- Experiments are not sufficient.

(1) On page 3, the definition of Z is problematic. The mask matrix M is applied *before* the softmax transformation, which means future values can have non-zero attention. This is because softmax will never produce zero probability. It would still be problematic even if M is applied after the softmax transformation because in this case, the attention for past elements could become very small and almost surely not sum to 1 (except for the last element). Therefore, regardless of the position of M, the attention will be questionable.

(2) Page 4, the weight w(c) is not defined for the weighted cross-entropy. It is claimed that such weighting is essential, but no evidence is provided to support this.

(3) The proposed potential function is not very well-motivated. It is not clear why it should be defined like this instead of other alternatives. Moreover, for never-visited states, the potential is set to 0, which seems to prevent exploration. This would potentially harm the performance in a new environment especially when the training trajectories are far from optimal.

(4) Sec.2.3 says "given an underlying environment state and a specific action, their contribution to future rewards is not fundamentally altered." Can you elaborate? Also what is "the rank of the rewards"?

(5) Experiment:
(5.1) Why Fig.5 and 6 do not have similar asymptotic returns? Given that they both correspond to 1 trigger and 1 (2) prize(s), the asymptotic return should be close.
(5.2) As mentioned above, it would be interesting to see whether SECRET will prevent exploration if the behavior agent is (heavily) biased. The random agent in the Triggers environment provides sufficient support for the whole state space, while the PPO agent in the DMLab is well-focused on the "good" regions. If a "bad" agent (say, exploits some low reward regions) is used, SECRET may slow down instead of speed up training in the transfer environment. This is an important scenario to see whether SECRET will potentially create a negative transfer.
(5.3) No other method from the literature is used for comparison. Several alternatives are discussed in the Related Work "credit assignment" section, but none is compared in the experiment.

Minor comments
- In other fields than RL -> in fields other than RL.
- The caption of Fig.3-left: there is no "key" in the Triggers environment. It uses switches.
- WT is not explained in Fig.6. 
- h(s) is defined as the observation given a state s, but it is not used in later discussion.
review_title: Official Blind Review #3

Meta-review: The paper introduces a novel approach to transfer learning in RL based on credit assignment. The reviewers had quite diverse opinions on this paper. The strength of the paper is that it introduces an interesting new direction for transfer learning in RL. However, there are some questions regarding design choices and whether the experiments sufficiently validate the idea (i.e., the sensitivity to hyperparameters is a  question that is not sufficiently addressed). Overall, this research has great potential. However, a more extensive empirical study is necessary before it can be accepted.
Decision: Reject