Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a strategy for single-trajectory transfer of a reinforcement learned policy.  They follow a typical approach in the few-shot supervised-learning community, of assuming that the plausible set of solutions may be modelled as a much lower dimensional latent variable, and then try to quickly infer that latent variable at test time.  In this case, the latent variable is ‘Z’.
At test time, they first run an exploitative algorithm (they call this the probe) to rapidly infer the value of Z, and thereby the optimal policy which this indexes, before switching to running the policy alone.

Weak reject.

I think that this paper (if indeed novel) is interesting, and I do agree that few-trajectory transfer in RL is a potentially impactful area in which to be working.  I think that the magnitude of the contribution is medium->low, however (namely that they have an approach which is wall-clock efficient, for single trajectory transfer), however the precise details of this contribution/claim have not been adequately tested:
We do not see experiments which highlight the wall-clock or inference cost competitiveness of the approach, and we do not see experiments against other, existing, approaches for latent-variable based RL (mostly these involve model-based RL, however these would probably be easy to derive for many of the test cases).
We also don’t see any experiments which evaluate the performance of SEPT as the time spent optimising the probe policy is varied.
As an aside, in figure 2(e) it appears that the Oracle policy is significantly outperformed by SEPT, do you have any thoughts as to why this might be?

To improve the paper, I would propose a slight rewrite to focus only on the core claim (i.e. why this model outperforms anything else for wall clock competitive single episode RL policy transfer).  I’d also be interested to see how this approach scales as the size of the potential policy set increases (e.g. if we had to produce a policy which would generalise for for all ball sports).
Any theoretical work would also be very welcome—one may be able to draw on existing work in the k-shot community to make a judgement as to what fraction of the trajectory is required before one should have a low entropy estimate of Z, and it would also be nice to know how the approach scales as the complexity of Z ramps up—i.e. as we need to support more tasks.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary

This paper addresses the problem of transfer in RL. After an agent is given an opportunity to train from a distribution of environments, we want an agent to perform well on the test environment. This paper specifically focuses on the setting where the state space, action space, reward space, and discount factor are the same across all environments, while the transition dynamics may differ. An environment's transition dynamics is assumed to depend on a hidden parameter that is not observed by the agent, in contrast to some previous work which assumes observability.

The idea of the main algorithm is as follows. During the training phase, a series of environments is presented. The agent is aware of the demarcations between the environments, but not of the identity (i.e., hidden parameter) of the environments themselves. A "probing" policy is run for a specified number of time steps. From the trajectory generated, a VAE is used to estimate the hidden parameter governing the transition dynamics. For the remaining time in the environment instance, a master policy, conditioned on the estimated parameter, is trained on the reward (any usual RL algorithm suffices; DDQN is used in the experiments).
The novelties of the approach are the problem setting (no access to optimal policies, immediate deployment at test-time), the joint training of a probing policy and VAE model, and a universal policy conditioned on the estimated hidden parameters. The experiments were conducted on domains that were, individually, both continuous and discontinuous with respect to the hidden parameters. The paper concludes that the proposed method improves upon baselines in terms of speed of reward attainment and computation time. Ablations were run for some of the components of the proposed algorithm. 
 
Decision: Weak Reject
 
The two key reasons for the Weak Reject were the following. 
1. Statistically insignificant results (3 runs)
2. Questions about baselines used (see question 6 below)

The proposed algorithm is promising and attempts to directly address limitations in previous literature. The paper thoroughly contextualizes and motivates the current approach in light of previous work in the literature and discusses possible advantages: simplicity and generality; the ability to deploy immediately at test-time instead of having to train another policy (from learning a universal policy); assuming that the hidden parameters are unobservable, which is more realistic; not assuming access to optimal value functions/policies; the algorithm is model-free, so one can avoid the computational and memory overhead (when compared to BNN) of learning a model and avoid compounding model error. I also appreciated the thorough discussion of the key algorithmic choices and their trade-offs in section 3. The presentation of ideas is clear throughout.
 
However, weak experiments keep this paper at a weak reject. The paper claims that the proposed algorithm "significantly outperforms" the baselines (abstract); however, more runs are needed to substantiate these claims. Only 3 runs were made for the test episode, which is not enough to be able to make empirical claims with non-trivial confidence (see https://arxiv.org/abs/1709.06560).  As well, since baseline methods are adapted from the original papers, more thorough hyperparameter sweeps should be performed. I wasn’t sure what “coarse coordinate search” was (pg. 6), but in general it would be good to write down the hyperparameters swept over. 

Some additional questions about the experiments:
1. Did you try oracle with lower dimensional hidden parameter embedding? (understanding more why oracle does possibly worse in HIV and Acrobot)
2. There is the claim on page 2 that the proposed algorithm is "orders of magnitude faster than best model-based method". What is the best model-free method?
3. Is the universal policy of SEPT optimized during test time?
4. Did you manage to replicate the results of DPT in Yao (2018)? Were architectures the same?
5. What is the significance of not requiring rewards at test-time?
6. Why were BNN, MAML, and EPOpt the baselines chosen? What about the other works mentioned in the related work section, like Tirinzoni (2018) or Paul (2019)? Did you try using an LSTM as a baseline to directly learn the policy? One could imagine treating the existence of a hidden parameter as inducing a partial observability problem. 
7. What is a "coarse coordinate search" over hyperparameters? What hyperparameters were tried?
8. Did you try regular DQN? DDQN w/o prioritized replay? 
9. Why were the cumulative reward curves not as significant for Acrobot and HIV as compared to 2D navigation?
10. Why different probe length settings for 2D navigation vs. Acrobot and HIV? How did these hyperparameters affect the time to solve?
11. Why do 2D-room and Acrobot have terminal rewards of 1000 and 10?

I would be willing to raise the score if the key experimental problems I noted were addressed. 
 
Minor comments/questions that did not impact the score
1. Should include the table of computational time in the main body if possible since computational efficiency is one of the core claims
2. Move DDQN comment in 3.2 to the experiments section
3. How does the probing policy approach relate to work in active perception? (https://link.springer.com/article/10.1007/s10514-017-9666-5)


EDIT: After the comments below which addressed my concerns, I have raised the score for this paper from a weak reject to an accept. 


review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The main contribution of this paper is to learn a universal policy that is able to perform near-optimally on test tasks with transition dynamics that were never observed during training. This is achieved by using a "probe policy" to generate short trajectories that are then used to learn a latent encoding to categorise the transition dynamics of the current task. The universal policy is then conditions on both the state and this encoding so that the learned policy can perform well on tasks with different dynamics.

Overall I really like this paper. There has been a big push in the RL community to start evaluating algorithms on test tasks that are different to trained tasks and this paper takes a good step in this direction.

My main concern with this paper, which I would appreciate some feedback from the authors on, is regarding the trajectory length of the probe policy:

1. The method seems to rely heavily on the ability to learn an accurate latent encoding from a short trajectory which will may not be the case in many domains. It is easy to construct some domain where the dynamics behave identically initially but have some major differences deeper into the task at hand. I noted that the authors did mention this in their conclusion as an area for future research but I want to know if they have any high level ideas on how to rectify this because it seems that this will be a major limitation of the proposed algorithm in practice at least for certain domains.

2. In the experiments I can see that in some cases a longer probe performed more poorly than a shorter probe (for example Fig3a for 2D navigation. My intuition tells me that a longer probe should outperform a shorter probe since it contains at least as much information. Can the authors explain why this occurs?
review_title: Official Blind Review #1

Meta-review: This is an interesting paper that is concerned with single episode transfer to reinforcement learning problems with different dynamics models, assuming they are parameterised by a latent variable. Given some initial training tasks to learn about this parameter, and a new test task, they present an algorithm to probe and estimate the latent variable on the test task, whereafter the inferred latent variable  is used as input to a control policy.

There were several issues raised by the reviewers. Firstly, there were questions with the number of runs and the baseline implementations, which were all addressed in the rebuttals. Then, there were questions around the novelty and the main contribution being wall-clock time. These issues were also adequately addressed.

In light of this, I recommend acceptance of this paper.
Decision: Accept (Poster)