Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summarize what the paper claims to do/contribute.
- The paper proposes a new method for zero-shot visual transfer for RL, SADALA. The method first learns a feature extractor with attention (to focus on realted features only) and then learns a policy in the source task and is able to transfer zero-shot int he target domain. The method is evaluated on two tasks: Cartpole-v1 (Gym) and "Collect Good Objects" (Deepmind Lab). It is compared against DARLA for both tasks and against Domain Randomization only for Cartpole. 

Clearly state your decision (accept or reject) with one or two key reasons for this choice.
Reject.
- The experiments of the paper were particularly weak. 
--More standard visual adaptation techniques like DANNs,ADDA, PixelDA/SimGAN, CycleGAN were not considered. 
--The results on domain randomization were not convincing: more details are necessary to determine what the experimental protocol was. One major question: what is the source domain in the case of domain randomization (for Fig. 6) In any case, I find it very hard to believe that simple domain randomization considered here can not fully solve this task for all visual pertrubations considered. 
-- In Fig. 5 the reconstruction is not correct.
-- Domain randomization was not tried on the DeepMind Lab example because of compute. However, I'd encourage the authors to try this. Converging will surely not be  linear to the number of perturbations considered as it seems to be implied. Also the OpenAI paper cited as an example where domain randomization took 100 years of simulation required for transfer is a problem of rather different scale: the domain gap there is between simulation and reality for an anthropomorfic robotic hand, and not a simple visual gap where the color of an identical environment are changed. 

-Related work discussion was insufficient
-- Related work section is missing and work is not adequately placed in the context of existing literature in the Introduction where some related work is indeed discussed.
-- Related work at the last sentence of the introduction is not discussed correctly. It is implied that all these works are on domain randomization which is not true. Also one work (Chebotar et al) is not relevant as from what I recall there was no visual gap. Finally most of these works deal with much more complex visual gaps so sample complexity is hard to be compared.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper proposes adding an attention mechanism to the DARLA beta-VAE approach to transfer learning. The beta-VAE, soft attention and policy are trained on appropriate source tasks and evaluated zero-shot on target tasks, using two more difficult continuous control domains with RGB observations. Results indicate some improvements to compared to the immediate relevant baseline which may be statistically significant, but it is not clear whether over 10% in practice. 

I cannot at this stage recommend acceptance for the following reasons:
1) The paper augments an existing method with a well understood attention mechanism, so the novelty of the approach is relatively low.
2) The experimental results are interesting, but I don't find them compelling enough to recommend acceptance based on the results alone.  The paper does not solve a major problem with the approach it is based on. In fact, the improvement seems to be smaller when the environment is more complex.
3) Several baselines which are cited in the paper are actually missing in the experiments, so it is hard to determine how important is that roughly 10% improvement compared to SOTA.


review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Pros:
This paper proposed a new method for zero-shot transfer learning under the reinforcement learning setting. The use of attention weights to regularize the latent states was fairly interesting.

Cons:
Limited applicability of the proposed methods
- The paper was restricted in a setting where rewards, actions, and true states were identical between source and target environments, and only the observed states differed due to differing renderers. Working under such a restricted setting was interesting in its own right, but it might also lead to limited applicability of the proposed method in the real-world setting.
- The proposed method focused on solving a very specific problem: learning a dis-entangled latent representation for images. As a result, the potential impact of the proposed methods could be minimal.

Limited technical novelty
- The proposed method, SADALA, was built on top of Higgins et al., 2017 (DARLA). The only difference was an added attention layer to the learning of latent states. As a result, the novelty of the proposed method was very incremental and limited from a technology perspective.
- Even with additional attention layer, the paper could have performed a more thorough study to help the readers understand and appreciate the idea. For example, this paper didn’t discuss the tradeoff between training SADALA over separate stages, versus training it from end to end. For example, why the weights of the pre-trained beta-VAE had to be frozen and used as weights in the state representation stage.

Insufficient experiments
-More thorough discussion of the qualitative results should be helpful to understand whether the attention weights helped the model to focus on the right thing. For example, this paper did study the quality of reconstruction in Figure 3-5 of the proposed method. When comparing Figure 3 and Figure 5, it appeared to me that the reconstructed the angle of the pole was different from the original one. And it seemed like attention weights did successfully ignored the color of the cart and pole, but it ignored the angle of the pole, which should be important to the learning task. Unfortunately, the paper didn't further explain the implication of such misrepresentation.

-Quantitative results 
* It would be interesting to all compare the proposed methods against model-agonistic methods like MAML
* It would be useful to include confidence intervals over different tasks.
* It would be useful to compare different methods with different parameter settings
* The authors mentioned “Visual Pendulum tasks” but didn’t include them in the paper


Reproducibility
- It's unclear to me how reproducible the research conducted in this paper was, and it would be useful to open source the code used to conduct the experiments.
review_title: Official Blind Review #2

Meta-review: This paper proposes a new method for zero-shot policy transfer in RL. The authors propose learning the policy over a disentangled representation that is augmented with attention. Hence, the paper is a simple modification of an existing approach (DARLA). The reviewers agreed that the novelty of the proposed approach and the experimental evaluation are limited. For this reason I recommend rejection.
Decision: Reject