Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary - Building on top of the domain randomization principle (used to train policies robust to domain-variations) to learn policies which transfer well to new domains, the paper proposes an approach to improve and speed-up learning / training over randomized environments. The paper operates in a settings where the policy to be transferred only has access to observations -- images, etc -- and not the complete underlying state of a (simulated) environment. The underlying idea is to -- (1) maintain two sets of actor-critic networks - a symmetric pair where the actor has access to the underlying state and an asymmetric pair where the actor has access only to image observations; (2) evenly gather experiences from behavioral policies of both actors and store them in a shared replay buffer and (3) learn to align the attention placed by the policies over objects in the environment for the state and observation based actors. The idea is to leverage privileged information about the state (which is strictly more informative compared to observations) to learn robust observation based policies. Experimental results indicate the proposed approach improves generalization performance compared to several ablations of the same on both in-distribution and out-of-distribution environments.

Strengths

- The paper is generally well-written and easy to follow. The authors generally do a good job of motivating the proposed approach by leveraging access to privileged information in the environment / simulation / renderer during training to get more robust observation policies to transfer to novel settings. The proposed approach is presented after appropriately grounding the problem setting and preliminaries and the authors clearly state and evaluate on the axes of research questions they care about.

- The proposed approach is somewhat novel and extends prior work on using shared replay buffers and asymmetric actor critic methods to accelerate training. Although the specific focus is on aligning object-level attention from a state-based actor and an observation based actor, the authors adopt design choices that help in preventing degenerate solutions -- for instance, the object-weighted squared error loss component to learn the observation attention module.

- The experimental results more-or-less support the claims of the paper (however, only in comparison with ablations and 1 baseline) in the sense we see improvements in terms of average returns and sample-efficiency plots. Furthermore, the authors conduct ablations to understand which components of the proposed approach contribute significantly in different environments -- for both extrapolated and interpolated environments.

- Sec 4.6 presents interesting analysis of the learned attention (observation) attention mechanism for both interpolated and extrapolated environments. It seems that attention is generally placed over relevant aspects (objects / links) of the environment and the associated takeaways seem feasible.

Weaknesses

- Having said that, there some weaknesses / questions which if addressed would make the paper stronger and help in increasing the rating of the paper.

- Access to the object-specific attention maps seems to be an assumption that might not scale well across simulators. More realistic / richer simulations (say, 3D reconstructions of indoor rooms) may not always offer this much privileged information -- one might have access to fine-grained reconstructions from multiple viewpoints, but not object level maps. This, combined with the fact that major gains have only been demonstrated over the specified continuous control domains (ignoring the Atari results), makes me slightly concerned about the scalability of the proposed approach in terms of more real-world + applicable domain-transfer scenarios. Maybe a more general approach that learns to match some intermediate representations of the state and observation based actors is a more general approach. Can the authors comment on this?

- Including the entropy loss while training the attention mechanism for the state based actor is justified only through feasible interpretations of the attention visualizations for the JacoReach experiments. I’m curious how important is it for the state-based attention to be sparse? Does it actually affect performance if the entropy loss is not included while learning the state-based attention module?

- Although the paper mentions experience is gathered evenly over the behavioral policies of both the actors it’s slightly unclear how that is being performed without actually referring to the algorithm pseudocode in the appendix. I would encourage the authors to include / move the same to the main paper. It makes the entire pipeline much easier to grasp.

- Transfer to novel environments (not necessarily with domain-shifts) has also been studied in context of providing exploration incentives (see InfoBot - https://arxiv.org/abs/1901.10902) in addition to a (sparse / dense) episodic reward. I would be curious to see how well does APRiL compare to such approaches in a setting where both are applicable - say the MultiRoomNXSY set of experiments in InfoBot (pointed above). This is to understand if the gains obtained from APRiL are very specific to domain-shifts or are largely applicable to any form of novel environment transfer. Can the authors comment on this?

Reasons for rating

Beyond the above points of discussion, I don’t have major weaknesses to point 	out. I generally like the paper. The authors do a good job of identifying the sliver in which they make their contribution and motivate the same appropriately. My major point of concern is centered is around the fact that APRiL (probably) assumes access to much privileged information which may not be generally available across all kinds of environments, etc. My rating of the paper is based on the strengths and weaknesses highlighted above. Addressing / Responding to those appropriately would definitely help in improving the rating of the paper.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Overall the method is interesting but I am not overly convinced the method provides a significant improvement over simpler methods that are not compared to the work here. Also, the generalization and extrapolation experiments need more description. As well, the work talks about how this method can be used to accelerate learning for transfer to real-robotic systems but does not have an example of this.

More detailed comments:
- When you are referencing a paper that is very similar to your method and you build off of it is good to link to the published version ( the reference for Asymmetric DDPG was published at RSS2018.)
- While the method is very interesting I feel that one of the obvious baselines that would be good to try is to train a model that maps o -> s (image to state). This is a very simple method and appears to accomplish the goals for the authors. A discussion on why this would not be good enough and some results to show that this performs poor would be good addition.
- Related to the last point why are object-specific segmentation maps needed? It seems like the method alone was not capable enough to learn from pixels alone. Also, Can these segmentation maps be visually described? It is not very clear. Where do the segmentation maps come from?
- Can you describe the shared replay buffer in more detail? What does no shared replay buffer mean?
- The caption in Figure 2 is rather sparse and comes before the explication of the different methods in the figure. This figure needs more explanation. What are the other versions of April? What is the baseline? There does not appear to be a significant improvement? Can you show more of a qualitative improvement via videos of the policy performance?
- It appears the method does not work as well on the Atari games can the authors provide some insights into why the method does not offer as much of an improvement? Is there no Attentive DQN for Pong, natural?
- I do not understand what experiment is being performed in section 4.3.2. Maybe it would help if you explained what a canonical image is? Where do you get the compressed state information?
- In 4.4 you show that APRiL does not do better than the baseline? What is the baseline?
- In section 4.5 you describe how the method generalizes to the task with additional distractors. Can you describe specifically how this experiment is designed? How do the distractors compare to the other objects in the scene? are their colors changed? How many are there? This section needs a lot more detail to understand how much the reader can surmise about the methods ability to generalize.
- In the same section, the authors say the image segmentation is crucial. Does this imply that maybe we should just put an image segmentation network in between the observation and the policy inputs? What is using this additional attention mechanism better than that simpler method?
- For Table 1: It is interesting that the baseline does better than the APRiL no share or no sup. It might be important to perform an exploration as to why having those features reduces performance and the baseline appears more robust.
- It would be nice to see this work applied to more tasks. For now the reacher and walking are the most interesting but those tasks are also somewhat basic. How would this method work on the walker 3d where there is even more partial observation?
- Can the authors explain more how they have deduced from the attention map that the attention has figured out how to indirectly discover where the Jaco links are without explicit attention? I do not see this.
- For figure 4, how the attention is visualized could be explained better. Are the white regions receiving all the attention? Also, What makes the difference between a Interpolation example and a Extrapolation example? Is it just more objects in the scene?
- The authors should also reference "Sim-to-Real Transfer of Robotic Control with Dynamics Randomization".
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The topic addressed by the paper is domain adaptation and transfer learning in the text context of deep reinforcement learning, in particular the “sim2real” problem, where a policy is learned in simulation and should be transferred to a physical agent in a real-world scenario. The work builds on the existing “asymmetric DDPG” formulation (Pinto et al., 2017), which exploits the fact that full states are sometimes available in simulated environments but not during deployment. In Pinto et al., this is addressed by learning an actor taking as input observations, and a critic which has access to the state.

The contribution of the paper is an extension of Pinto et al. by adding additional communication between the state head (which learns a critic) and the observation head (which learns the policy). This is done through an attention mechanism, now very classical in deep learning, which weights the different elements of the (state) input. The particularity here is that the attention mechanism is trained on the critic, which takes the full state as input, and then transferred to the observed input of the policy. The problem is that the state is very different from the observations, which are images. An alignment module expands the distribution over state variables to a distribution over observed objects.

One of my main concerns here is lack of justification and lack of clarity. While the abstract of the paper and introduction section are well written, and wants us to learn more about this interesting idea, the paper kind of falls apart in the subsequent chapters. The authors mention alignment of attention, but the exact motivation for the algorithm, i.e. the motivation for its formulation, are never provided. The description itself is also far from clear, as key design choices are not introduced and we discover them, or guess them, from equations. We don’t know, for instance, what the observations are, and think that they are images. However, apparently an object mask detector is run over the input images, as segmentation maps are input to the state-object alignment module, which needs to distribute the attention over states to attention over objects.

This also means that the formulation is quite specific to the task at hand, since it seems to be object-centric.

Figure 1 is another example of the same problem: There are many arrows, but their signification is unclear. Dashed lines are indicated to mean alignment, but alignment is not a standard term, for instance comparable to a computation connection, or a loss signal.

A similar confusion is found in the equations themselves, as h(s) is mapped to a vector c, but this vector decomposes into different objects of the environment. What is T? It being upper case could mean a matrix or a scalar value, but we are not sure. From the loss function we see that this seems to be vector (a distribution) of the same size as the attention vector h_o … which is over what?

Basically, as I understand it, the alignment loss minimizes some structured mapping between some attention vector over objects on the observation side and some attention vector over states on the critics side. This seems to be a weak loss signal, as the attention transformation needs to be learned together with the attention mechanism itself (and of course the actor and the critic).

At the beginning of section 3, the method is introduced of having an asymmetric part and a symmetric part, but I don’t see this, as this would require learning 4 predictors and not 2 (an actor and a critic). An asymmetric model, as given in Pinto et al., uses two predictors, an actor and a critic, with different loss functions and, more importantly, different inputs. Here, the authors claim that the observation module itself is asymmetric … but how can something by asymmetric if it contains only an actor? Same, the critic is supposed to be symmetric … but without an actor?

As I see it, w.r.t to the question whether the model is symmetric or asymmetric, the formulation is identical to Pinto et al., thus asymmetric. The difference lies in the attention mechanism and the alignment module.

At some point in the paper, this method is mentioned to be self-supervised … I am not sure which aspect of this work could be called self-supervised, but I agree that the definition of this relatively new term is sometimes ambiguous.

On the other hand, I think the method should be compared to “classical” self-supervision in RL, which corresponds to predicting information which is available during training but not available as input (depth prediction etc.). Here, a natural baseline seems to be a symmetric model (same input for actor and critic: observations) and to predict the state of the model and self-supervise it during training but not deployment.

The evaluation is unfortunately not convincing.

Two important baselines are missing:
-	Self-supervision, as mentioned above
-	Pinto et al., on which this method is based.

There are some hics in the results … for instance the curves in Figure 2 show learning in progress, they are not yet converged. Also, the standard deviations are quite big.

Another downside is that, although sim2real is used as a motivation for this paper (since this is the most typical scenario where states are available during training but not during testing), the experiments have not been performed using physical agents. Testing is performed on degraded simulated agents. I do understand that physical environments are harder to manage than simulated ones, but tiny physical environments can be obtained for reasonable prices, and this would have made the paper much stronger and more aligned with its core motivation. Simulated noise cannot replace a physical environment.

No details have been given on the additional distractor objects unseen during training.

The Walker 2D environment is described as “modified”, but how exactly, and why?

Section 2.1 (definition of POMDPs) seems to be unnecessary and can be deleted.

review_title: Official Blind Review #3

Meta-review: This paper tackles the problem of transferring an RL policy learned in simulation to the real world (sim2real). More specifically, the authors address the situation where the agent can access privileged information available during simulation, for example access to exact states instead of compressed representations. They perform experiments in various simulated domains where different aspects of the environment are modified to evaluate generalization.

Major concerns remain following the rebuttal. First, it is not clear how realistic it is to assume access to such privileged information in practice. Second, the experiments are not convincing since the algorithms do not appear to have reached convergence in the presented results. Finally, a sim2real work would highly benefit from real-world experiments.

In light of the above issues, I recommend to reject this paper.
Decision: Reject