Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a method for exploiting structure in locomotive tasks for efficiently learning low-level control policies that pass through waypoints while achieving some goal (typically 3D Cartesian position). This is in contrast to goal-conditioned RL policies that sample random goals during training and are thus sample inefficient, which are trained to execute one policy at a time. In particular, the paper proposes the notion of generalized experiences, where new trajectories are generated from existing trajectories, in such as a way that they are equivalent to each other (in this case translation and orientation invariant) with respect to actions.  

The idea proposed here, of exploiting environmental structures in order better generalize previous knowledge to new, unseen situations is an interesting direction for achieving sample efficiency in practical RL, such as in robotics. 

I have the following comments/questions.

1. If I understand correctly, rather than randomly sampling the environment, the paper proposes starting off with trajectories generated while learning some single-goal policy, and generate from these new ones that the agent must execute in the environment. In which case, the question is how much less interaction does the agent have with the environment compared to random goal sampling, to achieve the same performance?

2. What happens if you use a standard goal-conditioned RL with the Generalized Experiences, without training an IDM? For example, using VGCP with GE, where the goals for VGCP are teminating states of each trajectory. In other words, can a vanilla goal-conditioned RL benefit from the proposed trajectory sampling technique (GE), and how does that compare with random sampling and using the proposed latent representation technique?

3. How do you modify the LFM so that it only accepts the current state and goal, instead of a set of two e.m.o equivalent states and goals? Is the same query treated as two queries that are e.m.o equivalent?

4. In A.1.2, the paper mentions that the first half of the data generated using the RL algorithm is used for generating the second half of data for GE and LR, right? Which means GE and LR are trained from the same number of steps as the baselines, except that GE and LR make use of generalized experiences. However, the paper states in A2 that GE and LR learn from inferior samples. Can the authors please clarify this? My understanding is that GE are generated by modifying existing trajectories and letting the agent apply the same actions by interating with environment. Meaning that, given 2M samples, GE will generate 2M more samples that are e.m.o equivalent by interacting with the environment. 

5. Some of the references in the text do not have year of publication, e.g., Kulkarni et al.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method to learn locomotion and navigation to a goal location or through a set of waypoints for simulated legged robots. The contributions of this paper include 1) generalized experience, which is a data-augmentation technique to add more orientation-invariant experience, and 2) a latent representation to encode the state, the current location and the goal location. The paper compares the proposed method with a few baselines and demonstrates better performance.

My recommendation of this paper is Weak Reject. Although the method seems reasonable and the evaluation shows good results, I think that the paper can be improved for the following three reasons. 

First, it is not clear to me how the inference works if the goal is reasonably far away (e.g. can be reached in 10 steps) from the current position of the robot? Since the inverse dynamics model only outputs an action given the current position and the immediate goal (in the next time-step), how is the 10-step action sequence planned using the 1-step immediate goals? 

Second, the details of data collection are unclear to me. I believe that the policy \pi used for data collection plays an important role. Which \pi is used? It would be clearer to present this part in the main text, not Appendix. If the data is collected from an RL agent which learns to walk to the right, how does the robot learns to turn when walking across different waypoints (Figure 7)?  

Third, in this paper, the generalized experience is to add different initial orientations of the robot. I think that the similar effect can be achieved by reparameterizing (s, o, o') into polar coordinates: (s, \theta, r), where (\theta, r) is the intermediate goal location relative to the robot's current orientation and position. \theta=0 means the goal is in front of the robot, and r is the distance between the goal and the robot. In this representation, all the generalized experience will reduce to a single (s, \theta, r), which is invariant to the robot's orientation. This would already be a good latent space, without any learning. For this reason, I would suggest adding one more baseline to compare the proposed method against: using (s_t, \theta_t, r_t, a_t, s_{t+1}, \theta_{t+1}, r_{t+1}) to represent experience, then run the proposed method without generalized experience and latent representation. Will this baseline achieve similar or even better results? 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Learning Latent Representations for Inverse Dynamics using Generalized Experiences

In this paper, the authors propose to utilize the symmetry property in locomotion problems (more specifically navigation problems), and more efficiently generate additional training data from existing data, and learns a more efficient representation.

I tend to vote for rejection for this paper mostly because, while it seems to me to be a very efficient and practical engineering project, but relatively lack the novelty in terms of the algorithm.

Pros:
- The experiments are of good quality, providing a lot of ablation studied and hyper-parameter specifications.
- The proposed idea is combined with some of the state-of-the-art algorithms, showing it’s compatibility and good practical performance.

Cons:

- The proposed algorithm lack novelty.
Goal conditioned reinforcement learning, where a generalized inverse dynamics is used, has been widely studied in [2, 3].
And the use of symmetry has also been studied [1].
The augmentation of data by considering symmetry is relatively straight-forward.

- limited to navigation environments
The proposed methods do not seem to be directly applicable to tasks other than navigation, where a very task-specific goal position can be provided.


[1] Yu, Wenhao, Greg Turk, and C. Karen Liu. "Learning symmetric and low-energy locomotion." ACM Transactions on Graphics (TOG) 37.4 (2018): 144.
[2] Ding, Yiming, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. "Goal-conditioned Imitation Learning." arXiv preprint arXiv:1906.05838 (2019).
[3] Merel, Josh, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. "Neural probabilistic motor primitives for humanoid control." arXiv preprint arXiv:1811.11711 (2018).
review_title: Official Blind Review #1

Meta-review: Solid, but not novel enough to merit publication.  The reviewers agree on rejection, and despite authors' adaptation, the paper requires more work and broader experimentation for publication.
Decision: Reject