Learning Latent Representations for Inverse Dynamics using Generalized Experiences
Authors: Aditi Mavalankar
Sicun Gao
Venue:  (2020)
Keywords: deep reinforcement learning
continuous control
inverse dynamics model

Abstract: Many practical robot locomotion tasks require agents to use control policies that can be parameterized by goals. Popular deep reinforcement learning approaches in this direction involve learning goal-conditioned policies or value functions, or Inverse Dynamics Models (IDMs). IDMs map an agent’s current state and desired goal to the required actions. We show that the key to achieving good performance with IDMs lies in learning the information shared between equivalent experiences, so that they can be generalized to unseen scenarios. We design a training process that guides the learning of latent representations to encode this shared information. Using a limited number of environment interactions, our agent is able to efficiently navigate to arbitrary points in the goal space. We demonstrate the effectiveness of our approach in high-dimensional locomotion environments such as the Mujoco Ant, PyBullet Humanoid, and PyBullet Minitaur. We provide quantitative and qualitative results to show that our method clearly outperforms competing baseline approaches.

Under review as a conference paper at ICLR 2020
LEARNING LATENT REPRESENTATIONS FOR INVERSE DYNAMICS USING GENERALIZED EXPERIENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Many practical robot locomotion tasks require agents to use control policies that can be parameterized by goals. Popular deep reinforcement learning approaches in this direction involve learning goal- condiiiodd policies or value functions, or Inverse Dynamics Models (IDMs). IDMs map an agent's current state and desired goal to the required actions. We show that the key to achieving good performance with IDMs lies in learning the information shared between equivalent experiences, that they can be generalized to unseen scenarios. We design training process that guides the learning of latent representations to encode this shared information. Using a limited number of environment interactions, agent is able to efficiently navigate to arbitrary points in the goal space. We demonstrate the effectiveness our approach in high-dimensional locomotion environments such the Mujoco Ant, Py Bullet Humanoid, and PyBullet Minitaur. We provide quantitative and qualitative results show that our method clearly outperforms competing baseline approaches.
INTRODUCTION
In reinforcement learning (RL), an agent optimizes its behaviour to maximize a specific reward function that encodes tasks such as moving forward reaching target. After training, the agent simply executes the learned policy from its initial state until termination. In practical settings in robotics, however, control policies are invoked the lowest level of larger system by higher-level components such perception and planning units. In such systems, agents have to follow a dynamic sequence of intermediate waypoints, instead of following a single policy until the goal achieved. A typical approach to achieving goal-directed motion using RI. involves learning goal-conditioned policies or value functions (Schaul et al. (2015)). The key idea is to learn a function conditioned on combination of the state and goal by sampling goals during the training process. However, this approach requires a large number of training samples, and does not leverage waypoints provided by efficient planning algorithms. Thus, it is desirable learn models that compute actions to transition effectively between waypoints. A popular class of such models is called Inverse Dynamics Model (IDM) (Christiano al. (2016); Pathak et al. (2017)). IDMs typically map the current state (or a history of states and actions) and the goal state, to the action.
In this paper, we address the need of efficient control module by learning generalized IDM that can achieve goal-direction motion by leveraging data collected while training a state-of-the-art RL algorithm. We do require full information of the goal state, or a history of previous states to learn the IDM. We learn on a reduced goal space, such as 3-D positions to which the agent must learn to navigate. Thus, given just the intermediate 3-D positions, or waypoints, our agent can navigate to the goal, without requiring any additional information about the intermediate states. The basic framework of the IDM shown in Fig. I
The unique aspect of our algorithm is that we eliminate the need to randomly sample goals dur- ing training. Instead, we exploit the known symmetrie/equivvllnees of the system (as is common in many robotics settings) to guide the collection of generalized experiences during training. We propose class of algorithms that utilize the property of equivalence between transitions modulo the difference in fixed set of attributes. In the locomotion setting, the agent's 's transitions are sym- metric under translations and rotations. We capture this symmetry by defining equivalence modulo orientation among experiences. We use this notion of equivalence to guide the training of latent

---

Under review as a conference paper at ICLR 2020
Policy
Sp. Of
Inference
Env
Env
π
11+1
at
Waypoint
Training
(st, op 4p Sp+1 9p+1)
i
at
Experience
Data Collection
Basic framework of the Inverse Dynamics Model. Experiences are collected in normal training under single-goal reward function and used to train the IDM to produce actions for the agent to move from o, to 0r+1- inference time, the current state 8, and position on are passed to the IDM, along with immediate next goal/waypoint 0++1 to get the required action d;.-
Figure 1:
representations shared by these experiences and provide them as input to the IDM to produce the desired actions, as shown in Fig. + A common challenge faced by agents trained using RL tech- niques is lack of generalization capability. The standard way of training produces policies that work very well on the states encountered by the agent during training, but often fail on unseen states. Achieving good performance using IDMs requires both these components: collecting generalized experiences, and learning these latent representations, as we demonstrate in Section 6 Our model exhibits high sample efficiency and superior performance, in comparison to other methods involving sampling goals during training.
We demonstrate the effectiveness of our approach in the Mujoco Ant environment (Todorov et al. (2012) ) in OpenAl Gym (Brockman et al. (2016)) and the Minitaur and Humanoid environments in PyBullet (Coumans & Bail Bai (2016)) From limited number of experiences collected during training under a single reward function of going in one direction, our generalized IDM succeeds at navigating arbitrary goal positions in the 3-D space. We measure performance by calculating the closest distance to the goal an agent achieves. We perform ablation experiments to show that (1) collecting generalized experience
the form of equivalent input pairs boosts performance over all baselines, (2) these equivalent input pairs can be condensed into a latent representation that encodes relevant information, and (3) learning this latent representation is fact critical to success of our algorithm. Details of experiments and analysis of results can be found in Sections 5 6
RELATED WORK
Several recent works learn policies and value functions that are conditioned over not just the state space, but also the goal space (Andrychowicz al. (2017); [Schaul et al. (2015); Kulkarni et al. (2016) and then generalize those functions to unseen goals. Goal-condittoned value functions are also largely used hierarchical reinforcement learning algorithms (Kulkarni et al. (2016)), where the higher level module learns over intrinsic goals and the lower level control modules learn sub- policies reach those goals, the lower level control modules can efficiently execute goals pro- posed by the higher-level policy (Nachum et al.] (2018)) Ghosh et al. (2018) use trained goal- conditioned policies lear actionable latent representations that extract relevant information from the state, and use these pretrained representations to train the agent to excel at other tasks. |Pnn** al. (018) learn coal-conditioned value functions, and use them in a model-based control setting.
IDMs are functions that typically map the current state of the agent and the goal state that the agent aims to achieve, to the desired action. They have been used in a wide variety of contexts in existing literature. Christiano et al. (2016) train IDMs using history of states and actions and full goal state information for transferring models trained in simulation to real robots. Pathak et al. (201)) and Agrawal et al. (2016) use IDMs in combination with Forward Dynamics Models (FDMs) predict actions from compressed representations of dimeesiinnii inputs like RGB images generated by the FDM. Specifically, Pathak al. (2017) use IDMs to provide curiosity-based reward signal
2

---

Under review as a conference paper at ICLR 2020
Figure 2: We report the results of our algorithm and baselines the three locomotion environments shown above: Humanoid, Minitaur, and Ant. The red sphere indicates the goal in each environment.
in the general RL framework to encourage exploration; Agrawal et al. (2016) IDMs provide supervision for the learning of visual features relevant to the task assigned to the robot.
We circumvent the need learn goal-conditioned policies value functions by combining IDMs with known symmetric properties of the robot. We train an IDM conditioned on the state space and reduced goal space, using data collected while training any state-of-the-art RL algorithm. Our data collection is unique in that we exploit equivalences in experiences observed during training and learn latent representation space shared between such equivalent experiences. Our IDM produces the action given this latent representation as an input, leading to generalization over parts of the state and goal spaces unobserved during training.
PRELIMINARIES
In the general RI. framework (utton et al. (998)), ), a learning agent interacts with an environment
modeled as Markov Decision Process consisting of: 1) state space 8, 2) an action space A, 3) a probability distribution P: S x S A - [0, 1]. where P[ a) is the probability of tran- sitioning into state s' by taking action a in state s, reward function R S A S + I that gives the reward for this transition, and 5) a discount factor r. The agent learns a pol- icy :T: + A. parameterized by 8 while trying to maximize the discounted expected return (()) = E Beg. Be..e..... IEr=o ['2 Qz, S141)]. Goal-conditioned RL optimizes for learning policy that maximizes the return under a goal-specific reward function Rg On-policy RL algorithms, such Policy Gradient methods (Williams) (1992); Mnih et al.) (0166) and Trust Region methods (Schulman et al. ( Wu et al. (2017) use deep neural networks to estimate policy gradients, maximize surrogate objective function subject to certain constraints. Off-policy RL algorithms (Lillicrap et al. 2016): Haarnoja et al. 2018) incorporate elements of deep Q-learning (Mnih et al. 2013)) into the actor-critic formulation. Hindsight Experience Replay (HER) (Andrychowicz al. 2017 ) is popular technique used in conjunction with an off- off-policy RL algorithm to learn policies sample-efficient way from sparse rewards in goal-based environments.
LEARNING GENERALIZED INVERSE DYNAMICS
Our method leverages samples collected while training a state-of-the-art RL algorithm to train an IDM that maps the current state and desired goal position to the action required to reach the goal. There are four major components involved in this process: 1) collecting data while training the RL algorithm, 2) learning basic IDM that maps the current state and the desired goal to the required action, 3) collecting experiences that are equivalent to those observed, and using them to train the IDM, and 4) learning latent representation that generalizes this model to unseen parts of the state space by utilizing the equivalent experiences collected step 3. We elaborate on each of these in the following sections.
4.1 INITIAL TRAINING AND COLLECTING EXPERIENCE
Our goal in this steps is to collect data for our IDM in the process of learning policy under single reward function. Recall the motivation for learning IDMs: we want a model that can take in the current state of the agent and the next command, in the form of location in the space that the

---

Under review as a conference paper at ICLR 2020
Figure 3: Example of trajectories that are equivalent modulo orientation (e.m.o.). The red sphere represents the goal. Here, we that successively applying the same action two Ants in different initial orientations results in trajectories that are e.m.o.
agent should travel to. Thus, collect state, action, and position data from the transitions observed during the training process.
We emphasize the difference between the state space S and the goal space O. The state space S is high-dimensional, consisting of information related to joint angles, velocities, torques, etc. The goal space, O, is low-dimensional, consisting, in this case, of the 3-D coordinates of the goal position that the agent is supposed to navigate to.
Definition (Experiences). We define experiences as tuples (s, o' ,a), where is the current state of the agent, is its current 3-D position, and is the action that the agent performed to move from the state and position to the next position, intermediate goal, o'. We write E, {(s, o, o', a)i to denote all the experience tuples collected from trajectory T of length T.
4.2 LEARNING THE INVERSE DYNAMICS MODEL
Given set of experiences E, we can train the IDM using supervised learning techniques. Definition (Inverse Dynamics Model), We define the Inverse Dynamics Model (IDM) as
S O x A, 0,', + a
(1)
where s, o, of and represent the current state, current position, desired goal, and action respectively.
The IDM can reproduce seen actions on state and observation tuples that have appeared in the training data. However, it can not generalize good behaviour to states and observations that have not appeared in the initial training (see Fig, 6 for qualitative evidence). Our aim in the next steps is to generalize the observed experiences so that they can be used over previously unseen inputs from the O x O space.
4.3 COLLECTING GENERALIZED EXPERIENCES
One issue with the current hypothesis of collecting data for single reward is: the samples obtain are highly biased in that they predominantly contain samples for motion in one direction. As result, there unreasonable to expect it to generalize its behaviour in those parts. We can see qualitative evidence in Fig. 6 where the Humanoid can navigate to goals during training time (6)) using the basic IDM (corresponding method in the plots is RL), but fails when the goal lies outside the training distribution (6p).
certain parts of the input space that agent is unlikely to ever visit. So it is
In order to mitigate this bias in the state space introduced by single reward function, we collect generalized experience comprising experience tuples that are equivalent modulo orientation (e.m.o.) with respect to actions. We use θ to represent the orientation space.
Definition 3 (Equivalence modulo Orientation). For e1, e2 E E, e and e2 are e.m.o. with respect
to A + A(ei) = A(ez). This defines an equivalence mod è with respect to A over S x O x O
0s. ~A (3, Ô, o'), where Ya A,p(o' 8, o, a) = p(o') 8, 0,
(2)
A qualitative example of e.m.o. experiences is shown in Fig.

---

Under review as a conference paper at ICLR 2020
St, 0t. 1
Generalized Experience
c.m.o. samples
Shared weights
L2 Loss
Or-
Mcan Latent
Representation
Latent Embeddings
Latent Feature Model
Inverse Dynamics Model
Figure 4: Learning latent representations using generalized experience: The LFM generates the latent representations for two e.m.o. samples. We minimize the distance between them enforce equivalence, and also fit the IDM to predict the action correctly, given these latent representations.
Definition 4 (Generalized Experience). We collect unseen experiences equivalent to the observed experiences by taking trajectory S F., changing the initial orientation of the agent, leading to a different and repeating the same set of actions observed in We denote this operation by G, so (T) is new trajectory. The full generalized experience set obtained in this way written as G E' U G(E). so is unseen state that has not appeared while training the agent.
4.4 LEARNING LATENT REPRESENTATIONS
Despite using generalized experiences during training, the IDM does not always show great im- provements in tasks like navigating the desired goal position in arbitrary direction, as seen in Table We hypothesize that this is due to the agent failing to recognize e.m.o. experiences, and instead learning actions from irrelevant attributes of the state space. We use the e.m.o. experiences from the generalized experience set G to train a Latent Feature Model v that discards irrelevant in- formation from the state, and learns only shared information relevant the IDM produce actions.
Definition (Latent Feature Model). Our Latent Feature Model (LFM) aims to learn the equivalence between e.m.o. experiences from the generalized experience set G. We define the LFM v as
G R v8s8, 0(), =
where y is k-dimensional latent representation of the experience sample. We then modify the IDM á to produce the action from this latent representation as
A, ((v(s,
V(s,o,o'
In order to learn these two models: LFM and IDM, we need our objective function incorporate the property of equivalence modulo actions in the latent representations, and learn good mapping from these representations actions.
Since the LFM is used to generate latent representations for two e.m.o. experience samples simul- tancously, and then optimize their distance, we use a Siamese framework (Koch et al. (2013)) to model s. Our objective L1 minimizes the distance between the latent representations generated for e.m.o. experience samples.
(88, o, d') where (s, o, o,),,,) 03, E G, and (s,0,0') ~ (8,0,0 0') (5) Next, we use simple regression loss such as the L2 distance to fit the output of the IDM to the action. Here, the input to the IDM the mean of the latent representations generated for the e.m.o. experiences.
C. ||v
L2 [1 7)) -al2. where 7 as ((), and (8,0,0') ~A (8,0,0') (6)
5

---

Under review as a conference paper at ICLR 2020
We jointly learn these two models by minimizing weighted loss function:
L = XC1 + (1- 2) where À 10,
Fig. shows the training procedure we use for our IDM. Each e.m.o. sample pair from equivalent trajectories is passed as imput to the LFM, which generates the latent representations for the pair. The mean of these latent representations then passed as input to the IDM, which predicts the action to be taken. These two models are trained simultaneously, resulting in rich latent representations that can achieve goal-directed motion, generalizable to any arbitrary location.
Remark 1. important to note that at test time, only current state and goal are passed to LFM to generate the latent representation, which is used by the IDM to predict the required action.
EXPERIMENTS
We demonstrate the effectiveness of our overall approach by performing a series of ablation ex- periments, successively including each major component of our algorithm. In addition to a random baseline, we use Vanilla Goal-Conditioned Policy, and Hindsight Experience Replay as baselines for comparison with our algorithm. We demonstrate superior results using our algorithm on three loco- motion environments: Mujoco (ooorov et al. (2012) Ant environment in OpenAl Gym (Brockman) al. (2166), and Humanoid and Minitaur environments PyBullet (Coumans Bail (2016)). In order to failly evaluate performance, we train each baseline using the same number of environment interactions as our algorithm. We also maintain uniform seeds, initial states and goals across all methods. The details of the test setting, along with network architectures, learning rates, and other hyperparameters, are discussed the Appendix.
5.1 BASELINES
Since our method aims at achieving goal-directed motion, we compare with other on-policy and off-policy RL algorithms that are trained for this specific purpose.
Random Sampling (RS): In this baseline experiment, we collect (s,0,0',a) samples by taking random actions at each step. We use these samples to train the IDM.
Vanilla Conl-Conditioned Policy (VGCP): The second baseline algorithm we consider is VGCP, which takes as input the state and desired goal, and learns a policy this input space using any state-of-ther -fttt--arr RL algorithm. The policy is given by TVGCP: S. O + .A and is learnt using a state- of-the-art model-fie freee--fre RL technique. We use Proximal Policy Optimization (PPO) (SSchulman al.) 2017) for Ant and Minitaur, and Soft Actor-Critic (SAC) (Haarnoja et al.) 2018)) for Humanoid.
We select Soft Actor Critic (SAC) (Haarnoja et al. (2018) and Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. (2016)) as the off-policy algorithms used conjunction with HER for our algorithms. We report results on HER with both sparse and dense rewards. Sparse rewards indicate whether the target was successfully reached, and dense rewards include this information, in addition to control and contact costs. Throughout the paper, HER-Sp refers HER with sparse rewards, and HER-De refers to HER with dense rewards.
Hindsight Experience Replay (HER):
5.2 ABLATION EXPERIMENTS
Collecting Experience using standard RL algorithm (RL): We collect samples while training state-of-the-art RI. algorithms (Schulman et al. (2017); Haarnoja H et al. (2018)) rewarding them for going to the right (as is common in locomotion environments), and use them as the training data for IDM. More details are listed in the Appendix.
Collecting Generalized Experience (GE): We collect generalized experiences in the following manner: we save the trajectories followed by the agent while it learns policy for locomotion. For some/all of these trajectories (details in A.1.2). we rotate the initial state of the agent by a random angle, and repeat the actions taken in the original trajectory. The samples collected in this modified trajectory are e.m.o. to those in the original trajectory. All of these samples constitute generalized experiences, which
to train the IDM.

---

Under review as a conference paper at ICLR 2020
Model
Distance to target
Model
Distance target
Model
Distance target
Mean
Std. dev.
Mean
Std. dev.
Mean
Std. dev.
RS
3.159
0.579
RS
1.598
0.282
RS
3.166
0.569
RL
2.777
0.841
1.403
0.438
RL
2.155
1.054
VGCP
2.160
0.955
VGCP
1.358
0.326
VGCP
2.608
0.607
HER-Sp
2.902
1.138
HER-Sp
1.440
0.237
HER-Sp
2.602
0,953
HER-De
2.891
I.135
HER-De
1.380
0.251
HER-De
2.475
0.693
GE 1.936
0.977
GE
1.212
0.485
GE
1.721
0.978
LR
1.779
0.986
LR
0.904
0.482
LR
1.105
0.866
Table 1: Quantitative results of our methods and all baselines on the three environments: Humanoid, Minitaur, and Ant respectively. In each environment, LR emerges as clearly the best performer, enabling the agent to navigate closest to the goal.
Learning an IDM from Latent Representations (LR): We use the generalized experiences col- lected in the previous step extend the learned behaviour of the agent to unseen parts of the state space. We use the dual network architecture shown in Fig. in this experiment. We jointly train the LFM and IDM using a weighted loss function that minimizes the distance between the latent repre- sentations generated for e.m.o. experiences, and fits the output of the IDM to the desired actions.
Remark 2. We preprocess (s, o') to (s, where d is the unit vector in the direction - o', and provide it as input to the models.
5.3 NAVIGATING THROUGH A SERIES OF WAYPOINTS
We also show results task in which the agent has navigate through series of waypoints, to see if our IDM has indeed learnt the right actions. There are three questions we wish to answer through this experiment: 1) How much does the agent deviate from the intended optimal trajectory through the intermediate goals/waypoints? 2) How fast the agent able to accomplish each goal? 3) What is the agent's s style/gait of walking?
For this qualitative comparison, we found that neither HER nor VGCP with the current setting was able to generate good trajectories. Thus, we trained VGCP 20 million samples (our method 15 million environment samples). We use this trained policy to generate the VGCP trajectories in Fig. and our LR model (trained using 15 million samples) for the LR trajectories, For all trajec- tories, the agents are given the same initial orientation and goal position to ensure fair comparison.
RESULTS
In this section, analyze the performances of our algorithm and baselines gualitatively and quan- titatively. In particular, we discuss the following: 1) overall performances of all methods, 2) distri- bution of distances to the goal observed at test time for all methods, 3) comparison between perfor- mance of best baseline and LR on the waypoint navigation task, and 4) analysis speed, trajectory, and walking style observed in LR and the best baseline Humanoid.
The first two points are addressed in the following two subsections. The next two questions are discussed in detail in Fig. 7
6.1 ANALYZING OVERALL PERFORMANCE
We report the closest distance from the target that the agent is able to reach, as the evaluation metric for all our experiments. The mean and standard deviation of closest distance are reported in Table It is clear that for each of the three environments, GE and LR outperform all other baselines. In particular, LR is observed to be the best performing algorithm for each environment. This lends validation our claim that learning latent representations shared between equivalent experiences indeed boosts performance, instead of treating the equivalent experiences as independent training samples.
7

---

Under review as a conference paper at ICLR 2020
603
80
M
HER-Sp
Figure 5: Violin plots showing the distribution of test data i.e. closest distance from the goal for each episode for the 3 environments: (a) Humanoid (b) Minitaur (c) Ant.
0.20
-0.00
3.75
-100
-128
(8)
Gool
Origin
RL
VGGP
HER-Sp
HER-De
GE
Figure 6: Qualitative comparison between different algorithms at test time for Humanoid.
The results in table show that for the Minitaur and Humanoid environments, the performance of VGCP is better than most baselines, and nearly comparable to GE. However, it shows poor per- formance on the Ant environment. This anomaly arises from the fact that million samples are not enough to train a goal-conditioned policy for the Ant. The Minitaur and Humanoid, on the other hand, are trained for million and 15 million time steps respectively, thus enabling better goal-conditioned policies to be learnt, leading to much superior performance. For each of the envi- ronments, we observe that HER-Sp and HER-De both show poor performance, compared to that of GE, LR, and even VGCP.
6.2 DISTRIBUTION OF TEST SAMPLES
In this section, we analyze the violin plots in Fig. 5 These plots show the distributions of the closest distances from targets observed over 10,000 episodes for each algorithm. For each environment, see that for RL, and some cases, VGCP, HER, and GE, there is a small peak away from the mean, which gives a much lower distance than the mean distance. This suggests that there is significant number of episodes in which the Humanoid reaches very close the target. We analyze this discrepancy in performance qualitatively in 6 and conclude that the small peak consists of episodes in which the initial state and goal position have been observed during training, and thus, the goal-conditioned policy has already learnt the optimal actions for this scenario.
Across all environments, Ant and Minitaur particular, the peak the LR distribution is much lower than the other methods. This shows that in most episodes, the agent can successfully navigate to the goal; the slightly higher mean value and variance are due to the small number of episodes in which the agent fails at the beginning of the episode. This is to be expected because the actions taken by the IDM depend on the kind data collected during the training process. In some cases, the model may learn the wrong actions, leading the agent dying early in those episodes.
8

---

Under review as a conference paper at ICLR 2020
23
23
20-
1.-
10-
04-
C0-
-05
-0.
Orign WkysomyyCrah
Ln Iraectoryy VGEP retooy
Drigin
LB craenaation
.*. LB trajectoryy
Whypointtyaais
VCCP oriorration
VGKP trayectryy
Figure 7: Given series of waypoints, we check the ability of our Humanoid to navigate through them efficiently, and compare it with the best baseline, VGCP, trained on 20 million samples of environment interaction (our algorithm uses 15 million). (a) We plot the trajectory followed by each agent as it navigates through the waypoints. We see that LR can navigate through all goals much more easily than VGCP, without deviating. In addition this, LR enables the agent reach each goal much faster than VGCP, which unable to reach the last goal due to its slow speed, as the episode terminates in a fixed number of timesteps. (b) We plot the orientation of the agent uniform intervals throughout the episode. We observe that while the agent trained using LR walks forward as expected, the agent trained using VGCP, while managing move slowly closer to the goal, does so in an arbitrary fashion in that its orientation is seldom in the direction of its motion.
We also provide qualitative evidence prove that the distribution of closest distances in episodes is indeed biased by initial state and target configurations seen in the training data. We fix the initial state of the agent and generate goals specific region. We plot the results of 10 episodes for each method in Fig. 6 for the Humanoid environment. The initial state and goal configurations we show results for, consist of those experienced during training (left), and those not experienced during training (right). We see that for the configurations experienced at training time, most baseline methods, and our methods, GE and LR, are able to reach close to the goal. However, when the configuration lies outside the training distribution, only LR (sometimes GE) can navigate to the goal. We include more results in the Appendix.
7 CONCLUSION
We propose a new algorithm to achieve goal-directed motion for a variety of locomotion agents by learning the inverse dynamies model on shared latent representations for equivalent experiences. To this end, we take three important steps: (1) We utilize the experience collected by our agent while training a standard reinforeement learning algorithm, so that our IDM has " "good" samples in which the agent walks reasonably well. (2) We generalize this experience by modifying the initial config- uration for each observed trajectory in the collected data, and generate the equivalent trajectories.
(3) We learn the important shared information between such symmetric pairs of experience samples through a latent representation that is used as an input to the IDM to produce the action required for the agent to reach the goal. We provide extensive qualitative and quantitative evidence to show that our methods surpass existing methods to achieve generalization over unseen parts of the state space.
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In Advances in Neural Information Processing Systems 29, pp. 5074-5082, 2016.
Marcin Andrychowicz. Filip Wolski. Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systemx, pp. 5048-5058, 2017.

---

Under review as a conference paper at ICLR 2020
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016.
Paul F. Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from simulation real world through learning deep inverse dynamics model. CoRR, abs/1610.03518, 2016.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. Gitflub repository, 2016.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: 2017.
Dibya Ghosh, Abhishek Gupta, and Sergey Levine. Learning actionable representations with goal- conditioned policies, CoRR, abs/181 1.07819, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning, pp. 1861-1870, 2018.
Ashley Hill, Antonin Rafin,, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. com/ hi 1 sts/staalleeeee baselines 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations, ICLR 2015, 2015.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in Neural Information Processing Systems 29, pp. 3675-3683. 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR 2016, 2016.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICMI. 20/6, pp. 1928-1937, 2016.
Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnet (eds.), Advances in Neural Information Processing Systems 31, pp. 3303-3313. 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML, 2017.
Vitchyr Pong*, Shixiang Gu* Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep RL for model-based control. In International Conference on Learning Repre- sentations, 2018
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML215, pp. 1312-1320, 2015.
10

---

Under review as a conference paper at ICLR 2020
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML 2015, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press, 1998.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033, IEEE, 2012. ISBN 978-1-4673-1737-5. URL http:
//dblp.un htt.-:/wwec...e///////o/////////
Ronald J Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine learning, 8(3-4):229-255 1992.
Yuhuai Wu, Elman Mansimov, Roger Grosse, Shun Liao, and Jimmy Ba. Second-order optimiza- tion for deep reinforcement learning using froneccer-factored approximation. In NIPS 2017, pp- 5285-5294, 2017.
A APPENDIX
A.1 EXPERIMENTAL SETUP
A.1.1 ENVIRONMENTS
We perform experiments on the Mujoco (Todorov et al. 2012) Ant environment in OpenAl Gym [Brockman et al. 2016), and Humanoid and Minitaur environments in PyBullet (Coumans & Bai 2016). In each of these environments, for our methods, the agent is trained perform well on the 3-D locomotion task in one direction. For the baseline methods, the agent is trained to reach goals generated in different 3-D positions.
The reward function for collecting data for RL, GE and LR includes contact and control costs, and rewards for moving forward. The reward funetion for training VGCP includes contact and control costs, but instead of moving forward, the agent is encouraged to move in the direction of the goal. In the case of HER-Sparse, the agent only receives a reward if reaches the goal, and -1 otherwise. For HER-Dense, the agent receives weighted sum of distance the goal and contact and control costs as its reward.
Humanoid Humanoid is bipedal robot, with 43-D state space and 17-D action space. The state contains information about the agent's height, orientation (yaw, pitch, roll), 3-D velocity, and the 3-D relative positions of each of its joints (knees, shoulders, etc.). The action consists of the torques at these joints.
Minitaur Minitaur is quadrupedal robot, with a 17-D state space and 8-D action space. The state contains information about motor angles, torques, velocities, and the orientation of the base. The action consists of the torque to be applied at each joint.
Ant Ant is quadrupedal robot, with a 111-D state space and 8-D action space. The state space contains the agent's height, 3-D linear and angular velocity, joint velocities, joint angles, and the external forces at each link. The action consists of the torques at all the joints.
Our choice of these locomotion enviromments is driven by the motivation provided earlier: want an agent to navigate to a desired goal position in the 3-D space. In view of this, we do not use 2-D locomotion environments like Half-Cheettie Walker, or Hopper. Also, though Reacher and Pusher are goal- based environments, we do not use them as they are manipulation environments, and we aiming to achieve navigation by agents trained 3-D locomotion tasks.

---

Under review as a conference paper at ICLR 2020
A.1.2 TRAINING DETAILS
The details of number of environment interactions, network architectures for the IDM, LFM, policy, and the state-of-the-art RL algorithm used to train the policy (for baselines), or collect data (for our methods), are given in Table 2
We use the Adam optimizer (Kingma & Ba (2015) with learning rate of le-3 and batch size of 512 for all our methods across all environments.
For the Humanoid environment, we observed that the first ~ 5 million samples were predominantly bad samples that led to the Humanoid falling down very early in cach episode. Thus, we froze training after 5 million samples, when we had policy in which the agent was able to walk a few steps. We used the policy at that point to collect 10 million steps of environment interaction. For the Ant and Minitaur environments, we collected million and 4 million steps of environment interactions respectively, during training.
For RL, we used all steps encountered in on-policy data. For GE and LR, we used only the first half of training samples encountered while training the policy. The second half of the samples is generated by applying the generalized transformation G. explained in Section 4.3 We use the first million, 2 million and 5 million samples collected for the Ant, Minitaur and Humanoid (trained) respectively, and generate the rest by applying G.
The hyperparameters involved in training the LFM and IDM are: the dimension k of the latent representation space, and the ratio of the latent representation loss (C1) to the regression loss (C2), A. We set À = 0.25 for all 3 environments. We use k = 10 for the Ant and Minitaur environments, and k = 50 for the Humanoid environment. We discuss the impact of the latent representation dimension h on the performance of the model in the next section.
A.1.3 TEST SETTING
At test time, the agent is rotated by random angle. The target is set at distance of ~ 2 5 units from the agent at an angle of 11445,459 to the agent's orientation in the case of the Ant and Humanoid environments. For Minitaur, we set the target at distance of ~ 1.5 2.5 units from the agent at an angle of [-45%, ] to the agent's orientation. The intermediate goals, or waypoints, can be provided by any planning algorithm, since our approach is agnostic to the actual planning algorithm used. In our work, we use Model Predictive Control, i.e. we replan at each step.
Each episode consists of a maximum of 1000 steps for each environment, and the episode terminates when the agent reaches the goal, or falls down/dies. We report the closest distance from the target that the agent is able reach, for each episode. We select 10 random seeds and test the performance of each method on 1000 episodes for each random seed. In order to ensure that the comparison between all methods is indeed fair, we set the initial configuration of the agent and the target to be the same across all methods at test time.
A.2 ADDITIONAL ANALYSIS OF RESULTS
Why does our method perform better than baselines in spite of using inferior samples? It is important to note that since we take only the first half of the environment interactions for GE and LR. we are learning from essentially inferior samples consisting of actions that may not be optimal, as they have been encountered earlier in the training process. In spite of learning from inferior actions, our method outperforms the baselines. This is because our IDM formulation enables the learning of actions that can reach the next goal given the current state, which is the kind of behaviour we require in goal-directed motion. The focus of this model is more on learning actions that caused certain transitions, not on learning the most optimal actions that achieve the transition, although that would be the next best thing to do. In addition to this, our LFM framework enables the agent to achieve their equivalent transitions by producing the same action for all transitions that share a common latent representation. This enables agent generalize its motion to larger part of the goal space in comparison with baselines that struggle to achieve goal-directed motion for goals or states lying outside the training distribution.
12

---

Under review as a conference paper at ICLR 2020
Environment
of Interactions
Method
Architecture
Training Algorithm
RS
IDM: 256 x 256
Policy: 256 x 256
RL
SAC
IDM: 256 x 256
Humanoid
15M
VGCP
Policy: 256 x 256
SAC
HER- TER-Sparse
Policy: 64 64
SAC
DeRs-
Policy: 64 x 64
SAC
Policy: 256 256
GE
SAC
IDM: 256 x 256
Policy: 256 256
LR
LFM: 256 x 256
SAC
IDM: 256 x 256
RS
IDM: 256 x 256
Policy: 256 x 256
RL
PPO
IDM: 256 x 256
Minitaur
4M
VGCP
Policy: 256 x 256
PPO
HER- Sparse
Policy: 64 64
SAC
HER- Dense
Policy: 64 x 64
SAC
Policy: 256 256
GE
PPO
IDM: 256 x 256
Policy: 256 x 256
LR
LFM: 256 x 256
PPO
IDM: 50 x 50
RS
IDM: 256 x 256
Policy: 64 x 64
RL
PPO
IDM: 256 256
Ant
2M
VGCP
Policy: 64 x 64
PPO
HER Sparse
Policy: 64 64
SAC
TIER-Dense
Policy: 64 x 64
DDPG
Policy: 64 64
GE
PPO
IDM: 256 x 256
Policy: 64 x 64
LR
LFM: 256 256
PPO
IDM: 50 x 50
Table 2: Training details for Humanoid, Minitaur, Ant. Note that training algorithm refers to the RL algorithm used collect data (for our methods), or train the policy (for baselines). Note that the hyperparameters for all experiments (VGCP, HER, PPO, SAC), have been taken from existing implementations of these algorithms (Dhariwal et al. (211 7); Hill et al. (2018).
13

---

Under review as a conference paper at ICLR 2020
Why is our method more sample-efficient than the baselines? Our methods (GE and LR) do not utilize the latter half of samples collected during the training process, which would consist of more optimal actions, because that would imply using higher number of actual environment interactions than the baselines. In spite of this, our method outperforms all baselines for all environments. This is because goal-conditioned policies explore the state and goal spaces to achieve goal-directed motion. While exploration is in fact a desirable component in reinforcement learning, sampling from the goal space during training requires the agent to acquire sense of direction and the skill of locomotion simultaneously. This enlarged state space, comprising the original state and the goal, leads an increase in the number of samples required by the agent to learn good policy. Our latent representations, on the other hand, encode the property of equivalence modulo orientation with respect to actions, described in Section This enables our method to generalize representations of some samples of training experience to other parts of the state and goal spaces, which have not been encountered during training. As a result, the IDM learns to predict actions for states and goals it has never encountered before, if they have the same latent representation as those seen during training, thus boosting performance while remaining sample-efficient
Why does LR show qualitatively better results than VGCP? VGCP is trained a function of the state and goal space, and produces the action that the agent should take in order to reach the goal. The agent is rewarded for moving towards the goal, keeping control and contact costs minimal. There is no constraint on the type of motion exhibited by the agent, or the time taken by the agent to reach the goal. As result, we see in Fig. 7 that the agent trained using VGCP walks very slowly throughout both episodes, and follows non-optimal path to navigate through 3 out of the 4 waypoints. Due to its slow speed, is unable to reach the last waypoint. The LR agent, on the other hand, learns IDM from samples collected while training the locomotion task. Since the locomotion reward encourages walking forward while also keeping control and contact costs minimal, the LR agent always walks forward facing each waypoint, and follows smooth trajectory through all the waypoints, validating the superior performance of our IDM. We see that though the LR and VGCP agents are initialized in the same orientation, the VGCP agent walks with its back facing the waypoints, while the LR agent successfully adopts the optimal path and navigates through all waypoints adopting "natural" walking style.
Why does HER not perform well as other baselines? If we revisit the motivation behind the HER algorithm, we realize that it shows superior performance using fewer training samples over environments that have low-dimensional state and action spaces, and sparse reward setting. In our case, we are dealing with high-dimensional locomotion agents that require specific actions to walk, and navigate to the goal. If we use the sparse reward setting, most samples in the replay buffer consist of failures, as the agent first needs to learn to walk, after which it can successfully navigate to the goal. Even with the dense reward setting, there is significant improvement in performance because of this reason. HER performs very well on low-dimensional sparse-reward environments but it is difficult extend this behaviour to higher dimensions or dense rewards, especially where complex locomotion skills have to be learned in order to navigate to the goal.
How important is the latent representation dimension & in achieving good performance? The dimension of the latent representation k is perhaps the most important hyperparameter that deter- mines the performance of LR. Choosing a bad value of & could result in information loss if & di- mensions not sufficient to encode the relevant information for producing the action, or irrelevant information being encoded if & is too high. In our experiments, we use k = 10 for Ant and Minitaur, and l: = 50 for Humanoid. We show the impact of k on performance in the Ant environment in Fig. 9 We see that performance improves as we keep reducing the dimension up until k = 10. As we try to reduce further, performance suddenly drops, implying loss of information. Thus, the Ant's 111-D state and 3-D goal are reduced to a 10-D representation that is used by the IDM to produce actions to navigate the goal.
14

---

Under review as a conference paper at ICLR 2020
00
60
0.
-18
-12
-25
64
100
15
20
2.5
30
80
(00
Figure Qualitative comparison between different algorithms test time for Ant.
3.0
Distance from target
2.0
1.5
1.0
20
100
Figure 9: Impact of latent representation on performance for the Ant environment
15


Reviews:
[{"review_id": "rkgnfN29YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn locomotion and navigation to a goal location or through a set of waypoints for simulated legged robots. The contributions of this paper include 1) generalized experience, which is a data-augmentation technique to add more orientation-invariant experience, and 2) a latent representation to encode the state, the current location and the goal location. The paper compares the proposed method with a few baselines and demonstrates better performance.\n\nMy recommendation of this paper is Weak Reject. Although the method seems reasonable and the evaluation shows good results, I think that the paper can be improved for the following three reasons. \n\nFirst, it is not clear to me how the inference works if the goal is reasonably far away (e.g. can be reached in 10 steps) from the current position of the robot? Since the inverse dynamics model only outputs an action given the current position and the immediate goal (in the next time-step), how is the 10-step action sequence planned using the 1-step immediate goals? \n\nSecond, the details of data collection are unclear to me. I believe that the policy \\pi used for data collection plays an important role. Which \\pi is used? It would be clearer to present this part in the main text, not Appendix. If the data is collected from an RL agent which learns to walk to the right, how does the robot learns to turn when walking across different waypoints (Figure 7)?  \n\nThird, in this paper, the generalized experience is to add different initial orientations of the robot. I think that the similar effect can be achieved by reparameterizing (s, o, o') into polar coordinates: (s, \\theta, r), where (\\theta, r) is the intermediate goal location relative to the robot's current orientation and position. \\theta=0 means the goal is in front of the robot, and r is the distance between the goal and the robot. In this representation, all the generalized experience will reduce to a single (s, \\theta, r), which is invariant to the robot's orientation. This would already be a good latent space, without any learning. For this reason, I would suggest adding one more baseline to compare the proposed method against: using (s_t, \\theta_t, r_t, a_t, s_{t+1}, \\theta_{t+1}, r_{t+1}) to represent experience, then run the proposed method without generalized experience and latent representation. Will this baseline achieve similar or even better results? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rkehqHWYFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Learning Latent Representations for Inverse Dynamics using Generalized Experiences\n\nIn this paper, the authors propose to utilize the symmetry property in locomotion problems (more specifically navigation problems), and more efficiently generate additional training data from existing data, and learns a more efficient representation.\n\nI tend to vote for rejection for this paper mostly because, while it seems to me to be a very efficient and practical engineering project, but relatively lack the novelty in terms of the algorithm.\n\nPros:\n- The experiments are of good quality, providing a lot of ablation studied and hyper-parameter specifications.\n- The proposed idea is combined with some of the state-of-the-art algorithms, showing it’s compatibility and good practical performance.\n\nCons:\n\n- The proposed algorithm lack novelty.\nGoal conditioned reinforcement learning, where a generalized inverse dynamics is used, has been widely studied in [2, 3].\nAnd the use of symmetry has also been studied [1].\nThe augmentation of data by considering symmetry is relatively straight-forward.\n\n- limited to navigation environments\nThe proposed methods do not seem to be directly applicable to tasks other than navigation, where a very task-specific goal position can be provided.\n\n\n[1] Yu, Wenhao, Greg Turk, and C. Karen Liu. \"Learning symmetric and low-energy locomotion.\" ACM Transactions on Graphics (TOG) 37.4 (2018): 144.\n[2] Ding, Yiming, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. \"Goal-conditioned Imitation Learning.\" arXiv preprint arXiv:1906.05838 (2019).\n[3] Merel, Josh, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. \"Neural probabilistic motor primitives for humanoid control.\" arXiv preprint arXiv:1811.11711 (2018).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1ejU8BQqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method for exploiting structure in locomotive tasks for efficiently learning low-level control policies that pass through waypoints while achieving some goal (typically 3D Cartesian position). This is in contrast to goal-conditioned RL policies that sample random goals during training and are thus sample inefficient, which are trained to execute one policy at a time. In particular, the paper proposes the notion of generalized experiences, where new trajectories are generated from existing trajectories, in such as a way that they are equivalent to each other (in this case translation and orientation invariant) with respect to actions.  \n\nThe idea proposed here, of exploiting environmental structures in order better generalize previous knowledge to new, unseen situations is an interesting direction for achieving sample efficiency in practical RL, such as in robotics. \n\nI have the following comments/questions.\n\n1. If I understand correctly, rather than randomly sampling the environment, the paper proposes starting off with trajectories generated while learning some single-goal policy, and generate from these new ones that the agent must execute in the environment. In which case, the question is how much less interaction does the agent have with the environment compared to random goal sampling, to achieve the same performance?\n\n2. What happens if you use a standard goal-conditioned RL with the Generalized Experiences, without training an IDM? For example, using VGCP with GE, where the goals for VGCP are teminating states of each trajectory. In other words, can a vanilla goal-conditioned RL benefit from the proposed trajectory sampling technique (GE), and how does that compare with random sampling and using the proposed latent representation technique?\n\n3. How do you modify the LFM so that it only accepts the current state and goal, instead of a set of two e.m.o equivalent states and goals? Is the same query treated as two queries that are e.m.o equivalent?\n\n4. In A.1.2, the paper mentions that the first half of the data generated using the RL algorithm is used for generating the second half of data for GE and LR, right? Which means GE and LR are trained from the same number of steps as the baselines, except that GE and LR make use of generalized experiences. However, the paper states in A2 that GE and LR learn from inferior samples. Can the authors please clarify this? My understanding is that GE are generated by modifying existing trajectories and letting the agent apply the same actions by interating with environment. Meaning that, given 2M samples, GE will generate 2M more samples that are e.m.o equivalent by interacting with the environment. \n\n5. Some of the references in the text do not have year of publication, e.g., Kulkarni et al.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Solid, but not novel enough to merit publication.  The reviewers agree on rejection, and despite authors' adaptation, the paper requires more work and broader experimentation for publication.
Decision: Reject