Reinforcement Learning without Ground-Truth State
Authors: Xingyu Lin
Harjatin Singh Baweja
David Held
Venue:  (2020)
Keywords: Self-supervised
goal-conditioned reinforcement learning

Abstract: To perform robot manipulation tasks, a low-dimensional state of the environment typically needs to be estimated. However, designing a state estimator can sometimes be difficult, especially in environments with deformable objects. An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor inputs to actions. However, if this policy is trained with reinforcement learning, then without a state estimator, it is hard to specify a reward function based on high-dimensional observations. To meet this challenge, we propose a simple indicator reward function for goal-conditioned reinforcement learning: we only give a positive reward when the robot's observation exactly matches a target goal observation. We show that by relabeling the original goal with the achieved goal to obtain positive rewards (Andrychowicz et al., 2017), we can learn with the indicator reward function even in continuous state spaces. We propose two methods to further speed up convergence with indicator rewards: reward balancing and reward filtering. We show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards. We show that our method can perform complex tasks in continuous state spaces such as rope manipulation from RGB-D images, without knowledge of the ground-truth state.

Under review as a conference paper at ICLR 2020
REINFORCEMENT LEARNING TRUTH STATE
WITHOUT
GROUND-
Anonymous authors Paper under double-blind review
ABSTRACT
To perform robot manipulation tasks, -wwdimensioonl state of the environment typically needs to estimated. However, designing a state estimator can some- times be difficult, especially in environments with deformable objects. An alter- native is to learn an end-to-end policy that maps directly from high-dimensional sensor inputs to actions. However, if this policy is trained with reinforcement learning, then without a state estimator, it is hard specify a reward function based on high-dimensional observations. To meet this challenge, we propose a simple indicator reward function for goal-conditioned reinforcement learning: we only give a positive reward when the robot's observation exactly matches a target goal observation. We show that by relabeling the original goal with the achieved goal to obtain positive rewards (Andrychowicz et al. 2017). we can learn with the indicator reward function even continuous state spaces, We propose two meth- ods further speed up convergence with indicator rewards: reward balancing and reward filtering. We show comparable performance between our method and an oracle which uses the ground-tre truth state for computing rewards. We show that our method can perform complex tasks in continuous state spaces such as rope manipulation from RGB-D images, without knowledge of the ground-truth state.
INTRODUCTION
To perform robot manipulation tasks, a low dimensional state of the environment typically needs to be estimated. In reinforcement learning, this state is also used to com- pute the reward function. However, designing a state es- timator can be difficult, especially in environments with deformable objects, as shown in Figure [ An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor input to actions. However, with- out a state estimator, it is hard to specify reward function based on high-dimensional observations.
Past efforts to use reinforcement learning for robotics have avoided this issue in number of ways. One com- mon approach is to use extra sensors to determine the state of the environment during training, even if such sen- sors are not available at test time. Examples of this in- clude using another robot arm to hold all relevant ob- jects (Levine et al. 2016). placing an IMU sensor (Gu) et al. 2017; Yahya et al. 2017). motion capture markers on such objects Kormuushe et al. 2010). ensuring that all relevant objects placed on scales (Schenck & Fox 2016).
Figure 1: illustration of the rope push- ing task. The Sawyer robot is given an im- age of the current configuration of the rope and an image of the goal configuration (illus- trated as the translucent rope) and the task is to push the rope the goal configuration.
However, such instrumentation is not always easy to set up for each task. This is especially true for deformable object manipulation, such rope cloth manipulation, in which every part of the object must be instrumented order to measure the full state of the entire object. Attaching such sensors to food or granular material would present additional difficulties.
We present an alternative approach for goal-conditioned reinforcement learning for specifying re- wards using raw (e.g. high-dimensional and continuous) observations without requiring explicit state estimation access the ground-truth state of the environment. We achieve this using

---

Under review as a conference paper at ICLR 2020
simple indicator reward function, which only gives positive reward when the robot's observation exactly matches a target goal observation. Naturally, in continuous state spaces, we do not ex- pect any two observed states to be identical. Surprisingly, we show that we can learn with such an indicator reward, even in continuous state spaces, if we use goal relabeling (Kaelbling [993] Andrychowicz et al.] (017)), which relabels the original goal with the achieved observation such that a positive reward is given. As the indicator rewards produce extreme sparse positive rewards, we further introduce reward balancing to balance the positive and negative rewards, as well as reward filtering to filter out uncertain rewards.
We show theoretically that the indicator reward results in a policy with bounded suboptimality com- pared to the ground-truth reward. We also empirically show comparable performance between our method and an oracle which uses the ground-trun truth state for computing rewards, even though our method only operates on raw observations and does not have access to the ground-truth state. We demonstrate that an indicator reward can be used to teach a robot complex tasks such as rope manip- ulation from RGB-D images, without knowledge of the ground-truth state during training. Videos of our method can be found at athttps:// www.:ttesss.ooooo.oo.e.o gwwgle.ooooooooooooo www.cooocvi........oo wwww//iiww/ww////// image- rl
2
RELATED WORK
2.1 OBTAINING GROUND- STATE FOR TRAINING
Adding sensors: To obtain ground-truth states for calculating rewards, one approach is to perform state estimation. However, such an approach can be noisy and challenging to implement, especially for the deformable objects that we study in this work. Another approach is to add extra sensors during training to accurately record the state. For example, in past work, one robot arm (covered with a cloth at training time) is used to rigidly hold and move an object, while another robot arm learns manipulate the object (Levine et al. 2016). In such case, the object position can be inferred directly from the position of the robot gripper that is holding it. In other work on teaching a robot to open door, IMU sensor is placed on the door handle to determine the rotation angle of
the handle and whether or not the door has been opened (Gu et al. 2017 Yahya et al. (017). One can
also ensure that all relevant objects for task are placed on scales (Schenck & Fox 2016) affixed
with motion capture markers to obtain a precise estimate of their position (Kormushev et al. 2010). However, such instrumentation is challenging for deformable objects, granular material, food, or other settings. Further, such instrumentation is costly and time-consuming to setup; hence most of these previous approaches assume that such instrumentation is only available at training time and these methods do not allow further fine-tuning of the policy after deployment.
Training in simulation: Another common approach is to train the policy entirely in simulation in which ground-trun truth state be obtained from the simulator (Fang et al. 2018 Andrychowicz jet al. 2018b) Zhu et al., 2018 (aadehi & Levine, 2016 Pinto et al. 2018). Many approaches have been explored to try to transfer such policy from simulation to the real world, such as domain randomization (Tobin et al. 2017) or building a more accurate simulator (Tan et al.) [0188 Chebotar et al. 2018). However, obtaining an accurate simulator is often very challenging, especially if the simulator differs from the real-world in unknown ways. Further, building the simulator itself can be fairly complex. Because these methods require the ground-truth state to obtain the reward function, they require training in simulator and do not allow further fine-tuning after deployment in the real world; our method, in contrast, does not require the ground-trui state for the reward function.
2.2 ROBOT LEARNING WITHOUT GROUND-TRUTH STATE
Learning reward function without supervision: One line of work for learning a reward function is to first learn a latent representation and then derive the reward function based on the distance in the embedding space, such as cosine similarity. The representation can be learned by maximizing the mutual information between the achieved goal and the intended goal WWarde-farley et al. 2018), reconstruction of the observation with VAE (Nair et al. 2018). or learning to match keypoints with spatial autoencoders (Finn et al. 20166). Recent work also explicitly learns representation that is suitable for gradient based optimizer and then use it for specifying rewards (Yu et al.) 2019b). However, there no guarantee that these learned representation are suitable for deriving rewards for control. In addition, if the representation is pre-trained, it may also be incorrect in some parts of the observation space which can be exploited by the agent. Our approach is much simpler in that
2

---

Under review as a conference paper at ICLR 2020
the reward function does not have any parameters that need to be learned and empirically show better performance to some reward learning approaches.
Changing the optimization: Another approach is to forego maximizing a sum of rewards as is typically done in reinforcement learning and instead optimize for another objective. For example, one method is to choose one-step greedy actions based on a learned one-step inverse dynamics model; after training, the policy is then applied directly multi-step goal (Agrawal et al. 2016). An alternative method is to learn a predictive forward dynamics model directly in high-dimensional state space and use visual model-predee preddicieeeiiivee control (Finn et al.) 201 6a [Finn & Levine 2017 Ebert ict al., 2017 2018a b). Although these methods have shown some promise, predicting future high- dimensional observations (such images or depth measurements) is challenging. Another approach to obtain expert demonstrations and define an objective as trying to imitate the expert (Sermanet
et al. 2018 2016 Peng et al.) 2018 Finn et al. 2017). Our approach, however, applies even when demonstrations are not available.
2.3 MANIPULATING DEFORMABLE OBJECTS
Deformable object manipulation presents many challenges for both perception and control. One approach to the perception problem is to perform non-rigid registration to a deformable model of the object being manipulated (Huang al. 2015 et al./5/0111 Scchulman et al., 2013) [Wang et al.)
Grafflin & Berenson 2014). However, such an approach often slow, leading to slow policy learning, and can produce errors, leading to poor policy performance. Further, such an approach often requires a 3D deformable model of the object being manipulated, which may be difficult to obtain. Our approach applies directly -ighenimeniioaal observations of the deformable object and does not require prior model of the object being manipulated.
2011 Javdani et al. 2011 Miller et al. 2011: Cusumano- Towner et al. 2011 Philliss-Grafli
3 PROBLEM FORMULATION
In reinforcement learning, an agent interacts with the environment over discrete time steps. In each time step f, the agent observes the current state 8; and takes an action ag In the next time step, the agent transitions to a new state S based on the transition dynamics p($ 18,, 4z) and receives a reward r r(s,, dg, $141). The objective for the agent is learn a policy (a (st) that maximizes the expected future return R E E 7'r where y is discount factor.
3.1 GOAL-REACHING REINFORCEMENT LEARNING
In order for the agent to learn diverse and general skills, we define a goal reaching MDP (Schaul et al., 2015: Andrychowicz et al.) 2017) as follows: In the beginning of each episode, a goal state Sy sampled from goal distribution G. We learn goal conditioned policy x [t, 8g) that tries to reach any goal state from the goal distribution. We use goal conditioned reward function r( 8g) and optimize for Ko Ei= 7-r:]- The transition dynamics p(*c+ (8,, 0e) of the environment remain independent the goal.
In many real-world scenarios, it is often difficult to construct a well-shaped reward function. Past work has shown that sparse rewards, combined with an appropriate learning algorithm, can achieve better performance than poorly-shaped dense rewards in goal-reaching environments (Andrychow- icz al. 2017). We thus define sparse reward function that only makes the binary decision of whether the goal is reached or not. Specifically, let S ) be a subset of the state space such that any state in this set is determined to be sufficiently close to 8, (in some unknown metric); in other words, if the environmental state is within S, (8g). then the task of reaching 8g can be considered be achieved [ Naturally, we can assume that 8y E S+ (sg). binary reward function can then be defined as
R4 St+1 E S4())) R_
T(4441 5n)
(I)
8t41 $ S4g)),
where R4 and R_ are constants representing the rewards received for achieving the goal and failing to achieve the goal, respectively.
S4 is function that maps from the space to subset of the space.
3

---

Under review as a conference paper at ICLR 2020
3.2 REWARDS FROM IMAGES
In many cases, the ground-truth state 8 is unknown and we cannot directly use the true reward function defined in Equation [ Instead, the agent observes nigh-dimensional observations or from sensors, from which we must instead define a proxy reward function 1(0r+1, 0g). The question now becomes how to choose f to be optimal for reinforcement learning?
The most common approach in robotics state estimator might be hard to obtain, such as for deformable object manipulation. c.g. laundry folding food preparation. We therefore investigate whether alternative reward function that does not depend on state estimation can be used. Specifically, let us consider a general reward function defined in observation space of the form
perform state estimation. However, in many cases, the
o++1 (0,) R4
f(oc+1:0g)
R_
o++1 & (0g)),
where Og is a representation of the goal in observation space and 0+ (0,) is a subset of the obser- vation space for which will give positive rewards. A number of past approaches have proposed various methods for learning an observation-bant based reward function f(or+1, 0g) WWade--frreyy et al.) 2018 Nair al. 2018: Florensa et al., 2019; Yu et al., 2019b). However, these approaches do not analyze the properties needed by such a reward function to enable optimal learning. Next we will investigate trade-offs between different choices of Ô+ (og) and how they will affect policy training time when trained with rewards of (g).
4 REWARD MISCLASSIFIICTTION
We will now investigate how to design good proxy reward function F(or+1, based on raw sensor observations, that we can use to train the policy; we desire for the policy trained with
Increasing rate of false positive rewards
Increasing rate of false negatives rewards
1816
P52
IN52
1M13
1N15
F(or+1, 0g) to optimize the orig- inal reward r(s++1, 8g) based the ground-truth state (which we do not have access to). Our first insight into choosing a good proxy reward function f(0 0,)) is that we should think about reward functions in terms of false positives and false negatives. Let us define false positive reward occur when the agent ceives a positive reward based on our proxy reward function f(o++1, 0g) when it would have re- ceived negative reward based on the original reward function r($++1, (g)) sg). In other words, sup- pose that an unknown function f maps from an observation to its corresponding ground-trus trunn state, 8, for). The function f exits as we assume that the environment is fully observable and no two states will have the same observation. Then false positive reward occurs when 0 E (o,) while f(0++1) & (sy). Similarly, a false negative reward can be defined.
INES
As increase false negative/positive rewards, the learning curves with false positive rewards affected more severely.
Figure 2:
Intuitively, both false positive rewards and false negative rewards can negatively impact learning- However, for any estimated reward function f(or41; 0y), we will have either false positives or false negatives (or both) unless we have access to a perfect state estimator. To design good proxy reward function, we must ask: which will more negatively affect learning: false positives or false negatives?
The two types of mistakes are not symmetric. As we will see, a false positive reward can signif- icantly hurt policy learning, while false negative reward is much more tolerable. Under false positive reward, the agent receives a positive reward (under the proxy reward function f(o, 0g)) for reaching some observation o, even though the agent should receive negative reward based on the corresponding ground-truth state (o) under the original reward function r r(st, 8y). This false posi- tive reward will encourage the agent to continue to try to reach the state fo)) even though reaching this state does not achieve the original task since f(o & S+g)).
On the other hand, false negative rewards are much more tolerable. Under a false negative, the agent observes some observation such that f(o) E (xg)) but the agent receives negative reward.
4

---

Under review as a conference paper at ICLR 2020
However, if the agent still receives a positive reward for some other observation o' such that f(d') S+ (*,),, then the agent can still learn to reach the goal states S (8g), though learning might be slower and the learned policy may be suboptimal.
We provide simple example verify this intuition. Consider a robot arm reaching task, with the observation space O E R2 being the 3D position of the end-effector (EE). Note that here the observation space coincides with the state space and the agent directly observes the states. The action space A E R controls the position of EE. The true reward is defined by S+ (8y) = {o ||st $gll2 c)- We define types of noisy reward functions used for training. The reward function FFP gives the same rewards as the true reward function, except that with a probability of pFp (False Positive Rate), negative reward will be flipped a positive reward. The reward function fFN can be similarly defined, where a positive reward will be flipped to a negative reward with a probability of PFN (False Negative Rate). For this experiment, we use standard reinforcement learning algorithm DDPG (Lillicrap al., 2016) combined with goal relabeling (Andrychowicz et al. 2017). . The learning performance of this same algorithm with different noisy rewards can be shown in Figure 2 We can see that the agent is able learn the task even with very large false negative rate. But when the false positive rate increases, the performance sharply decreases.
5 APPROACH
5.1 INDICATOR REWARDS
Following this idea, we propose using a proxy reward function that does not have any false positive rewards. To do so, we will use an extreme reward function of (_g) {og). In other words, we will use an indicator reward function:
R fmd 0g) = R
{
01+1 = 0g Oc+1 # Og:
(3)
It should be clear that this reward function will have no false positives, since the reward is positive only if 0t+1 og. which implies that f(or+1) = f(o,). or equivalently, St+1 = Sg. As Sg E S (sg) by definition, all positive rewards are true positives. However, this reward function is extremely sparse and has many false negatives. In fact, without goal relabeling, in continuous state spaces, would expect all rewards to be negative under this indicator reward function, as no two observations in continuous spaces will ever be identical. Next, we will describe how to learn with this reward function with goal relabeling.
5.2 GOAL RELABELING FOR OFF-POLICY LEARNING
Fortunately, for off-policy multi-goal learning, we can adopt the goal relabeling technique intro- duced in (Kaelbling 1993] Andrychowicz et al. 2017) to learn the goal-conditioned Q-function. Suppose that some transitions (Ot, de, 0c+1) are observed when the agent takes action a; T 0y) with a goal of 0y. Because Q-learning is an off-policy reinforcement learning algorithm, can replace the goal observation Og with any other observation og' in our Bellman update of the Q-function. This works because the transition dynamics p(*++1|8 a and likewise the observation transition dynamics p(0++1 lor, dt), are independent of the goal 0g- Specifically, for some transi- tions, we will choose replace 0g with the observation 0,41- By re-labeling 0g with o141 and using our indicator reward function, we will have that Fina(Or-1) 0g) fina(Dr-1: o++1) R.,. Thus, using goal relabeling, we can get positive rewards, even when using an indicator reward function in continuous state spaces.
5.3 REWARD BALANCING AND FILTERING
As mentioned above, after sampling a batch data, we train the Q-function with goal relabeling. We use three different strategies for choosing which goals to use for relabeling: with probability pl, we relabel 0g with o,-1, which will receive a positive reward under our indicator reward function. With probability P2, we relabel the goal 0g with oy with an observation from some future time t' step within the episode. The indicator reward function will most likely give a negative reward in this case, which is possibly a false negative because the new goal is possibly considered achieved based on the state-based, ground-truth reward funciton. Finally, with probability p3, the original
5

---

Under review as a conference paper at ICLR 2020
goal (with relabeling), which will again most likely give negative reward under the indicator reward function; as before, this might be a false negative.
Reward balancing: We refer to "reward balancing" as setting Pi = = 0.45 and P3 = 0.1, leading us to receive positive rewards approximately 0.45 of the time and negative rewards approx- imately 0.55 of the time. Thus the ratio of positive and negative rewards that we use to train the Q-function approximately balanced, even with indicator rewards. From another perspective, pr and p2 determine the relative frequency between providing positive rewards and propagating rewards to other timesteps in the episode. Additionally, training with small fraction of the original goals (i.e. pa) can be seen as a regularization which ensures that the distribution of the relabeled goals moves towards the original goal distribution.
Reward filtering: While false negative rewards do not hurt learning as much as false positives, we still wish to avoid them if possible to improve the convergence time of the learned policy. We achieve this using "reward filtering," in which we filter out transitions that we suspect of having high chance of being false negatives. We refer to "reward filtering" as discarding a sampled transition if its Q value is above threshold go- If the assigned reward is negative based the proxy reward function but the Q-value is sufficiently high, then there is a chance that this reward a false negative. To reduce the fraction of false negatives, we filter out such transitions and do not use them for training.
We can estimate how to set the threshold qo as follows: for a given transition (or, at, 0t+1), 01+1 Og. we know that (ind(0 0g) ) = R4. In this case, Q" (o,, dt, 0g)) = R1 - 7), where Q" is the optimal Q-function, assuming the optimal policy will continue receive (discounted) positive rewards in the future. Similarly, if 0t+1 # Og. then
og) R_ Since we know that the policy starting from o, will thus receive at least one negative reward before receiving positive rewards, then Q" at, 0y) S R_ + y TR+/ /11 - 7). Thus, we can set a threshold qu- where R_ + /R+(11 - 7) < 90
Y); if corresponding reward Find find (0441, ) is likely to be a false negative (assuming that the Q-function has been trained well): we thus filter out such rewards, to reduce the number of false negatives that we use for training. We can see that go is set a rather conservative fitlering value. Additionally, the Q-function is initialized to a relatively low value to avoid overestimation of the Q-function which can lead to incorrect filtering in the beginning of the training.
R+/(1
find that Q(o,, Og. aq) > qo. then the
6 ANALYSIS
In this section, we analyze the performance of learning with indicator rewards. We first interpret the goal conditioned Q-function as a measure of the time to reach one observation from another.
6.1 MINIMUM REACHING TIME INTERPRETATION
Let us define = D (ot, a, (og)) as the number of time steps it takes for the policy π to go from the current observation og. starting with action a,, to reach the set (og) of goal observations. For simplicity, we assume that, once the agent receives a positive reward, it will take actions to continue to receive positive rewards. The Q-function can be written as
(x at, 0g) R_ + + + d-1 R Td
RR + + R4
1 R.
(4)
(R+ RR)
Now it can be easily seen that, as long R4 R_, Qx is strictly monotonically decreasing w.r.. d. As such, maximizing Q= over æ is equivalent to minimizing the time the agent takes to reach the goal Ō+(o)) Note that this is true for varying definitions of Ó4(0)): thus the policy trained under the true reward function (Equation [ will minimize D₂ (o az, (0,))) whereas the policy trained under the indicator reward function will minimize D, (or, aes {o,}) (slightly overloading notation for Dz). Below we will show how this interpretation of the policy's 's behavior at convergence can lead to a simple analysis of the suboptimality of the learned policy under the indicator reward.
6.2 ANALYSIS OF SUR-OPTIMALITY
Due to the false negative rewards given by the indicator function Finds the learned policy may not be optimal with respect to the original reward function r(8e+1, 8y) defined in Equation [ Here

---

Under review as a conference paper at ICLR 2020
give the worst case bound for the policy learned with the indicator reward. Following the minimum reaching time interpretation of the previous section, we evaluate the performance
the policy in terms of the time it takes to reach the set of goal observations O+ (0g) from the current observation. Given 0r, Og. denote i as the minimum number of time steps to reach from or to the set of true goal observations, i.c. & D(o, O+(%y)). Let 22 (oz, 0g) be the minimum time to reach from or to Og. Define the diameter of this goal observation set as (ax(D(o1, 02) 02 O_ }.
From the optimality of (2, we know that
b S D(or,)) + ((), Vo
Thus,
12 S min D(or, + D(o, 0g) S + (E0+(og)
(5) From the analysis in the previous section, the optimal policy which optimizes the indicator reward will reaches 0g in l2 time steps; since 0g E O+ (0g),
know that this policy will reach O+ (o,) in some time 13 S fz.. Also recall that we have defined t such that the optimal policy under the true reward function of Equation will reach O+ (0)) in 1 steps. Thus fz/t1 < (t1 + /t is upper bound on the suboptimality of the policy trained under the indicator reward, at convergence.
EXPERIMENTS
Our experiments address the following questions:
1. In the case of visual input, how much are the sample efficiency and the final performance affected without assuming access to the ground-truth reward?
2. How much does reward balancing and filtering improve learning efficiency?
3. Can our method scale to real world robotic tasks?
We denote our method, which uses indicator rewards with reward balancing and filtering, as Indi- aator+Balance+Fitter We compare our method with the following baselines:
Oracle: This method assumes access to the ground-truth reward from state space r(st, sg).
Indicator: This is an ablation of our method, without reward balancing and filtering.
Auto Encoder (AE): We train an autoencoder with an L2 reconstruction loss of the im- age observation, jointly with the RL agent. We then use cosine similarity in the learned embedding space provide dense rewards, as similarly compared in (Warde-farley
al., 2018). Specifically, assuming the learned encoding of an observation o is Φ(o) after T.2 normalization, the reward will be r(o, = (0, (o)) o(o,)).
Variational Auto Encoder (VAE): Similarly to the AE baseline, VAE is jointly trained with the RL agent to provide rewards, as done in (Nair et al. 2018). For a fair comparison, the goal sampling strategy for this baseline is kept the same other approaches.
Distributional Planning Network (DPN) YYu et al., 2019b): DPN aims to learn a repre- sentaiton that is suitable for gradient based planner to reach a goal observation. Following (Yu et al. 2019b). we first pre-train DPN using randomly generated samples and then use the learned representation for giving rewards.
Only Oracle uses the ground-truth, state-based reward function. We use the standard off off-policy learning algorithm DDPG (Lillicrap et al., 2016) with goal relabeling (Andrychowiez et al., (017). For methods without reward balancing, we re-label the current goal with an achieved goal sampled
uniformly from one of the future time steps within the same episode with a probability of 0.9; otherwise, the original goals are used. For all the environments, the ground-truth rewards are based
on the La distance in the state space: R, if || syl < and f. otherwise. More details on algorithms, architectures, and hyperparameters can be found in the Appendix.
We first evaluate all the methods in set of simulated environments in MuJoCo (Todorov et al. 2012), where both current and goal observation are given by RGB-D images:
Reacher: Teach two-link arm to reach randomly located position in 2D space.

---

Under review as a conference paper at ICLR 2020
FetchReach: Move the end effector of the Fetch robot random position in 3D.
RopePush: Push rope from random initial configuration into target configuration.
The first two environments above are standard environments from Gym (Andrychowicz et al, 2018a). For the more complex RopePush task, the robot needs to push T5-link rope to a tar- geted pose, as shown in Figure [ To accelearate learning, we fix the orientation of the gripper and parameterize the action as (I1, 91, 22, V2) E R²² denoting the starting and ending position of one push from the gripper. We generate the initill rope pose by giving the rope random push from fixed location. The goal poses are generated by giving the rope two more pushes based on the initial push (these pushes hidden from the policy). The robot can give three pushes push the rope the goal pose. More details on environments can be found in Appendix A.
The results are shown in Figure 3 We that our method (ndicator-ballnce-Fiile achieves nearly the same performance as the Oracle even though our method operates only from RGB-D images and does not have access to the ground-trute state. For the RopePush environment, only the Oracle and our method are able to learn to achieve the task to a reasonable accuracy. For AE, VAE and DPN, the learned representation may not lead to a perfect reward function everywhere and the agent will exploit states that yield high proxy rewards, even though the goal is not achieved. Instead, our method does not require learning a reward function and outperforms these baselines.
VAE
Oracle
Indicator+ Balance+ Filter
Indicator
DPN
AE
Visual Reacher
V sual FetchReeahh
RogePush
Final Final Distance to Goal
rinal rinal Distance to Goel
Distaicee Goal
1.00
0.20
0.20
0.80
0.15
0.60
0.10
0.10
0.40
0.05
0.20
0K
50K
100K
150K
200K
0K
50K
100K
150K
200K
25K
50K
75K
100K
Timestaps
Timestaps
Timestops
Figure 3: The final distance to goal of different methods in different cavironments throughout the training. The observations are from the RGB-D images rendered in simulation.
In Appendix we perform an ablation where we removed either Balance or Filter from our method. These experiments show that both Balance and Filter are important for optimal performance across the environments tested. In Appendix D, we show experiments in which the ground-truth states are used as inputs to the policy but are not used to compute the rewards. these experiments, we also that our method has similar performance to the Oracle and outperforms the other approaches.
7.1 INDICATOR REWARDS WITH REAL IMAGES
Using RGB-D observations and goals, we train Sawyer robot for 3-dimensional reaching task. Figure I shows example observation and goal images well as the distance to goal throughout training. As before, our method (In- performs similarly to the Oracle terms of final goal distance; the baseline of Indicator Rewards without balanc- ing filtering performs significantly worse and diverges in the end due to many negative re- wards, many of which are false negative.
Final Distance to Goal
0.4
Visual Sawyer Reacher
Oracle
0.2
0.9
5K
10K
15k
20K
Timesteps
8 CONCLUSION
Figure 4: An example observation image (top left) and goal image (top right); final distance goal (bottom).
In this work, we show that we can train a robot to perform complex manipulation tasks directly from high- dine-siimnnional images, without requiring access to the ground-truth state in either the policy input or the reward function. We empirically show that our method enables a robot to learn complex skills for manipulating deformable objects, for which state estimation is often challenging, including real-world experiment.
8

---

Under review as a conference paper at ICLR 2020
We provide theoretical analysis which shows that the optimal policy under the indicator reward has bounded sub-optimality compared to the optimal policy under the ground-truth reward. We hope that our method will enable robot learning in the real world in cases where it is difficult to add extra sensors or accurately simulate the environment.
REFERENCES
Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physies. In NIPS, pp. 5074-5082, 2016.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In NIPS, pp. 5048-5058, 2017.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177
2018a.
Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. arXiv preprint
2018b.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv:1810.05667, 2018.
Marco Cusumano-Towner Arjun Singh, Stephen Miller, James O'Brien, and Pieter Abbeel. Bring- ing clothing into desired configurations with limited perception. In ICRA, pp. 3893-3900, IEEE, 2011.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. arXiv preprint arXiv: 2017.
Frederik Ebert, Sudeep Dasari, Alex Lee, Sergey Levine, and Chelsea Finn. Robustness via retrying: Closed-loop robotic manipulation with self-supervised learning. arXiv preprint arXiv: f0I.000000333 2018a.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual fore- sight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint 2018b
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kurenkov, Viraj Mehta, Li Fei-Fei, and Silvio Savarese. Learning task-oriented grasping for tool manipulation from simulated self-supervision. RSS, 2018.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pp. 2786-2793. IEEE, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems, pp. 64-72, 2016a.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. ICRA, pp. 512-519. IEEE, 2016b.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learnig, arXiv preprint arXiv: 170.009909 2017.
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self-supervised learning of image embedding for continuous control. Workshop at NIPS 2018, 2019.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In ICRA, pp. 3389-3396. IEEE, 2017.

---

Under review as a conference paper at ICLR 2020
Sandy H. Huang, Jia Pan, George Mulcaire, and Pieter Abbeel. Leveraging appearance priors in non- rigid registration, with application to manipulation of deformable objects. In IROS, pp. 878-885. IEEE, 2015.
Shervin Javdani, Sameep Tandon, Jie Tang, James F O'Brien, and Pieter Abbeel. Modeling and perception deformable one-dimensional objects. ICRA, pp. 1607-1614, IEEE, 2011.
Leslie Pack Kaelbling. Learning to achieve goals. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, Chambery, France, 1993. Morgan Kaufmann.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2014.
Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Robot motor skill coordination with em-based reinforcement learning. In IROS, pp. 3232-3237. IEEE, 2010.
Alex X Lee, Max Goldstein, Shane Barratt, and Pieter Abbeel. A non-rigid point and normal registration algorithm with applications to learning from demonstrations. In ICRA, pp. 935-942, 2015.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-337, 2016.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Stephen Miller, Mario Fritz, Trevor Darrell, and Pieter Abbeel. Parametrized shape models for clothing. ICRA, pp. 4861-4868. IEEE, 2011.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Xue Bin Peng. Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. arXiv preprint arXiv:1810.0083 2018.
Calder Phillip--rraffin and Dmitry Berenson. A representation deformable objects for motion planning with no physical simulation. In ICRA 2014, pp. 98-105. IEEE, 2014.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. 2018.
Fereshteh Sadeghi and Sergey Levine, Cad2rl: Real single-image flight without a single real image. arXiv preprint orXiv:1611.04220 1.04201, 2016.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320, 2015.
Connor Schenck and Dieter Fox. Guided policy search with delayed sensor measurements, arXiv preprint arXiv: 03076, 2016.
John Schulman, Alex Lee, Jonathan Ho, and Pieter Abbeel. Tracking deformable objects with point clouds. In ICRA, pp. 1 1130-1137. IEEE, 2013.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation 2016.
learning. arXiv preprint
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. T'mm--ootrassive networks: Self-supervised learning from video. In ICRA, pp. 1134-114 1141, IEEE, 2018.
Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. RSS, 2018
10

---

Under review as a conference paper at ICLR 2020
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization transferring deep neural networks from simulation to the real world. In IROS, pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS 2012, pp. 5026-5033. IEEE, 2012.
Ping Chuan Wang, Stephen Miller, Mario Fritz, Trevor Darrell, and Pieter Abbeel. Perception for the manipulation of socks. In IROS 2011, pp. 4877-4884, IEEE, 2011.
David Warde-farley, Tom Van De Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised Control through Non-parameterie Discriminative Rewards.
ICLR, pp. 1-17,201 2018.
Ali Yahya, Adrian Li, Mrinal Kalakrishnan, Yevgen Chebotar, and Sergey Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In IROS, pp. 79-86. IEEE, 2017.
Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control through distributional planning networks. https:://iithb..oo wtwp:ttabbcom/ttaannyy9991111 2019a.
Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, and Chelsea Finn. Unsupervised visuomotor control
through distributional planning networks. Robotics: Science and Systems, 2019b.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint
2018.

---

Under review as a conference paper at ICLR 2020
A ENVIRONMENT DETAILS
During evaluation, for all environments, binary sparse reward is given each time step. A positive reward R. 1 is given when the goal is reached, i.c. || - syyl S e and a negative reward R_ = -1 is given otherwise. Other environment details are summarized in Table
For the RopePush environment, to the time for computing initial and goal configuration, 10,000 initial configurations of the rope are pre-computed and cached which are later used for training.
Environment
Observation Dimension
Goal Rendered Dimension
Action
Horizon
(m)
Dimension
Dimension
(T)
Reacher
10
100x100x3
50
0.01
FetchReach
10
100x100x4
50
0.05
FetchPush
25
50
0.05
RopePush
45
30
100x100x3
3
0.1
VisualReacher
100x100x4
25
0.1
(Sawyer)
Table I: Summarized environment details. The observation and goal dimension the dimensions of the low dimension state representation when available. The rendered dimensions the dimensions of the rendered RGBD images used the visual experiments.
Sawyer robot experiment details: The observation is recorded with Intel RealSense D435 depth camera. The goal observations are sampled by moving the robot arm to a uniformly sampled location in a cuboid of diagonal length 1.3m. The episode was considered successful if the end effector moved within 0.1 m from goal location the end of the episode (we used time horizon of 25 steps). The trained policy performs position control and outputs end effector displacement within range of -0.05m to 0.05m in each direction.
B HYPER-PARAMETERS
All the experiments are run for two random seeds. The hyper-paranen parameters of the training algorithm with indicator rewards summarized in Table For all experiments with visual observation, the parameters of the convolution layers are shared among the observation input and goal input. Due to the complexity of the RopePush environment, spatial softmax layer (Levine al. 2016) with an output size of 32 is applied before the fully connected layers.
Parameter
Value
shared
positive reward (R4)
negative reward (R_)
-1
reward filtering (go)
R_ RR 7) R4 (1 - 7)
optimizer
Adam Kingma & Ba 2014)
learning rate
0.001
discount (y)
target network smoothing (r) nonlinearity
0.98
tanh
state observation
replay buffer size
106
minibatch size network architecture
256
3 hidden layers with 256 neurons for each
visual observation
replay buffer size minibatch size
5 10
128
network architecture
convolution layers followed by hidden layers
with 256 neurons for each
Table Summarized hyper parametrrs..ss....
12

---

Under review as a conference paper at ICLR 2020
C
ABLATION ANALYSIS
We show different ablations of our methods in Figure 5 environment, filtering is required for the policy to leam. On the other hand, the Reacher and FetchReach environments show that balancing is required for optimal performance. In all cases, Indicator+ Balance++ilttte consistently performs better than all the ablated methods. Thus, these sults show that both balancing and filtering are important for optimal performance across range of
We can see that, in the RopePush
tasks.
Visual Reacher
Visual FetchReach
Visual RopePush
0.3
0.8
Surcess
0.6
Success
0.2
0.6
0.4
0.1
0.2
0.0
0.0
0K
100K
200K
0K
100K
200K
25K
50K
75K Timesteps
100K
Timesteps
Timesteps
FetchReach
Visuel RopePush
Final Distance Goal
Final Distance to Goal
0.20
0.15
indiator-eattenc -ileene-----
Final Distance
0.15
idicator-Balrnee Indicator
0.6
0.10
0.10
0.4
0.05
0.05
0.2
100K Timesteps
200K
100K
50K
100K
Timesteps
Timesteps
Figure 5: The success (top) and the final distance to goal (bottom) of different ablations of our method.
With indicator rewards, do not have any false positive rewards but may have many false negative rewards. In Figure 6 we show the accuracy of the given rewards using different approaches with indicator rewards. The ground-truth rewards are calculated based on the ground-truth states which we do not have during training and are only used for analysis. We can see that with a longer time horizon, reward balancing is more important for the Visual Reacher and Visual FetchReach environment, which significantly lower the false negative rates. On the other hand, both reward filtering and balancing are important in the Visual RopePush environment. In all the cases, the accuracy of the rewards
improved lot and sometimes almost perfect with reward balancing and filtering and we can see that this is necessary for the good performance of using indicator rewards. Visual RopePush
Visual Reacher
Visual FetchReach
False Negelive Rate
Fabs Negative Rate
0.800
0.100
False Negative
0.300
Ind cetor Filee rntrattre Tollnoo- ther color Sulance
0.075
0.200
0.600
0.100
0.400
0.000
0K
100K
200K
100K
200K
25K
50C
75K
100K
Timesseps
Timesteps
Timesteps
Visual RopePush
0.800
1.000
Accuracy
0.980
Roward Accuracy
Reward Accuracy
0.980
0.600
0.960
0.960
0.400
0.940
0.920
0K
100K Timesteps
200K
100K Timesteps
200K
25K
50K
75K
100K
Timesteps
Figure 6: The false negative rate and the reward accuracy calculated from batch sampled from the replay buffer of different ablations of our method.
13

---

Under review as a conference paper at ICLR 2020
D LEARNING WITH INDICATOR REWARDS WITH STATE INPUT
The performances of different learning methods when learning with low dimensional state represen- tation are shown Figure In all the environments, using indicator rewards with reward balancing and filtering achieves comparable performances to Oracle. Compare this method to the Indicator baseline which does not have reward balancing and filtering, Indicator+Ballnee+Fiii better sample efficiency. Interestingly, in the FetchReach environment, the default distance thresh- old for receiving an R4 reward is set to 0.05. Thus, the policy that learns with this reward stops learning when the policy reach to such a distance to the goal, while the policy learned with indicator rewards keep reaching closer to the goals. This shows another benefit of using the indicator reward that the user does not need to tune the hyper-parameter to achieve the best performance.
achieves
Searher
FebrhBeach
SopePush
1.0
10
0.8
Seccess
0.6
0.6
0.44
Eola Not Pike
0.
SdK
100K
150K
OK
SGK
100K TiTrosApps
150K
200C
OK
2002K
2000K
3000K
DK
200K
4OCK GODI BOOK Timestopo
Timestops
RopePush
Reacher
FetchReach
PetchPuch
Desternee Goal
0.06
0:25
0.04
DL30
0.1
0.00
0.05
0.00
0.00
0.0
2000K
3000K
OK
200C
400K
50DIC
BOOK
Timsstpps
Timestaps
Timestoss
Timesteps
Figure 7: The success (upper row) and the final distance to goal (lower row) of different methods in different environments throughout the training. The success is defined as the mean probabily of getting the R4 reward over all time steps. The final distance to goal is defined as the I.2 distance the goal in the state space, in the last time step of the episode. The input the policy is the low dimension state representation.
E IMPLEMENTATION DETAILS OF BASELINES
Auto Encoder (AE): The network architecture for the encoder is four convolutional layers each followed by max- max-poolingg layer. The latent code a dimension of 32. The decoder is four up-pooling and up-convolution layers. The auto encoder is trained jointly with the RL. agent by the reconstruction loss of the observation images in the sampled transition. The learning rate is set to 0.001. We then use cosine similarity in the learned embedding space to provide dense rewards, as similarly compared in (Warde-farley et al.) 2018). Specifically, assuming the learned encoding of an observation o is Φ(o) after L2 normalization, the reward will be r(o, 0y) mar(0, o(o)² ((og)).
Variational Auto Encoder (VAE) The VAE uses the same architecture and training pro- cedure as the AE baseline. Following (Nar et al., 2018), we use a variant of the VAE, 3- 3-VAE with 9 5. Following (Nair al. 2018), given the latent representation 2, 2g of the current observation and the goal observation Og, the rewards are given by
r(o,0)) = -1|2 2g 2gll2,
which is the negative of the Euclidean distance.
Distributional Planning Network (DPN) (Yu al. 2019b) We use the released imple- mentation (Yu et al. 2019a) and use the default hyperparameters. The network architecture for encoder is three convolutional layers, each with kernel size 5, stride 2. After each convolutional layer, there is layer normalization and sigmoid non-linearity. The output rep- resentation from CNN goes into fully connected layer with latent code dimension of 128. For each environment, we collected 20,000 transitions from a random policy and performed 100,000 minibatch updates. The input dimension for each of the three environments is the same as the Rendered Dimension column specified Table Once the DPN is pre-traie are-trained, the representation is fixed during the training of the RI. agent. Following YYu et al. (0196)) given the latent representation 2, 2g of the current observation and the goal observation, the
14

---

Under review as a conference paper at ICLR 2020
rewards are given by
r(o, 0g) = exp(((ddeen 2g. 8)111),
where
(2r, 8); - +
for |ay| S 6
otherwise
6 is set to 0.85. To avoid overflow of the rewards during the exponential, we normalize and 2g such that they have norm of after the representation is trained.
F SENSITIVITY ON FILTERING THRESHOLD
In Section 5.3, give the range of the filtering threshold fo- In previous experiments, the middle value of this range. Here, we train with different values go within this range. The results for the rope pushing task is shown in Figure 8 We can see that the learning is very stable with different values of 40-
Visual RopePush
1.0
60=1.0
o2=1.4
0.0
6C-1.8
to
0.6-
QP-2.
QC-3.0
0.4
0.2
0.0 20C
60K
120K
Figure 8: Learning with indicator rewards on the rope push environment using different filtering threshold. The range given in Section 5.3 is [1, 3]. Here we show values of evenly spaced within this range.
15


Reviews:
[{"review_id": "rylZXUeCKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary ###\n\nIn this paper, the authors focus on the problem of goal conditioned reinforcement learning. Specifically, the authors consider the setting where the agent only observes vision as input and the ground truth state is not observable by the agent. In this setting, it is hard to specify a reward function since the reward function has to compute rewards from images.\n\nThe authors first consider the setting of using a proxy reward function, and argue that false positive errors in the proxy reward function hurt the policy training significantly more than false negative errors. The authors demonstrate this empirically in a simple simulated robot arm reaching setting. Then the authors propose to use an indicator reward function that eliminate all false positive errors. The authors combine the indicator reward function with a mixture of goal relabeling schemes and a heuristic way of filtering out data with false negative rewards.\n\nThe authors evaluated the proposed method on 3 simulated robotic manipulation environments and one real robot environment. The results presented by the authors suggest that the proposed method performs better than baseline methods.\n\n\n### Review ###\n\nOverall I think this paper presents an interesting idea in learning goal conditioned policies from vision. The idea is very well presented and authors include many empirical evidence to support the proposed method. However I do find a number of shortcomings that need to be addressed.\n\n\nPro:\n1. The idea for this paper is really well presented. The structure of the paper is well organized and  the authors include informative explanations and empirical evidence to support the crucial assumption that false positive rewards are worse than false negative rewards. The results for the main experiments are also easy to understand.\n\n2. The paper includes a fairly comprehensive set of ablation studies for each part of the proposed method in the appendix. The ablation study clearly illustrated the effects of balancing different goal relabeling schemes and filtering transitions.\n\n\nCon:\n\n1. I’m not convinced about the magnitude of novelty in this paper. The indicator reward has already been used in HER[1], and the balancing of relabeling schemes seems like a direct extension of the various relabeling schemes proposed in HER. It seems to me that the only novelty of this paper comes from the filtering techniques for false negative rewards, which I do not think is enough for this venue.\n\n2. The experiment results are not very strong for the proposed method. In two of the three simulated robotics environments, the proposed method performs similarly to the indicator + balance configuration, which in my opinion is only a slight variation of HER. Therefore, I do not find the claimed advantage of the proposed method to be convincing.\n\n\nThe idea in the paper is well presented and carefully investigated. However, I am still not convinced about the novelty of the proposed idea and the magnitude of performance improvement. Therefore, I would not recommend acceptance before these problems are addressed. \n\n\n\nReferences\n[1] Andrychowicz, Marcin, et al. \"Hindsight experience replay.\" Advances in Neural Information Processing Systems. 2017.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklB4gqjKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to apply HER to image-based domain, assigning rewards based only on exact equality (and not using an epsilon-ball). The authors also propose to (1) filter transitions that are likely to cause false negative rewards and (2) balance the goal relabeling so that the number of positive and negative rewards are equal. The authors demonstrate that these two additions result in faster and better learning on a simulated 2d and 3d reaching, as well as a rope task. The authors also show that the method works on training a real-world robot to reach different positions from images.\n\nOverall, the paper is a fairly straightforward and simple extension of HER, and the proposed changes seem to result in consistent improvements. Studying the importance of false positive vs false negative rewards is an interesting problem, but some of the details of the analysis are missing. Most of the writing of the paper is relatively clear, but there are a couple of assumptions that should be discussed more clearly. The analysis section seemed particularly confusing, as it introduces new notation and concepts (D without a subscript, goal observation set diameter) without explaining their significance. The experiments are relatively simple, making it difficult to judge whether or not the method will work on more challenging domains.\n\nIn more details:\n1. The related works section feels unnecessarily long. \n\n2. The analysis section is quite confusing to me and I do not understand its significance. It seems to say that reaching og rather than O+(og) is not very bad, if O+(og) is small. While this makes sense, it's not clear how the derivations of the paper add to this intuition. In particular, the paper derives an upper bound for t3/t1, which equals\n\n time to reach og / time to reach O+(Og)\n\nWhy is this \"an upper bound on the suboptimality of the policy\"?\n\n3. Can details of Figure 2 be given? This seems like a very important experiment, so it is a shame that it is not discussed or analyzed in detail.\n\n4. An assumption that is critical to this method's applicability is the assumption that no two states will receive the same observation. Without this assumption, the statement, \"the reward is positive only if ot+1 = og, which implies that f(ot+1) = f(og), or equivalently, st+1 = sg\" is false, and so therefore the statement, \"It should be clear that this reward function will have no false positives\" is also false. I do think that this is a reasonable assumption to make for some domains, but the authors should make that explicit.\n\n5. For the \"Indicator\" baseline, what is the probability used for relabeling with a future state? I assume p1=0, but what are p2 and p3?\n\n6. The ablations (Section C, Figure 5) indicates that only the filtering matters. I don't think this detracts from the author's submission, but it would be good for the authors to highlight the main paper. Currently, the statements, \"These experiments show that **both** Balance and Filter are important for optimal performance across the environments tested.\" seems unsubstantiated by the experiments--only filtering seems important.\n\n7. The authors should clarify the difference between the \"true reward\" and the \"original reward.\" What is the relationship? The last paragraph of Section 4 discusses false positive rates and says \"true reward is defined by O+(og) = {o | ||o − og||2 < epsilon}.\" However, two paragraphs earlier, the paragraph also discuses false positive rewards and says, \"under the original reward function r(st, sg).\" The experiment section then says \"Oracle uses the ground-truth, state-based reward function.\" Is the assumption that the \"true\" reward (defined in image space) is a perfect representation for the \"original\" reward (defined in state space)? Does this method work if this assumption is violated?\n\n8. (very important) How sensitive is the method to q0, and how was it chosen for the experiments?\n\n9. Figure 6 in the Appendix is very difficult to read.\n\nI would be inclined to raise my score if the authors\na. clarified some questions above\nb. conducted ablations to study the importance of q0, the threshold\nc. added experiments on more challenging domains (e.g. cube pushing, pickup) that demonstrated the effectiveness of the method on more domains", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BygDCwcstH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tackles the problem of self-supervised reinforcement learning through the lens of goal-conditioned RL, which is in line with recent work (Nair et al, Wade-Farley et al, Florensa et al, Yu et al.). The proposed approach is a simple one - it uses the relabeling trick from (Kaelbling, 1993; Andrychowicz et al., 2017) to assign binary rewards to the collected trajectories. They apply two simple tricks on top of relabeling:\n\n1. Reward balancing: Balancing the number of 0/1 rewards used for training the policy.\n2. Reward filtering: A heuristic that rejects certain negative-reward transitions for learning if the q value for the transition is greater than some threshold q_0.\n\nWhile I like the simplicity of the proposed approach when compared to competing methods, my overall recommendation is reject based on the current state of the paper, because of the following reasons:\n\n1. The technical novelty is quite limited - the paper mostly uses the framework from Andrychowicz et al., 2017 with a specific choice of epsilon (=0) for giving positive rewards. The method does reward balancing, but similar goal-sampling have been used in prior work like Nair et al., 2018, and is not essential to obtaining good results (Appendix C). The main technically novel component is likely the reward filtering mechanism, but I find it to be somewhat ad-hoc since it assumes that the Q-values learned by the Q-network to be reasonably good during training time, which is not the case for most modern Q-learning based methods [1, 2]. \n2. The provided analysis is not particularly illuminating, see my detailed notes below.\n3. The experiments are underwhelming, see my detailed notes below. \n\nI would be willing to overlook 1 or 2 if the authors did a more thorough experimental evaluation which showed the method working well when compared to alternatives, but that is not the case right now. \n\nNote on Section 6 (analysis) \nThe authors provide a simple analysis in Section 6 to bound the suboptimality of the learned policy. Unless I’m missing something, the resulting bound of t_3 <= t_1 + d is trivially true, since d is defined to be the diameter of O_{+}(o_g), and t_1 is the number of timesteps taken by an optimal policy to go from o_t to O_{+}(o_g). As a result, I don’t find this analysis illuminating or interesting - perhaps the author can provide counter arguments here to change my mind. \n\nNote on Section 7 (experiments) \nFor sim experiments, two of the tasks are extremely simple (free space reaching in 2D and 3D, respectively) where essentially everything works - the proposed method, baselines and ablations. The third task of rope manipulation is fairly interesting at a first look - but it appears to have been greatly simplified. The authors consider a high-level action space and an episode of only three timesteps. Further, the authors make the simulated robot arm invisible in the supplementary videos, which greatly simplifies the problem visually. Since the entire motivation is about learning well in real world settings, I feel this is a bit underwhelming. Figure 1 is misleading, since it shows a visible robot arm in front of a rope. This also appears to hint that the method did not work well with realistic visuals, highlighting a major limitation of the proposed approach. I think it would be valuable to include such failures (and discussions around them) in future submissions. \n\nFor the real world experiments, the task being considered is extremely simple (free space reaching), and does not even require pixel observations. Even for this simple task, the error achieved by the method is 10cm (starting error was 20cm), which is quite poor - robotic arms like the Sawyer should be able to achieve much lower errors. Even the oracle reward achieves an error of 10cm, which might indicate a bug in the author’s real world robotic setup. In comparison, prior work such as Nair et al. is able to tackle harder problems in the real world (like non-prehensile pushing). \n\nMinor points\n- Section 4 contains a nice discussion on false positives and false negatives when using non-oracle reward functions for reinforcement learning, where they also perform a simple experiment to show how false positives can negatively impact learning much more severely than false negatives. This does a good job of motivating the method (i.e. avoiding false positives), but also undermines the motivation behind reward filtering, which is perhaps the main technically novel component of the proposed approach. \n- Section 2.3 (i.e. related work on deformable object manipulation) states that \"Our approach applies directly to high-dimensional observations of the deformable object and does not require a prior model of the object being manipulated.”, and only cites prior work that assumes access to deformable object models. However, there is recent work that enable similar manipulation skills without access to such models. For example, Singh et al. [3] are able to learn to manipulate a deformable object (i.e. a piece of cloth) directly from high-dimensional observations using deep RL in the real world, and do not require object models (or ground truth state), but do require other forms of sparse supervision.\n- Typo: On page 7, “is kept the same a other approaches.” -> “is kept the same as other approaches.”\n\n[1]: Diagnosing Bottlenecks in Deep Q-learning Algorithms. Fu et al., ICML 2019\n[2]: Double Q-learning. V. Hasselt. NIPS 2010\n[3]: End-to-End Robotic Reinforcement Learning without Reward Engineering. Singh et al., RSS 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of reinforcement learning with goal-conditioned agents where the agents do not have access to the ground truth state.  The paper builds on the ideas in hindsight experience replay (HER), a method that relabels past trajectories with a goal set in hindsight.  This hindsight mechanism enables indicator reward functions to be useful even with image inputs.  Two technical contributions are reward balancing (balancing positive and negative experience) and reward filtering (a heuristic for removing false negatives).  The method is tested on multiple tasks including a novel RopePush task in simulation. 

The reviewers discussed strengths and limitations of the paper.  One strength was that the writing was clear for the reviewers. One limitation was the paper's novelty, as most of these ideas are already present in HER with the exception of reward filtering.  Another major concern was that the experiments were not sufficiently informative.  The simulation tasks did not adequately distinguish the proposed method from the baseline (in two of the three tasks) and the third task (RopePush) was simplified substantially (using invisible robot arms).  The real world task did not require the pixel observations.  The analysis of the method was also found to be somewhat limited by the reviewers, though this was partially addressed by the authors.

This paper is not yet ready for publication since the proposed method has insufficient supporting evidence.  A more thorough experiment could provide stronger evidence by showing a regime where the proposed method performs better than alternatives.
Decision: Reject