Reinforcement Learning without Ground-Truth State

1 INTRODUCTION To perform robot manipulation tasks , a low dimensional state of the environment typically needs to be estimated . In reinforcement learning , this state is also used to compute the reward function . However , designing a state estimator can be difficult , especially in environments with deformable objects , as shown in Figure 1 . An alternative is to learn an end-to-end policy that maps directly from high-dimensional sensor input to actions . However , without a state estimator , it is hard to specify a reward function based on high-dimensional observations . Past efforts to use reinforcement learning for robotics have avoided this issue in a number of ways . One common approach is to use extra sensors to determine the state of the environment during training , even if such sensors are not available at test time . Examples of this include using another robot arm to hold all relevant objects ( Levine et al. , 2016 ) , placing an IMU sensor ( Gu et al. , 2017 ; Yahya et al. , 2017 ) , motion capture markers on such objects ( Kormushev et al. , 2010 ) , or ensuring that all relevant objects are placed on scales ( Schenck & Fox , 2016 ) . However , such instrumentation is not always easy to set up for each task . This is especially true for deformable object manipulation , such as rope or cloth manipulation , in which every part of the object must be instrumented in order to measure the full state of the entire object . Attaching such sensors to food or granular material would present additional difficulties . We present an alternative approach for goal-conditioned reinforcement learning for specifying rewards using raw ( e.g . high-dimensional and continuous ) observations without requiring explicit state estimation or access to the ground-truth state of the environment . We achieve this using a simple indicator reward function , which only gives a positive reward when the robot ’ s observation exactly matches a target goal observation . Naturally , in continuous state spaces , we do not expect any two observed states to be identical . Surprisingly , we show that we can learn with such an indicator reward , even in continuous state spaces , if we use goal relabeling ( Kaelbling , 1993 ; Andrychowicz et al. , 2017 ) , which relabels the original goal with the achieved observation such that a positive reward is given . As the indicator rewards produce extreme sparse positive rewards , we further introduce reward balancing to balance the positive and negative rewards , as well as reward filtering to filter out uncertain rewards . We show theoretically that the indicator reward results in a policy with bounded suboptimality compared to the ground-truth reward . We also empirically show comparable performance between our method and an oracle which uses the ground-truth state for computing rewards , even though our method only operates on raw observations and does not have access to the ground-truth state . We demonstrate that an indicator reward can be used to teach a robot complex tasks such as rope manipulation from RGB-D images , without knowledge of the ground-truth state during training . Videos of our method can be found at https : //sites.google.com/view/image-rl . 2 RELATED WORK . 2.1 OBTAINING GROUND-TRUTH STATE FOR TRAINING . Adding sensors : To obtain ground-truth states for calculating rewards , one approach is to perform state estimation . However , such an approach can be noisy and challenging to implement , especially for the deformable objects that we study in this work . Another approach is to add extra sensors during training to accurately record the state . For example , in past work , one robot arm ( covered with a cloth at training time ) is used to rigidly hold and move an object , while another robot arm learns to manipulate the object ( Levine et al. , 2016 ) . In such a case , the object position can be inferred directly from the position of the robot gripper that is holding it . In other work on teaching a robot to open a door , an IMU sensor is placed on the door handle to determine the rotation angle of the handle and whether or not the door has been opened ( Gu et al. , 2017 ; Yahya et al. , 2017 ) . One can also ensure that all relevant objects for a task are placed on scales ( Schenck & Fox , 2016 ) or affixed with motion capture markers to obtain a precise estimate of their position ( Kormushev et al. , 2010 ) . However , such instrumentation is challenging for deformable objects , granular material , food , or other settings . Further , such instrumentation is costly and time-consuming to setup ; hence most of these previous approaches assume that such instrumentation is only available at training time and these methods do not allow further fine-tuning of the policy after deployment . Training in simulation : Another common approach is to train the policy entirely in simulation in which ground-truth state can be obtained from the simulator ( Fang et al. , 2018 ; Andrychowicz et al. , 2018b ; Zhu et al. , 2018 ; Sadeghi & Levine , 2016 ; Pinto et al. , 2018 ) . Many approaches have been explored to try to transfer such a policy from simulation to the real world , such as domain randomization ( Tobin et al. , 2017 ) or building a more accurate simulator ( Tan et al. , 2018 ; Chebotar et al. , 2018 ) . However , obtaining an accurate simulator is often very challenging , especially if the simulator differs from the real-world in unknown ways . Further , building the simulator itself can be fairly complex . Because these methods require the ground-truth state to obtain the reward function , they require training in a simulator and do not allow further fine-tuning after deployment in the real world ; our method , in contrast , does not require the ground-truth state for the reward function . 2.2 ROBOT LEARNING WITHOUT GROUND-TRUTH STATE . Learning a reward function without supervision : One line of work for learning a reward function is to first learn a latent representation and then derive the reward function based on the distance in the embedding space , such as cosine similarity . The representation can be learned by maximizing the mutual information between the achieved goal and the intended goal ( Warde-farley et al. , 2018 ) , reconstruction of the observation with VAE ( Nair et al. , 2018 ) , or learning to match keypoints with spatial autoencoders ( Finn et al. , 2016b ) . Recent work also explicitly learns a representation that is suitable for gradient based optimizer and then use it for specifying rewards ( Yu et al. , 2019b ) . However , there is no guarantee that these learned representation are suitable for deriving rewards for control . In addition , if the representation is pre-trained , it may also be incorrect in some parts of the observation space which can be exploited by the agent . Our approach is much simpler in that the reward function does not have any parameters that need to be learned and we empirically show better performance to some reward learning approaches . Changing the optimization : Another approach is to forego maximizing a sum of rewards as is typically done in reinforcement learning and instead optimize for another objective . For example , one method is to choose one-step greedy actions based on a learned one-step inverse dynamics model ; after training , the policy is then applied directly to a multi-step goal ( Agrawal et al. , 2016 ) . An alternative method is to learn a predictive forward dynamics model directly in a high-dimensional state space and use visual model-predictive control ( Finn et al. , 2016a ; Finn & Levine , 2017 ; Ebert et al. , 2017 ; 2018a ; b ) . Although these methods have shown some promise , predicting future highdimensional observations ( such as images or depth measurements ) is challenging . Another approach is to obtain expert demonstrations and define an objective as trying to imitate the expert ( Sermanet et al. , 2018 ; 2016 ; Peng et al. , 2018 ; Finn et al. , 2017 ) . Our approach , however , applies even when demonstrations are not available . 2.3 MANIPULATING DEFORMABLE OBJECTS . Deformable object manipulation presents many challenges for both perception and control . One approach to the perception problem is to perform non-rigid registration to a deformable model of the object being manipulated ( Huang et al. , 2015 ; Lee et al. , 2015 ; Schulman et al. , 2013 ; Wang et al. , 2011 ; Javdani et al. , 2011 ; Miller et al. , 2011 ; Cusumano-Towner et al. , 2011 ; Phillips-Grafflin & Berenson , 2014 ) . However , such an approach is often slow , leading to slow policy learning , and can produce errors , leading to poor policy performance . Further , such an approach often requires a 3D deformable model of the object being manipulated , which may be difficult to obtain . Our approach applies directly to high-dimensional observations of the deformable object and does not require a prior model of the object being manipulated . 3 PROBLEM FORMULATION . In reinforcement learning , an agent interacts with the environment over discrete time steps . In each time step t , the agent observes the current state st and takes an action at . In the next time step , the agent transitions to a new state st+1 based on the transition dynamics p ( st+1|st , at ) and receives a reward rt+1 = r ( st , at , st+1 ) . The objective for the agent is to learn a policy π ( at|st ) that maximizes the expected future return R = E [ ∑∞ t=0 γ trt+1 ] , where γ is a discount factor . 3.1 GOAL-REACHING REINFORCEMENT LEARNING . In order for the agent to learn diverse and general skills , we define a goal reaching MDP ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) as follows : In the beginning of each episode , a goal state sg is sampled from a goal distribution G. We learn a goal conditioned policy π ( at|st , sg ) that tries to reach any goal state from the goal distribution . We use a goal conditioned reward function rt = r ( st+1 , sg ) and optimize for Esg∼G [ ∑∞ t=0 γ trt ] . The transition dynamics p ( st+1|st , at ) of the environment remain independent of the goal . In many real-world scenarios , it is often difficult to construct a well-shaped reward function . Past work has shown that sparse rewards , combined with an appropriate learning algorithm , can achieve better performance than poorly-shaped dense rewards in goal-reaching environments ( Andrychowicz et al. , 2017 ) . We thus define a sparse reward function that only makes the binary decision of whether the goal is reached or not . Specifically , let S+ ( sg ) be a subset of the state space such that any state in this set is determined to be sufficiently close to sg ( in some unknown metric ) ; in other words , if the environmental state is within S+ ( sg ) , then the task of reaching sg can be considered to be achieved.1 Naturally , we can assume that sg ∈ S+ ( sg ) . A binary reward function can then be defined as r ( st+1 , sg ) = { R+ st+1 ∈ S+ ( sg ) R− st+1 /∈ S+ ( sg ) , ( 1 ) where R+ and R− are constants representing the rewards received for achieving the goal and failing to achieve the goal , respectively . 1S+ is a function that maps from the state space to a subset of the space .

Summary: The paper tackles the problem of self-supervised reinforcement learning through the lens of goal-conditioned RL, which is in line with recent work (Nair et al, Wade-Farley et al, Florensa et al, Yu et al.). The proposed approach is a simple one - it uses the relabeling trick from (Kaelbling, 1993; Andrychowicz et al., 2017) to assign binary rewards to the collected trajectories. They apply two simple tricks on top of relabeling: