Zero-Shot Reward Specification via Grounded Natural Language

1 INTRODUCTION Previous efforts have explored image-based goal specification , with significant successes in visual navigation and manipulation tasks ( Pathak et al. , 2018 ; Nair et al. , 2018 ; Fu et al. , 2018 ; Singh et al. , 2019 ) . Yet existing image-based goal specification paradigms are limited because they are typically limited to a particular scene instance in an environment , whereas a semantic goal comprises multiple possible scene configurations . Reinforcement learning offers one of the most appealing premises in the study of AI : from a reward signal alone , algorithms which learn optimal policies that maximize expected reward can learn to perform navigation , dexterous manipulation , and host of other impactful tasks . However , discovering or specifying a reward function for a given task is often a very challenging problem , especially when one is considering agents that can learn from uninstrumented environments , e.g. , from raw image observations alone . We wish to have an agent that can learn purely from pixels , with no access to the underlying state of the environment at any point during learning or task execution . Achieving this goal without access to an instrumented reward function has been exceedingly challenging . One can use image-based reward specification to cause a robot agent to navigate to a particular chair next to a specific tall plant , but that agent may not always succeed at the generic task of “ go to a chair next to a tall flowering plant ” : e.g. , if the goal specification image shows a red chair next to a plant with a yellow flower the agent may navigate away from a scene with a blue chair next to a red flower , depending on the model ’ s underlying image representation . To be sure that the true goal is properly specified irrespective of the invariances of the model ’ s underlying perceptual representation , a user may have to provide a set of goal image examples that cover the variation of the target concept , potentially a very expensive undertaking to collect or generate . We advocate semantic reward specification via grounded natural language , where a user describes a goal configuration in the world using a natural language description referring to entities in the world . This direction has long been a “ holy grail ” of AI research and a presumed capability of AI science fiction—the ability to instruct a robot with natural language—yet attempts have been limited by the state of the art in grounded language perception . It also falls under the general umbrella of leveraging large-scale passive data to bootstrap embodied learning , where we rely on language as a means to provide necessary reward shaping missing in visual data alone . Several previous efforts train reward functions or policies that take natural language as input for goal description ( Oh et al. , 2017 ; Bahdanau et al. , 2018 ; Zhou & Small , 2020 ; Goyal et al. , 2020 ; Fu et al. , 2019 ; Hermann et al. , 2017 ; Shao et al. , 2020 ) . They all however rely on reward signals that have access to state of the system or demonstrations of the task distribution they are training on . There are works that use human videos to learn reward functions to train their agent with ( Shao et al. , 2020 ; Sermanet et al. , 2018 ; 2016 ) , but they require a curated dataset of humans performing the tasks . Recently however , the advent of large-scale multimodal training data together with large capacity language and vision deep learning models has significantly advanced the state of the art . A steady series of innovations have advanced grounded language modeling , from early work on multimodal translation and fusion models ( Barnard et al. , 2003 ; Quattoni et al. , 2007 ; Guadarrama et al. , 2016 ) , to large-scale joint embedding models ( Frome et al. , 2013 ; Radford et al. , 2021 ) , to the plethora of multimodal transformer models currently under investigation ( Su et al. , 2019 ; Lu et al. , 2019 ; Chen et al. , 2019 ; Hu & Singh , 2021 ) . CLIP , in particular , demonstrated a transformative advance on zero-shot object recognition ( Radford et al. , 2021 ) . We could specify a goal state to a robot by simply offering a description of the goal configuration in natural language and using the CLIP embedding dot product with an observed image to evaluate proximity to goal state . Surprisingly , this can work in simple cases , for examples see the top example in Figure 5 . But for more complex goals performance can be poor , especially with complex captioned scenes involving spatial relationships and on domains that differ from web images ( as shown in ( Radford et al. , 2021 ) ) and the bottom example in that figure . To overcome these limitations , we propose a factored grounding of ‘ what ’ vs ‘ where/how ’ aspects of goal state ( Figure 2 ) , and offer a novel spatial-salience scheme for the former . We argue that existing ( e.g. , CLIP-like ) models can be used to ground the what aspects of a goal quite effectively , including appropriate attribute and concept-level generalization , while a separate where/how module can ground spatial relationship aspects of goal configuration . After training many tasks with this reward scheme we learn a language conditioned policy that can execute new tasks in a zero-shot fashion – from a natural language description of task – without having to train a new policy for every new task . In summary , our contributions are as follows : 1 . A zero-shot reward function to learn a set of robotic manipulation tasks on raw pixels without access to state . ( Zero-shot in the sense that no goal images or human demonstrations are provided at training time or test time . ) 2 . A distilled language-conditioned multi-task policy on several tasks learned using the zero-shot reward function which is able to deploy new unseen tasks . 2 METHOD . 2.1 BASE ZERO-SHOT REWARD MODEL : DOT-PRODUCT VISUOLINGUISTIC REWARD . Our base model is the most intuitive way to use the CLIP model to compute reward . Our base model simply computes reward by taking a dot product between the goal language input feature and image observation feature through CLIP ’ s language and image encoders respectively : rt = EI ( It ) ·EL ( g ) A subset of tasks can be learned with this reward model . We visualize the limits of this model on two environments in Figure 5 . 2.2 FULL ZERO-SHOT REWARD MODEL : SPATIALLY GROUNDED VISUOLINGUISTIC REWARD . CLIP-Saliency for Phrase Grounding We have developed a simple method which factors phrase grounding and spatial relationship processing , which can be used to specify a reward function that operates only on image content using a natural language description . Figure 2 illustrates our full model leverages CLIP ’ s encoders in a very different way than the base model . Our model first parses the goal language description g into object noun phrases go1 ... gok and the desired object interaction . The object noun phrases are passed through CLIP ’ s language encoder and the current image observation is passed through CLIP ’ s image encoder . We use the language encodings as class embeddings of each object noun phrase to perform a saliency analysis ( using Grad-CAM ( Selvaraju et al. , 2017 ) ) on the image encoding of the observation on the last convolutional layer ( specifically , the ReLU layer of CLIP ’ s ResNet-50 backbone ) . Saliency models such as Grad-CAM generally output a heatmap of the features that indicate a class exists in the current image input . The state extractor in Figure 2 computes the “ state ” of each object using the argmax of the saliency heatmap . See Figure 3 . Spatial Grounding Heuristics Our model determines task completion using a spatial grounding rule based reward function that takes the object pixel coordinates of one or more camera views and desired spatial relationship as input . Our spatial heuristics can be used to describe a broad set of manipulation tasks to push or place objects to semantic locations . A reward of 1 is given to the RL algorithm for every timestep that the conditions of the reward criteria are met for a given spatial relationship . The oracle reward uses true x , y , z spatial positions of the objects to determine if the desired spatial relationship is reached and also outputs 1 to the RL algorithm for every timestep the conditions of the task are met . Our spatial heuristics are fully defined in Table 1 . The first set ( left of , right of , on top of , below , and in between ) assume coordinates in a front camera view and the semantics are defined in that camera view with positive y in the upward direction and positive x in the right direction . The second set ( in front of & behind ) has access to a left camera view with coordinate x2 pointing towards the right which is towards the front in the first camera view and y2 pointing upward similar to the first camera view . The second camera is needed to know if the object is placed in the front or behind another object correctly . The third set ( close to & inside of ) require access to a front camera view with a 45 degree downward tilt towards the ground . This camera view is needed to see if the object is getting closer to another object in two orthogonal directions at once where as a left only or front view only allows you to determine one dimension of closeness . For “ inside of ” a 45 degree camera helps the agent see if the object is going inside another object without occlusion . The threshold for “ inside of ” is much smaller than for “ close to ” since the centroid can come much closer when an object goes inside a container object . 2.3 LANGUAGE CONDITIONED MULTI-TASK POLICY . Now that we have a method for learning tasks in a zero-shot reward fashion , we would like to be able to not require training a new policy for every new task and ideally have a language conditioned multi-task policy that takes in the goal text description of an unseen task and executes the task without needing any new samples from the environment . To approach this goal , we first learn several tasks via reinforcement learning using our zero-shot reward model . We then create a large dataset of the rollouts of those polices and pair each trajectory with the goal text description of the tasks it was trained to learn . We then use behavioral cloning ( supervised learning for predicting actions ) to learn a policy that takes images and text goal task description as input and actions as target outputs . We use CLIP ’ s language encoder as the text goal embedding that is fed to the multi-task policy . In addition we use image augmentation techniques to aid behavioral cloning in learning more robust policies . 3 EXPERIMENTS . 3.1 PHRASE GROUNDING AND BASE ZERO-SHOT REWARD MODEL VISUALIZATION . Object detectors are one way to extract object states , however , they are usually not used off the shelf and need to be fine-tuned with in domain data . In Figure 3 we show pretrained Mask R-CNN ( He et al. , 2017 ) outputs on different camera views for the block stack environment . As you can see in the first two subfigures , the blocks are not proposed as objects with Mask R-CNN from both far and close camera views . This is a demonstration of the need for in-domain fine-tuning of object detectors to work in the environment you want to use . Our Grad-CAM output from CLIP however , highlights exactly the objects we are interested in from the object noun phrases that describes each object in the last two subfigures . The CLIP model has been trained on a much larger dataset than Mask R-CNN and has a language component that allows us to request for the location of the object noun phrase of interest . With Mask RCNN however , even if the object proposals were good , it would not necessarily classify objects it has not been trained for and therefore not output a filtered set of object proposals . In other words , we would not know which object is which if the detector hasn ’ t been trained with the label of objects we care about . In Figure 5 we show what our base reward model outputs on two goal descriptions : 1. inverted pendulum 2. yellow object close to a blue object . For the first goal description we observe that the dot product increases as the pendulum becomes more inverted from either side as desired . For the second goal description we observe that as the blue object gets closer to the yellow object the dot product increases except for the closest image where it dips which results in an undesired output . We observed through these two goal descriptions and many others that the base reward model is not sufficient for recognizing object spatial relationships . The encoders are good at identifying what objects are in the image however , which is what we leverage for our full reward model .

Summary: This paper demonstrates how to use pretrained CLIP model as a reward function for an RL agent. This approach enables flexible goal specification using language. The proposed reward generation method uses CLIP to identify the relevant objects and a separate module to compute the reward based on a specified spatial relationship. This separation produces a better reward function than using the dot product between the embedded image vector and language vector.