Hallucinative Topological Memory for Zero-Shot Visual Planning

1 INTRODUCTION . For robots to operate in unstructured environments such as homes and hospitals , they need to manipulate objects and solve complex tasks as they perceive the physical world . While task planning and object manipulation have been studied in the classical AI paradigm [ 20 , 9 , 30 , 10 ] , most successes have relied on a human-designed state representation and perception , which can be challenging to obtain in unstructured domains . While high-dimensional sensory input such as images can be easy to acquire , planning using raw percepts is challenging . This has motivated the investigation of datadriven approaches for robotic manipulation . For example , deep reinforcement learning ( RL ) has made impressive progress in handling high-dimensional sensory inputs and solving complex tasks in recent years [ 7 , 4 , 15 , 23 ] . One of the main challenges in deploying deep RL methods in human-centric environment is interpretability . For example , before executing a potentially dangerous task , it would be desirable to visualize what the robot is planning to do step by step , and intervene if necessary . Addressing both data-driven modeling and interpretability , the visual planning ( VP ) paradigm seeks to learn a model of the environment from raw perception and then produce a visual plan of solving a task before actually executing a robot action . Recently , several studies in manipulation and navigation [ 13 , 29 , 5 , 22 ] have investigated VP approaches that first learn what is possible to do in a particular environment by self-supervised interaction , and then use the learned model to generate a visual plan from the current state to the goal , and finally apply visual servoing to follow the plan . One particularly promising approach to VP is the semi-parametric topological memory ( SPTM ) method proposed by Savinov et al . [ 22 ] . In SPTM , images collected offline are treated as nodes in a graph and represent the possible states of the system . To connect nodes in this graph , an image classifier is trained to predict whether pairs of images were ‘ close ’ in the data or not , effectively learning which image transitions are feasible in a small number of steps . The SPTM graph can then be used to generate a visual plan – a sequence of images between a pair of start and goal images – by directly searching the graph . SPTM has several advantages , such as producing highly interpretable visual plans and the ability to plan long-horizon behavior . However , since SPTM builds the visual plan directly from images in the data , when the environment changes – for example , the lighting varies , the camera is slightly moved , or other objects are displaced – SPTM requires recollecting images in the new environment ; in this sense , SPTM does not generalize in a zero-shot sense . Additionally , similar to [ 5 ] , we find that training the graph connectivity classifier as originally proposed by [ 22 ] requires extensive manual tuning . In this work , we propose to improve both the robustness and zero-shot generalization of SPTM . To tackle the issue of generalization , we assume that the environment is described using some context vector , which can be an image of the domain or any other observation data that contains enough information to extract a plan ( see Figure 1 top left ) . We then train a conditional generative model that hallucinates possible states of the domain conditioned on the context vector . Thus , given an unseen context , the generative model hallucinates exploration data without requiring actual exploration . When building the connectivity graph with these hallucinated images , we replace the vanilla classifier used in SPTM with an energy-based model that employs a contrastive loss . We show that this alteration drastically improves planning robustness and quality . Finally , for planning , instead of connecting nodes in the graph according to an arbitrary threshold of the connectivity classifier , as in SPTM , we cast the planning as an inference problem , and efficiently search for the shortest path in a graph with weights proportional to the inverse of a proximity score from our energy model . Empirically , we demonstrate that this provides much smoother plans and barely requires any hyperparameter tuning . We term our approach Hallucinative Topological Memory ( HTM ) . A visual overview of our algorithm is presented in Figure 1 . We evaluate our method on a set of simulated VP problems of moving an object between obstacles , which require long-horizon planning . In contrast with prior work , which only focused on the success of the method in executing a task , here we also measure the interpretability of visual planning , through mean opinion scores of features such as image fidelity and feasibility of the image sequence . In both measures , HTM outperforms state-of-the-art data-driven approaches such as visual foresight [ 4 ] and the original SPTM . 2 BACKGROUND . Context-Conditional Visual Planning and Acting ( VPA ) Problem . We consider the contextconditional visual planning problem from [ 13 , 29 ] . Consider deterministic and fully-observable environments E1 , ... , EN that are sampled from an environment distribution PE . Each environment Ei can be described by a context vector ci that entirely defines the dynamics oit+1 = m ( o i t , a i t|ci ) , where oit , a i t are the observations and actions , respectively , at timestep t from context ci . For example , in the illustration in Figure 1 , the context could represent an image of the obstacle positions , which is enough to predict the possible movement of objects in the domain.1 As is typical in VP problems , we assume our data D = { oi1 , ai1 , ... , oiTi , ci } i∈ { 1 , ... , N } is collected in a self-supervised manner , and that in each environment Ei , the observation distribution is defined as Po ( ·|ci ) . At test time , we are presented with a new environment , its corresponding context vector c , and a pair of start and goal observations ostart , ogoal . Our goal is to use the training data to build a planner Qh ( ostart , ogoal , c ) and an h-horizon policy πh . The planner ’ s task is to generate a sequence of observations between ostart and ogoal , in which any two consecutive observations are reachable within h time steps . The policy takes as input the image sequence and outputs a control policy that transitions the system from ostart to ogoal . As the problem requires a full plan given only a context image in the new environment , the planner must be capable of zero-shot generalization . Note that the planner and policy form an interpretable planning method that allows us to evaluate their performance separately . For simplicity we will omit the subscript h for the planner and the policy . Semi-Parametric Topological Memory ( SPTM ) [ 22 ] is a visual planning method that can be used to solve a special case of VPA . where there is only a single training environment , E and no context image . SPTM builds a memory-based planner and an inverse-model controller . At training , a classifier R is trained to map two observation images oi , oj to a score ∈ [ 0 , 1 ] representing the feasibility of the transition , where images that are ≤ h steps apart are labeled positive and images that are ≥ l are negative . The policy is trained as an inverse model L , mapping a pair of observation images oi , oj to an appropriate action a that transitions the system from oi to oj . Given an unseen environment E∗ , new observations are manually collected and organized as nodes in a graph G. Edges in the graph connect observations oi , oj if R ( oi , oj ) ≥ sshortcut , where sshortcut is a manually defined threshold . To plan , given start and goal observations ostart and ogoal , SPTM first usesR to localize , i.e. , find the closest nodes inG to ostart and ogoal . A path is found by running Dijkstra ’ s algorithm , and the method then selects a waypoint owi on the path which represents the farthest observation that is still feasible under R. Since both the current localized state oi and its waypoint owi are in the observation space , we can directly apply the inverse model and take the action ai where ai = L ( oi , owi ) . After localizing to the new observation state reached by ai , SPTM repeats the process until the node closest to ogoal is reached . Conditional Variational Auto-Encoder ( CVAE ) [ 25 ] is a deep generative model that can be used for learning a high-dimensional conditional distribution Po ( ·|c ) . The CVAE is trained by maximizing the evidence lower bound ( ELBO ) : LCV AE = −DKL ( qθ ( z|o , c ) |rψ ( z|c ) ) + Eqφ ( z|o , c ) [ log pθ ( o|z , c ) ] , where qθ ( z|o , c ) is the encoder that maps observations and contexts to the latent distribution , pθ ( o|z , c ) is the decoder that maps latents and contexts to the observation distribution , and rψ ( z|c ) is the prior that maps contexts to latent prior distributions . Together pθ , qφ , rψ are trained to maximize the variational lower bound above . We assume that the prior and the encoder are Gaussian , which allows the DKL term to be computed in closed-form . Monte-Carlo sampling and the reparametrization trick [ 12 ] are used to approximate the gradient of the loss . Contrastive Predictive Coding ( CPC ) [ 17 ] extracts compact representations that maximize the causal and predictive aspects of high-dimensional sequential data . A non-linear encoder genc encodes the observation ot to a latent representation zt = genc ( ot ) . We maximize the mutual information between the latent representation zt and future observation ot+k with a log- 1We used such a context image in our experiments . We assume that in a practical application , observing the domain without the robot would be feasible , making this setting relevant to applications . bilinear model2 fk ( ot+k , ot ) = exp ( zTt+kWkzt ) . This model is trained to be proportional to the density ratio p ( ot+k|zt ) /p ( ot+k ) by the CPC loss function : the cross entropy loss of correctly classifying a positive sample from a set X = { o1 , ... , oN } of N random samples with 1 positive sample from p ( ot+k|zt ) and N − 1 negative samples sampled from p ( ot+k ) : LCPC = −Eot , ot+k [ log fk ( ot+k , ot ) ∑ oj∈X fk ( oj , ot ) ] . 3 HALLUCINATIVE TOPOLOGICAL MEMORY . SPTM has been shown to solve long-horizon planning problems such as navigation from first-person view [ 22 ] . However , SPTM is not zero-shot : even a small change to the training environment requires collecting substantial exploration data for building the planning graph . This can be a limitation in practice , especially in robotic domains , as any interaction with the environment requires robot time , and exploring a new environment can be challenging ( indeed , [ 22 ] applied manual exploration ) . In addition , similarly to [ 5 ] , we found that training the connectivity classifier as proposed in [ 22 ] requires extensive hyperparameter tuning . In this section , we propose an extension of SPTM to overcome these two challenges by employing three ideas – ( 1 ) using a CVAE [ 25 ] to hallucinate samples in a zero-shot setting , ( 2 ) using contrastive loss for a more robust score function and planner , and ( 3 ) planning based on an approximate maximum likelihood formulation of the shortest path under uniform state distribution . We call this approach Hallucinative Topological Memory ( HTM ) , and next detail each component in our method .

Summary: The paper presents a method for learning agents to solve visual planning, in particular to navigate to a desired goal position in a maze, with a learned topological map, i.e. a graph, where nodes correspond to positions in the maze and edges correspond accessibility (reachability in a certain number of steps). The work extends previous work (semi parametric topological memory, ref. [22]) in several ways. It claims to address a shortcoming of [22], namely the fact that the graph is calculated offline from random rollouts, by using a conditional variational auto-encoder to predict a set of observed images which could lie between the current position and the goal position, and, most importantly from a context image which describes the layout of the environment. These predicted images are then arranged in a graph through a connectivity predictor, which is trained from rollouts through a contrastive loss. Training is performed on multiple environments, and the context vector provides enough information for this connectivity network to generalize to unseen environments.  At test time, the agent navigates using a planner and a policy. The planner calculates the shortest path on a graph where edges are connectivity probabilities, and the policy is an inverse model trained on the output of the planner.