Hallucinative Topological Memory for Zero-Shot Visual Planning
Authors: Kara Liu
Thanard Kurutach
Pieter Abbeel
Aviv Tamar
Venue:  (2020)
Keywords: Visual Planning
Model-Based RL
Representation Learning

Abstract: In visual planning (VP), an agent learns to plan goal-directed behavior from observations of a dynamical system obtained offline, e.g., images obtained from self-supervised robot interaction. VP algorithms essentially combine data-driven perception and planning, and are important for robotic manipulation and navigation domains, among others. A recent and promising approach to VP is the semi-parametric topological memory (SPTM) method, where image samples are treated as nodes in a graph, and the connectivity in the graph is learned using deep image classification. Thus, the learned graph represents the topological connectivity of the data, and planning can be performed using conventional graph search methods. However, training SPTM necessitates a suitable loss function for the connectivity classifier, which requires non-trivial manual tuning. More importantly, SPTM is constricted in its ability to generalize to changes in the domain, as its graph is constructed from direct observations and thus requires collecting new samples for planning. In this paper, we propose Hallucinative Topological Memory (HTM), which overcomes these shortcomings. In HTM, instead of training a discriminative classifier we train an energy function using contrastive predictive coding. In addition, we learn a conditional VAE model that generates samples given a context image of the domain, and use these hallucinated samples for building the connectivity graph, allowing for zero-shot generalization to domain changes. In simulated domains, HTM outperforms conventional SPTM and visual foresight methods in terms of both plan quality and success in long-horizon planning. 

Under review as a conference paper at ICLR 2020
HALLUCINATIVE TOPOLOGICAL MEMORY FOR ZERO- SHOT VISUAL PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
In visual planning (VP), agent learns plan goal-directed behavior from ob- servations of a dynamical system obtained offline, c.g. images obtained from self- supervised robot interaction. VP algorithms essentially combine data-driven per- ception and planning, and are important for robotic manipulation and navigation domains, among others. A recent and promising approach VP is the semi- parametric topological memory (SPTM) method, where image samples are treated as nodes in a graph, and the connectivity in the graph is learned using deep image classification. Thus, the learned graph represents the topological connectivity of the data, and planning can be performed using conventional graph search methods. However, training SPTM necessitates a suitable loss function for the connectivity classifier, which requires non-trivial manual tuning. More importantly, SPTM is constrieted in its ability to generalize to changes in the domain, its graph is constructed from direct observations and thus requires collecting new samples for planning. In this paper, we propose Hallucinative Topological Memory (HTM), which overcomes these shortcomings. In HTM, instead of training a discrimina- tive classifier we train an energy function using contrastive predictive coding. In addition, we learn a conditional VAE model that generates samples given a con- text image the domain, and use these hallucinated samples for building the connectivity graph, allowing for zero-shot generalization to domain changes. In simulated domains, HTM outperforms conventional SPTM and visual foresight methods in terms of both plan quality and success in long-horizon planning.
INTRODUCTION
For robots operate in unstructured environments such as homes and hospitals, they need to manip- ulate objects and solve complex tasks as they perceive the physical world. While task planning and object manipulation have been studied in the classical AI paradigm 120 9 30 10], most successes have relied on a human-designed state representation and perception, which can be challenging to obtain in unstructured domains. While high-dimensional sensory input such as images to acquire, planning using raw percepts is challenging. This has motivated the investigation of data- driven approaches for robotic manipulation. For example, deep reinforcement learning (RL) has made impressive progress in handling high-ding ligh-dimensional sensory inputs and solving complex tasks in recent years
be easy
15 23
One of the main challenges in deploying deep RL methods in human-centric environment is inter- pretability. For example, before executing potentially dangerous task, it would be desirable to visualize what the robot is planning to do step by step, and intervene if necessary. Addressing both data-driven modeling and interpretability. the visual planning (VP) paradigm seeks to learn a model of the environment from raw perception and then produce visual plan of solving task before actu- ally executing a robot action. Recently, several studies manipulation and navigation [3] 29 22 have investigated VP approaches that first learn what is possible to do in particular environment by self-supervised interaction, and then use the learned model to generate a visual plan from the current state to the goal, and finally apply visual servoing follow the plan.
One particularly promising approach to VP is the sarameerai topological memory (SPTM) method proposed by Savinov et al. (22). In SPTM, images collected offline treated as nodes in a graph and represent the possible states of the system. To connect nodes in this graph, an image classifier is trained predict whether pairs of images were 'close" in the data not, effectively

---

Under review as a conference paper at ICLR 2020
learning which image transitions are feasible in small number of steps. The SPTM graph then be used to generate visual plan - a sequence of images between a pair of start and goal images - by directly searching the graph. SPTM has several advantages, such as producing highly interpretable visual plans and the ability to plan long-horizon behavior.
However, since SPTM builds the visual plan directly from images in the data, when the environ- ment changes - for example, the lighting varies, the camera is slightly moved, or other objects are displaced SPTM requires recollecting images in the new environment; in this sense, SPTM does not generalize in a zero-shot sense. Additionally, similar to [S], we find that training the graph connectivity classifier as originally proposed by [22] requires extensive manual tuning.
Training Data
HTM Visual Planning
Contaxt
Test Context
Hallucinated graph
Context
Start
Context
Test Goal
HTM Model
Conditional VAE
Connectivity Energy
Context
High
Visuual Plan
Random
Noise
Hallucinated Images
HTM illustration. Top left: data collection. In this illustration, the task is move green object between gray obstacles. Data consists of multiple obstacle configurations (contexts), and images of random movement of the object each configuration. Bottom left: the elements of HTM. A CVAE is trained to hallucinate images of the object and obstacles conditioned on the obstacle image context. connectivity energy model trained to score pairs of images based on the feasibility of their transition. Right: HTM visual planning. Given pair of start and goal images, we first use the CVAE to hallucinate possible images of object and obstacles. Then, connectivity graph (blue dotted lines) is computed based on the connectivity energy, and we plan for the shortest path from start to goal on this graph (orange solid line). For exccuting the plan, visual servoing controller is later used to track the image sequence.
Figure 1:
context image and
In this work, we propose to improve both the robustness and zero-shot generalization of SPTM. To tackle the issue of generalization, vector, which can be an image of the domain or any other observation data that contains enough information to extract plan (see Figure 1 top left). We then train conditional generative model that hallucinates possible states of the domain conditioned on the context vector. Thus, given an unseen context, the generative model hallucinates exploration data without requiring actual explo- ration. When building the connectivity graph with these hallucinated images, we replace the vanilla classifier used in SPTM with an energy-based model that employs contrastive loss. We show that this alteration drastically improves planning robustness and quality. Finally, for planning, instead of connecting nodes in the graph according an arbitrary threshold of the connectivity classifier, as in SPTM, we cast the planning path in graph with weights proportional to the inverse of proximity score from our energy model. Empirically, demonstrate that this provides much smoother plans and barely requires any hy- perparameter tuning. We term our approach Hallucinative Topological Memory (HTM), visual overview of our algorithm is presented in Figure D
assume that the environment is described using some context
inference problem, and efficiently search for the shortest
We evaluate our method on a set simulated VP problems of moving an object between obstacles, which require horizon planning. In contrast with prior work, which only focused on the success of the method executing task, here we also measure the interpretabilityy of visual planning, through mean opinion scores of features such as image fidelity and feasibility of the image sequence.
2

---

Under review as a conference paper at ICLR 2020
In both measures, HTM outperforms state-of- ofrthh-art data-driven approaches such visual foresight and the original SPTM.
BACKGROUND
Visual Planning and Acting (VPA) Problem. We consider the context-
conditional visual planning problem from [3 29]. Consider deterministic and fully-observable environments ,.,..., Ex that are sampled from an environment distribution Pr. Each environment E; can be described by a context vector & that entirely defines the dynamics o/+1 = m(ff. ajfcz), where of, of are the observations and actions, respectively, at timestep from context q, For example, in the illustration in Figure T the context could represent image of the obstacle positions, which is enough to predict the possible movement of objects in the domain | As is typical in VP problems, assume data D foi, a ...., 022 Gi (e)iel....... is collected in self-supervised manner, and that in each environment E,, the observation distribution is defined as P. (-(cc)). At test time, we presented with a new environment, its corresponding context vector c, and pair of start and goal observations Ostart; Ogoat- Our goal is to the training data to build planner Qu Ogont, c) and an h-horizon policy Ty- The planner's task is to generate a sequence of observations between Ostart and O Oyoat. in which any two consecutive observations are reachable within h time steps. The policy takes as input the image sequence and outputs a control policy that transitions the system from Ostart to Ogoul- As the problem requires full plan given only a context image in the new environment, the planner must be capable of zero-shot generalization. Note that the planner and policy form an interpretable planning method that allows us to evaluate their performance separately. For simplicity we will omit the subscript h for the planner and the policy.
Semi-Parametric Topological Memory (SPTM) 22 is visual planning method that can be used to solve special case of VPA. where there is only single training environment, E and context image. SPTM builds a memory-based planner and an inverse-mode model controller. At training, a classi- fier R is trained to map two observation images og, Oj to score E [0, 1] representing the feasibility of the transition, where images that are < à steps apart are labeled positive and images that are 2 I negative. The policy is trained oz, Oj to an appropriate action a that transitions the system from o, to 0j.
inverse model L, mapping pair of observation images
Given unseen environment & new observations manually collected and organized as nodes in a graph G. Edges in the graph connect observations oz, 0j if R(o,, oj) 2 8shorteut- where Sshortout a manually defined threshold. To plan, given start and goal observations Ostart and Ogoat. SPTM first uses H to localize, i.c., find the closest nodes in G to Ostart and Ogool. A path is found by running Dijkstra's algorithm, and the method then selects a waypoint Ow, on the path which represents the farthest observation that is still feasible under R. Since both the current localized state o; and its waypoint Ou,, are the observation space, we can directly apply the inverse model and take the action where a; L(o;, Ow,)). ). After localizing to the new observation state reached by a,, SPTM repeats the process until the node closest to Ogoat reached.
Conditional Variational Auto-Encoder (CVAE) 255 is deep generative model that can be used for learning a high-dimensional conditional distribution Po(-lc). The CVAE is trained by maximizing the evidence lower bound (ELBO): LCVAE -DKE (go(zlo, (z(e)())e) + (xooc)) [logpe(o)z: (o(o) c)], where go(2|o,c) c) is the encoder that maps observations and contexts to the latent distribution, Pe (ol c) is the decoder that maps latents and contexts to the observation dis- tribution, and (2|c) is the prior that maps contexts to latent prior distributions. Together pe, qa T are trained to maximize the variational lower bound above. We assume that the prior and the encoder Gaussian, which allows the DEL term to be computed in closed-form. Monte-Carlo sampling and the reparametrization trick [12] are used to approximate the gradient of the loss.
Contrastive Predictive Coding (CPC) [17 extracts compact representations that maximize the causal and predictive aspects of high-dimensional sequential data. A non-linear encoder Jene encodes the observation or a latent representation 2 = (o)) We maximize the mu- tual information between the latent representation 2 and future observation ot+ with a log-
We used such context image in our experiments. We assume that in practical application, observing the domain without the robot would he feasible, making this setting relevant applications.
3

---

Under review as a conference paper at ICLR 2020
bilinear model P fir fo((044)00)) exp(= W22)) This model is trained to be proportional to the density ratio plot+</=)//pop// by the CPC loss function: the cross entropy loss of correctly classifying positive sample from a set X = {o,,... oN3 of N random samples with 1 pos- itive sample from p(or+s|2s)) and N 1 negative samples sampled from p(or+k): Lopc
- -Eo, Prok log Fuloo- k.00)) Eojex (a(3,00))
3 HALLUCINATIVE TOPOLOGICAL MEMORY
SPTM has been shown solve long-horizon planning problems such navigation from first-person view [22]. However, SPTM is not zero-shot: even a small change to the training environment re- quires collecting substantial exploration data for building the planning graph. This be limi- tation in practice, especially in robotic domains, as any interaction with the environment requires robot time, and exploring new environment can be challenging (indeed, [22] applied manual ex- ploration). In addition, similarly to Sl we found that training the connectivity classifier as proposed in [2 requires extensive hyperparameter tuning.
In this section, we propose an extension of SPTM to overcome these two challenges by employing three ideas- - (1) using a CVAE (25] to hallucinate samples in zero-shot setting, (2) using con- trastive loss for more robust score function and planner, and (3) planning based on an approximate maximum likelihood formulation of the shortest path under uniform state distribution. We call this approach Hallucinative Topological Memory (HTM), and next detail each component in our method.
3.1 HALLUCINATING SAMPLES WITH CVAE
We propose zero-shot learning solution for automatically building the planning graph using only a context vector of the new environment. Our idea is that, after seeing many different environments and corresponding states of the system during training, given new environment we should be able to effectively hallucinate possible system states. We can then use these hallucinations in lieu of real samples from the system in order build the planning graph. To generate images conditioned on During training, we learn the prior latent
context, we implement a CVAE as depicted in Figure [ distribution rip (2|c), modeled as a Gaussian with mean (c()) and covariance matrix E(c), where μu-) and X are learned non-linear neural network transformations. During testing, when prompted with a new context vector c, we can sample latent vectors 21, nN c ~ (u(e), E(c)), and pass them through the decoder pe (z]2z, c) for hallucinating samples in replacement of exploration data.
3.2 LEARNING THE CONNECTIVITY VIA CONTRASTIVE Loss
A critical component in the SPTM method is the connectivity classifier that decides which image transitions are feasible. False positives may result in impossible short-cuts in the graph, while false negatives make the plan unnecessarily long. In (22), the classifier was trained diseriminatively, using observations in the data that were reached within h steps as positive examples, and more than steps as negative examples, where h and are chosen arbitrarily. In practice, this leads to three important problems. First, this method is known to be sensitive to the choice of positive and negative labeling [5]. Second, training data are required to be long, non-cyclic trajectories for a high likelihood of sampling 'true' negative samples. However, self-supervised interaction data often resembles random walks that repeatedly visit a similar state, leading to inconsistent estimates on what constitutes negative data. Third, since the classifier is only trained to predict positively for temporally nearby images and negatively for temporally far away images, its predictions of mediun-distance images can be arbitrary. This creates both false positives and false negatives, thereby increasing shortcuts and missing edges in the graph.
To solve these problems, propose to learn connectivity score using contrastive predictive loss [7]. Similar to CVAE, we initialize a CPC encoder Jenc that takes in both observation and context, and a density-ratio model f& that does not depend on the context. Through optimizing the CPC objective, f& of positive pairs are encouraged to be distinguishable from that of negative pairs. Thus, it serves proxy for the temporal distance between two observations, leading to a connectivity score for planning. Theoretically, CPC loss is better motivated than the classification loss in SPTM
Thhe original CPC model has an additional autoregressive memory variable 17 We drop it in our formu-
lation as our domains are fully observable and do not require memory.
4

---

Under review as a conference paper at ICLR 2020
as it structures the latent space clear objective: maximize the mutual information between current and future observations. In practice, this results in less hyperparameter tuning and a smoother distance manifold in the representation space. Finally, instead of only sampling from the same trajectory as done in SPTM, our negative data are collected by sampling from the latent space of trained CVAE the replay buffer. Without this trick, we found that the SPTM classifier fails to handle self-supervised data.
3.3 PLANNING AS INFERENCE
Planning Algorithm. Given a start observation Ostart, goal observation Ogoat sampled from a potentially new environment E", and the context vector c, we propose 4-step planning algorithm. First, we hallucinate exploration data by sampling from the latent space Po(-le) of the CVAE. Sec- ond, we build a fully-connected weighted graph G(V, E) by forming connections between all i generated image samples o; = p 24 c), where pe is the trained CVAE decoder and 2 is the vector sampled from the CVAE prior. We choose our edge weight between nodes i and j as choices: (1) an energy model, i.e., the inverse of [x. 2, or (2) a density ratio, i.e., the inverse of normalized fk between two nodes, ie. fx( (fa, This score reflects the difficulty in transitioning to the next state from the current state by self-supervised exploration. The learned connectivity graph G can be viewed graph planning methods to efficiently perform visual planning. In the third step, we find the short- est path using Dijkstra's 's algorithm on the learned connectivity graph G between the start and end node. In the fourth step, we apply our policy to follow the visual plan, reaching the next node in our shortest path and replan every fixed number of steps until we reach 0gont. For the policy, we train an inverse model which predicts actions given two observations that are within h steps apart.
of two
topological memory upon which we can use conventional
Maximum likelihood trajectory with Dijkstra's. We show that the CPC loss can be utilized to inference problem, and results in effective planning algorithm.
cast the planning problem After training the CPC objective to convergence, we have fa (Or+&, or) x p(or+4/0:)/pp) (0(+k)))))))))))))))))) [17]. compute the normalizing factor Eo'ev [f oz)] for each or by averaging over all nodes in the graph. Let's define our non-negative weight from O to O1t as w( (0 o,) Evev [f (o', o.)) (o.4s, o.) p(o:+*)/p((t+xll)
To estimate p(or+slo;)) ) ///(pOoo=z x(0++)),
A shortest-path planning algorithm finds oo...... Or that minimizes Et=o w(or, 0c+1) such that 00 Ostart, OT Ogoat- By Jensen's inequality and the Markovian property of loy.-000 Or we have that, log T
00,-,, 1 Et= w(o,, 0++1) 2 1 Ei=o logw(ot; 0++1) = - Σ ((oo)(()++)) - T
log p(0)+110.)) = + Er p(o,) log p(o,, -.. 0T 0r-1 ing that the self-supervised data distribution is approximately uniform, the shortest path algorithm with proposed weight w maximizes lower bound on the trajectory likelihood given the start and goal states. In practice, this leads to more stable planning approach and yields more feasible plans.
Ostart: OT @goai), Thus, assum-
4 RELATED WORK
Reinforcement Learning. Most of the study of data-driven planning has been under the model-free
RL framework 23 15 24). However, the need design a reward function, and the fact that the
learned policy does not generalize to tasks that are not defined by the specific reward, has motivated the study of model-based approaches. Recently,
investigated model-based RL from pixels on
Mujoco and Atari domains, but did not study generalization to a new environment. 6 4 explored model- based RL with image-based goals using visual model predictive control (visual MPC). These methods rely on video prediction, and are limited in the planning horizon due to accumulating errors. In comparison, our method does not predict full trajectories but only individual images, mitigating this problem. Our method can also use visual MPC replacement for the visual servoing policy.
Self-supervised learning. Several studies investigated planning goal directed behavior from data obtained offline, e.g., by self-supervised robot interaction [| Nair et al. [16] used an inverse model to reach local sub-goals, but require human demonstrations of horizoorizon plans. Wang et al. 29 solve the visual planning problem using a conditional version of Causal InfoGAN [13]. However, as training GAN is unstable and requires tedious model selection (2], we opted for the CVAE-baaed approach, which is much more robust.
5

---

Under review as a conference paper at ICLR 2020
Classical planning and representation learning. In classical planning literature. task and motion
planning also separates the high-level planning and the low-level controller [30 (7 10]. In these
works, domain knowledge is required to specify preconditions and effects at the task level. Our approach only requires data collected through self-supervised interaction.
Other studies that bridge between elassical planning and representation learning include 13
These works, however, do not consider zero-shot generalization. While Srinivas et al. 226 and Qureshi al. [9 learn representations that allow goal-directed planning to unseen environments, they require expert training trajectories. Ichter and Pavone [8] also generalizes motion planning to new environments, but require a collision checker and valid samples from test environments.
5 EXPERIMENTS
29 focused real robotic tasks with visual input. While impressive, such results can be difficult to reproduce or compare. For example, is not clear whether manipulating a rope with the PR2 robot [29] is more or less difficult than manipulating a rigid object among many visual distractors [4]. In light of this difficulty, we propose a suite of simulated tasks with an explicit difficulty scale and clear evaluation metrics. Our domains consider moving a rigid object between obstacles using Mujoco (28), and by varying the obstacle positions, we can control the planning difficulty. For example, placing the object in a cul-de-sac would require non-trivial planning compared to simply moving around an obstacle along the way to the goal. We thus create two domains, as seen Figure
Recent work in visual planning (e.g., 13
Block wall:: A green block navigates around static red obstacle, which can vary in position.
2. Block wall with complex obstacle: Similar the above. but here the wall is 3-link object which can vary in position, joint angles, and length, making the task significantly harder.
Figure 2: Block wall domain (left) and block wall with complex obstacle (right) domain. The top row shows some example contexts. The bottom rows show example observations.
With these domains, aim to asses the following attributes:
Does HTM improve visual plan quality over state-of-the-arr VP methods
22
How does HTM execution success rate compare state-of-the-art VP methods?
How well does HTM generalize its planning unseen contexts?
We discuss evaluation metrics for these attributes in Section To fully assess success of HTM relative to other state-of-the-art VP methods, we run these evaluation metrics on SPTM [22 and Visual Foresight [4] In the first baseline, since vanilla SPTM cannot plan new environment, we use the same samples generated by the same CVAE as HTM, and then build the graph by assigning edge weights in the graph proportional to their exponentiated SPTM classifier score. We also give it the same negative sampling proceedure HTM. The same low-level controller also used to follow the plans. the second baseline, Visual Foresight trains a video prediction model, and then performs model predictive control (MPC) which finds the optimal action sequence through random shooting. For the random shooting, we used iterations of the cross-entropy method with 200 sample sequences. The MPC acts for 10 steps and replans, where the plamning horizon T is 15. We use the state-of-the-art video predictor as proposed by Lee et al. [14] and the public code provided by the authors. For evaluating trajectories in random shooting, studied cost functions that suitable for our domains: pixel MSE loss and green pixel distance. The pixel MSE loss computes the
We found that negative exponential weighting, which requires tuning, performed slightly better in results than best tuned version of the original SPTM edge weighting through thresholding.

---

Under review as a conference paper at ICLR 2020
pixel distance between the predicted observations and the goal image. This provides sparse signal when the object pixels in the plan can overlap with those of the goal. We also investigate a cost function that uses prior knowledge about the task - the position of the moving green block, which is approximated by calculating the center of mass of the green pixels. As opposed to pixel MSE, the green pixel distance provides smooth cost function which estimates the normalized distance between the estimated block positions of the predicted observations and the goal image. Note that this assumes additional domain knowledge compared to HTM.
5.1 EVALUATION METRICS
We design a set of tests that measure both qualitative and quantitative performance of an algorithm. To motivate the need for qualitative metrics, we reiterate the importance of planning interpretability; it is highly desirable that the generated plan visually make sense so as to allow a human to approve of the plan prior to execution.
Qualitative Visual plans have the essential property of being intuitive, in that the imagined trajectory perceptually sensible. Since these qualities are highly subjective, we devised set of tests to evaluate plans based on human visual perception. For each domain, we asked 5 participants to visually score 5 randomly generated plans from each model by answering the following questions: (1) Fidelity: Does the pixel quality of the images resemble the training data?; (2) Feasibility: Is each transition in the generated plan executable by a single action step?; and (3) Compleieness: Is the goal reachable from the last image in the plan using single action? Answers were in the range [0,1], where 0 denotes No to the proposed question and means Yes. The mcan opinion score were calculated for each model.
Quantitative In addition to generating visually sensible trajectories, a planning algorithm must also be able successfully navigate towards predefined goal. Thus, for each domain, we selected 20 start and goal images, each with an obstacle configuration unseen during training. Success was measured by the ability get within some L2 distance to the goal in a n steps less, where the distance threshold and n varied on the domain but was held constant across all models. A controller specified by the algorithm executed actions given an imagined trajectory, and replanning occurred every r steps. Specific details can be found in the Appendix D
5.2 RESULTS
As shown in Table 5.2 HTM outperforms all baselines in both qualitative and quantitative mea- surements across all domains. In the simpler block wall domain, Visual Foresight with green pixel distance only succeeds under the assumption of additional state information of the object's location. the other algorithms do not have. However, in the complex obstacle domain, Visual Foresight fails to perform comparably to our algorithm, regardless of the additional assumption. We also com- pared our method with SPTM, using the same inverse model and CVAE to imagine testing samples. However, without a robust classification loss and improved method of weighting the graph's edges, SPTM often fails to find meaningful transitions.
In regards to perceptual evaluation, Visual Foresight generates realistic transitions, as seen by the high participant scores for feasibility. However, the algorithm is limited in creating visual plan within the optimal = 15 timesteps. Thus, when confronted with a challenging task of navigating around convex shape where the number of timesteps required exceeds T, Visual Foresight fails to construct a reliable plan (see Figure and thus lacks plan completeness. Conversely, SPTM is able to imagine some trajectory that will reach the goal state. However, as mentioned above and was confirmed in the perceptual scores, SPTM fails to select feasible transitions, such as imagining trajectory where the block will jump across the wall split into two blocks. Our approach, on the other hand, received the highest scores of fidelity, feasibility, and completeness. Finally, we show in Figure 3 the results of our two proposed improvements to SPTM in isolation. The results clearly show that a classifier using contrastive loss outperforms that which uses Binary Cross Entropy (BCE) loss, and furthermore that the inverse of the score function for edge weighting is more successful than the best tuned version of binary edge weights.
*For plans require > T steps, found that emror across the image translations accumulate and the predicted image drastically decreases in interpretability. This optimal value of T is consistent with that of (4)
7

---

Under review as a conference paper at ICLR 2020
Algorithms
Domain
Fidelity
Feasibility
Completeness
Execution Success
0.86 .05
0.84
1.00 + .00
100%
HTM (1)
2
0.95 + .03
0.92 = .11
1.00 = .00
95 %
0.75 + .09
0.88 :.14
1.00 + .00
95 %
HTM (2)
2
0.96 .03
0.96 .08
0.96 = .08
100 %
1
0.40 + .11
0.00 = .00
1.00 = .00
55%
SPTM with CVAE
2
0.92 + .07
0.00 = .00
1.00 = .00
30%
Visual Foresight 4
1
0.74 - .08
0.84 = .16
0.04 +. 08
25%
(pixel MSE loss)
2
0.59 + .16
0.64 + 21
0.00 + .00
0%
Visual Foresight [4]
0.80 .07
0.84 .16
0.04 .08
90%
(green pixel distance)
2
0.69 + 14
0.56 + 21
0.00 + .00
35%
20%
Inverse Model
2
25%
Table 1: Qualitative and quantitative evaluation for the the block wall (1) and block wall with com- plex obstacle (2) domains. Qualitative data also displays the 95% confidence interval. Note HTM (I) refers to edge weighting using the energy model, and (2) is weighting using the density ratio, as described in 3.3
Start
Goal
2.00
SPTM
1.75
1.25
Start
Goal
0.75
0.50
0.00
Binary
Figure 3: Left: HTM plan examples (top) and Visual Foresight (green pixel distance) plan examples (bottom). Note Visual Foresight is unable to conduct long-horizon plan, and thus greedily moves in the direction of the goal state. Right: Comparison of score function and edge weighting function by examining final average distance to the goal state for 10 test start/goal pairs (the lower the better). For the score function, we denote the energy model structured with contrastive loss CPC and the classifier as proposed in [22 with BCE loss as SPTM. For the edge weighting function, we test the binary edge weighting from the original SPTM paper, the inverse of the score function, and the inverse of the normalized score function.
DISCUSSION
We propose a method that is visually interpretable and modular - we first hallucinate possible con- figurations, then compute connectivity between them, and then plan. Our HTT generalize to unseen environments and improve visual plan quality and execution success rate over state-of-the-art VP methods. Our results suggest that combining classical planning methods with data-driven per- ception can be helpful for long-horizon visual planning problems, and takes another step in bridging the gap between learning and planning. In future work. we plan combine HTM with Visual MPC for handling more complex objects, and use object-oriented planning for handling multiple objects. Another interesting aspect is improve planning by hallucinating samples conditioned on the start and goal configurations, which can help reduce the search space during planning.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
[1] Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to Experiential learning intuitive physics. In Advances Neural Information Processing Systems, pages 5074-5082, 2016.
poke by poking:
[2] Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. arXiv preprint arXiv:1902.08893,, 2019.
[3] Masataro Asai and Alex Fukunaga. Classical planning in deep latent space: Bridging the subsymbolic-symbolii boundary. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[4] Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv:1812.005500 2018.
[5] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. arXiv preprint vrXiv:1906.05253, 2019.
[6] Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2786-2793. IEEE, 2017.
[7] Tuomas Haarojj,, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05905, 2018.
[8] Brian Ichter and Marco Pavone. Robot motion planning in learned latent spaces. IEFE Robotics and Automation Letters, 4(3):2407-2414, 2019.
[9] Leslie Pack Kaelbling and Tomás Lozano-Pérez. Hierarchical planning in the now. In Work- shops at the Twenty-Forthh AAAI Conference on Artificial Intelligence, 2010.
[10] Leslie Pack Kaelbling and Tomás Lozano-Péres Integrated task and motion planning in belief space. The International Journal of Robotics Research, 32(9- 10): 1227,22, 2013.
[11] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Camphell, Kon- rad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model- based reinforcement learning for atari. arXiv preprint arXiv: 2019.
[12] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1312v:13 14, 2013.
[13] Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. Advances Neural Information Process- ing Systems, pages 8733-8744, 2018.
[14] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804 01523, 2018.
[15] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[16] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope ma- nipulation. In Robotics and Automation (ICRA), 2017 IEEE International Conference on, pages 2154611...... IEEE, 2017.
[17] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807 007.03748, 2018.

---

Under review as a conference paper at ICLR 2020
[18] Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE interattonnal conference on robotics and automation (ICRA), pages -406-3413. IEEE, 2016.
[19] Ahmed H Qureshi, Anthony Simeonov, Mayur Bency, and Michael C Yip. Motion planning networks. In 2019 International Conference on Robotics and Automation (ICRA), pages 1- 2124. IEEE, 2019.
[20] Stuart J. Russell and Peter Norvig. Artificial Intelligence A Modern Approach (3. internat. ed.). Pearson Education, 2010.
[21] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234 2242, 2016.
[22] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological mem- ory for navigation. arXiv preprint 1xXiv:8803.00655. 2018.
[23] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-
1897, 2015.
[24] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam. Mare Lanc- al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
[25] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Advances in neural information processing systems, pages 3483-3491 2015.
[26] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal planning networks. arXiv preprint 1804 2018.
[27] Siddharth Srivastava, Eugene Fang, Lorenzo Riano, Rohan Chitnis, Stuart Russell, and Pieter Abbeel. Combined task and motion planning through an extensible planner-independent inter- face layer. In 2014 IEEE international conference on robotics and automation (ICRA), pages 639-646, IEEE, 2014.
[28] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033, 5033. IEEE, 2012.
[29] Angelina Wang, Thanard Kurutach, Kara Liu, Pieter Abbeel, and Aviv Tamar. Learning robotic manipulation through visual planning and acting. arXiv preprint arXiv:1905. 19050::190. 2019.
[30] Jason Wolfe, Bhaskara Marthi, and Stuart Russell. Combined task and motion planning for mo- bile manipulation. In Twentieth International Conference on Automated Planning and Schedul- ing, 2010.
10

---

Under review as a conference paper at ICLR 2020
A DISCRIMINATIVE MODELS: CLASSIFIER VS. ENERGY MODEL
In this section, we assume the dataset as described in VPA, D = -......, 1i There are two ways of learning model to distinguish the positive from the negative transitions.
Classifier: As noted above, SPTM first trains a classifier which distinguishes between an image pair that is within I steps apart, and the images that far apart using random sampling. The classifier is used to localize the current image and find possible next images for planning. In essence. the classifier contains the encoder ge that embeds the observation and the the score function that takes the embedding of each image and output the logit for sigmoid function. The binary cross entropy loss of the classifier can be written follows:
log f(2, Z(4k) + loog
L9 (D
E
+ (21,2(44))
+ f(zz 26
(ee, 21th)
Σ
log
(Z, 2c+k) + (e 2t + (1 FAa)) fe 24
(B1(Rt41)~D
where 2; is random sample from D.
Energy model: Another form of discriminating the the positive transition out of negative transitions through an energy model. Oord al. [17 learn the embeddings of the current states that are predictive of the future states. Let g be an encoder of the input æ and x ge (z) be the embedding. The loss function can be described as cross entropy loss of predicting the correct sample from N + 1 samples which contain positive sample and N negative samples:
folz,, 21+R)
LCPo(0, (D) --
Σ log (NDD
(6(2,,2488) + E. fe
where [u, v) exp (u² vbr) and 2t 2
the random samples from D.
Note that when the number of negative samples the loss function resembles the SPTM.
B MUTUAL INFORMATION (MI)
This quantity measures how much knowing one variable reduces the uncertainty of the other vari- able. More precisely, the mutual information between random variables X and Y can be de- scribed as 7(X, Y) = H(X) H(XIY) H(Y) H() = Ex,y PX.Y
PXPY
C ADDITIONAL PROOF
Lemma C.I. For any random variable X,Y and a deterministic function g. I(X,Y) >
o 1(X;1) H(X) H((YY) ((X) H(XIY, z H(X) - H(X)g(Y))
F5
I(X)9(Y))
D ADDITIONAL RESULTS AND HYPERPARAMETERS
Thiis also known the data processing inequality.

---

Under review as a conference paper at ICLR 2020
Start
Goal
Figure 4: HTM plan examples. The hallucination allows the planner to imagine how to go around the wall even though it has not seen the context before.
God
Figure 5: Visual Foresight plan examples. The plans do not completely show the trajectory to the goal.
Table 2: Data parameters.
Domain
Domain 2
contexts
150
400
initializations per context
50
30
trajectory length
20
100
action space
U|-.05, 05, 05
U[-.1,
Table 3: Planning hyperparameters.
Domain
Domain 2
no. of samples from CVAE
300
500
L2 threshold for success n (timesteps to get to goal) r (timesteps until replanning)
.75
500
400
200
80
5
Figure 6: Sample observations (top) and contexts (bottom). In this domain, object can be trans- lated and rotated (SE(2)) slightly per timestep. The data are collected from 360 different object shapes with different number of building blocks between 3 to 7. Each object is randomly initialized 50 times and each episode has length 30. The goal is to plan a manipulation of an unseen object through the narrow gap between obstacles in zero-shot.
12

---

Under review as a conference paper at ICLR 2020
Figure 7: HTM evaluated on real data. Ground truth start and goal are the leftmost and rightmost images, respectively, in the row.
Figure 8: SPTM evaluated on real data. Ground truth start and goal are the leftmost and rightmost images, respectively, in the row.
Figure 9: HTM evaluated hallucinated data. Ground truth start and goal the leftmost and rightmost images, respectively, in the row.
13


Reviews:
[{"review_id": "Syx34y_sYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a novel visual planning approach which constructs explicit plans from \"hallucinated\" states of the environment. To hallucinate states, it uses a Conditional Variational Autoencoder (which is conditioned on a context image of the domain). To plan, it trains a Contrastive Predictive Coding (CPC) model for judging similarities between states, then applies this model to hallucinated states + start/end states, then runs Dijkstra on the edges weighted by similarities.\n\nI vote for accepting this paper as it tackles two important problems: where to get subgoals for visual planning and what similarity function to use for zero-shot planning. Furthermore, the paper is clearly written, the experiments are well-conducted and analyzed.\n\nDetailed arguments:\n1. Where to get subgoals for visual planning is an important question persistently arising in control tasks. SPTM-style solution is indeed limited because it relies on an exploration sequence as a source of subgoals. Every time the environment changes, data would need to be re-collected. Getting subgoals from a conditional generative model is a neat solution.\n2. Benchmarking similarity functions is crucial. One productive way to approach zero-shot problems is to employ similarity functions, but the question arises: what algorithm to use for training them? The paper compares two popular choices: CPC and Temporal Distance Classification (in particular, R-network). It thus provides guidance that CPC might be a better algorithm for training similarity functions.\n3. The paper is well-positioned in the related work and points to the correct deficiencies of the existing methods. It also features nice experimental design with controlled complexity of the tasks, ablation studies and two relevant baselines.\n\nI would encourage the authors to discuss the following questions:\n1) Fidelity in Table 3 - why is it lower for SPTM compared to HTM if both methods rely on the same generated samples? Is it because HTM selects betters samples than SPTM for its plans?\n2) Why is fidelity larger for SPTM in a more complex task 2?\n3) Same question about fidelity/feasibility for HTM1/2?\n4) Are there any plans to open-source the code?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkgfxqIAtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents HTM, an extension of the semiparametric topological memory method that augments the approach with hallucinated nodes and an energy cost function. The hallucination is enabled by a CVAE, conditioned on an image of the environment, and allows the method to generalize to unseen environments. The energy cost function is trained as a contrastive loss and acts as a robustness score for connecting the two samples. The underlying graph is then used to plan for several top view planning problems.\n\nThe paper is well written and clear. I believe such latent representations are an interesting approach to solving visual navigation and general planning. HTM provides an interesting and useful extension to SPTM, allowing both generalization to unseen environments and a more robust loss function. The \n\nMy primary concern is the lack of rigorous experimentation to validate the concept and push it’s limits. The results in Table 1 show HTM outperforms baselines clearly on the given problems, but how it performs on more complex problems is unclear. These problems are dynamically simple and the obstacles are easily identified. Some more difficult problems may be:\n- The mazes in SPTM or environments from https://arxiv.org/pdf/1612.03801.pdf.\n- The original SPTM paper focuses on visual navigation from first person views. How does this method apply to such situations? How does the context translate to this scenario?\n- Planning in real environments with real images, as done in [6].\n\nOther comparisons and notes:\n- Can the method be applied to higher dimensional problems (dimensionality of the underlying space) where planning may be more difficult? E.g. SE(2), robot arms or other agents from UPN [26]. Application with actual 3D workspace problems too would be interesting as the image context may underspecify the environment.\n- The energy cost function acts as a proxy for connection probability when traversing an edge. This may also be useful for dynamical systems (e.g., the mujoco ant navigating a maze). Are there limitations for the method on such problems, e.g., edges may no longer be symmetric?\n- How does SPTM compare when the space has been explored already?\n- Can more quantitative results been shown such as solution path cost?\n- Provide definitions for Fidelity, feasibility, and completeness and the source of data (polling human’s) in the Table 1 caption.\n- “As shown in Table 5.2, “, should be renamed to Table 1.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkePIyMquB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for learning agents to solve visual planning, in particular to navigate to a desired goal position in a maze, with a learned topological map, i.e. a graph, where nodes correspond to positions in the maze and edges correspond accessibility (reachability in a certain number of steps). The work extends previous work (semi parametric topological memory, ref. [22]) in several ways. It claims to address a shortcoming of [22], namely the fact that the graph is calculated offline from random rollouts, by using a conditional variational auto-encoder to predict a set of observed images which could lie between the current position and the goal position, and, most importantly from a context image which describes the layout of the environment. These predicted images are then arranged in a graph through a connectivity predictor, which is trained from rollouts through a contrastive loss. Training is performed on multiple environments, and the context vector provides enough information for this connectivity network to generalize to unseen environments.  At test time, the agent navigates using a planner and a policy. The planner calculates the shortest path on a graph where edges are connectivity probabilities, and the policy is an inverse model trained on the output of the planner.\n\nWhile the idea of a topological memory with dynamic graph creation is certainly interesting, the work is unfortunately not well enough executed and the paper structured written in a way which makes it up to impossible to grasp what has been really done, as much information is missing which would be required for understanding. \n\nAs a first example, we are never really told what the observations are, which the agent sees. The different figures of the paper show very small images with a 2D maze from a bird’s eye view consisting of a walls arranged in a single connected component (mostly 1 to 3 strokes) in red color and an agent shown as a position indicated as a green dot. Are these the observed images? In absence of any other information, this is what we need to assume, and then this problem is fully observable and does not seem to be very challenging. Given the figures, even a handcrafted algorithm should be able to calculate the optimal solution with Dijkstra’s algorithm on a graph calculated from the pixel grid.\n\nThis important missing information alone makes it difficult to assess the paper, but the rest of the writing is similarly confusing. The authors focus on very short and dense descriptions of mathematical properties, but seem to have forgotten to ground the different symbols and to connect them to physical entities of the problem. The technical writing is in large parts disconnected from the problem which is addressed by it.\n\nFurther examples are:\n\n-\t“the data (…) is collected in a self-supervised manner”: what does this mean? Self-supervision is way of creating loss from data without labels, but I am not sure what is meant by collecting data this way.\n-\tThe paragraph on CPC in section 2 can only be understood if the contrastive loss is known. To make the paper self-consistent, this should be properly explained, and tied to a training procedure which details how exactly the positive and negative samples are defined … and collected.\n-\tThe CPC objective in section 2 is only loosely connected to its usage in section 3.2. Barely writing “we optimize a CPC” objective is not sufficient for understanding how this objective is really tied to the different entities of the problem. This paper contains maths (which is always a pleasure to read), but it is not a purely and abstract mathematical problem - a real task is addressed, so it needs to be connected to it. This connection has certainly been done by the authors while they were working on the problem, but they should also communicate it to the reader.\n-\tThe section on ML trajectory is too dense and should be rewritten. I don’t understand what the authors want to tell us here. Basically, a (generalized) Dijkstra is run on a graph, where edge weights are the density or density ratios learned by the CPC objective, and if the edge weights are probabilities, that the shortest path corresponds to a trajectory likelihood. This is known, and this information is buried in a dense set of equations which are difficult to decipher and do not add any further value to the paper.\n-\tThe connection between the planner (generalized Dijkstra) and the policy is never explained. We don’t know how the policy is trained and how it works.\n\nOne of the downsides of the method is that it requires a context image. This image is responsible for the generalization to unseen environments, but it is a major drawback, as the image must be created beforehand. The authors claim that the context image must only contain the layout in any format which makes it possible to extract information about navigational space from it, but in the experiments the context image corresponds to the full map – and it is probably equivalent to the observed images, but we can’t be sure as we haven’t been told. In any case, it is far from sure how this could generalize to more complex environments, let alone 3D navigation as is currently addressed in standard simulators like VizDoom, GIBSON, Matterport, Deepmind Lab, Habitat AI etc. \n\nThe authors’ claim that the proposed environment requires long-term planning, but looking at the images this does not seem to be the case. \n\nThe paper claims to perform zero-shot generalization and to adapt to changes in the environment, like the slight changes in camera motion, variations in lightning, but it unclear how the solution solves this claim.\n\nHow does the agent determine that a goal has been reached, without ground truth information? \n\nWhat happens, if the hallucinated images are disconnected (form several connected components) or are disconnected from the current position and/or from the goal position?\n\nAs mentioned, the method is evaluated on an environment, which is too simple. The experiments are difficult to assess, as we don’t really know what the agent observes. An information asymmetry is mentioned (visual foresight having the object’s (=agent’s) position and the others not) … but if the proposed method observes the bird’s eye view, it can infer the agent’s position (as the position of the green dot).\n\nSubjective evaluation by humans on this kind of simple data does not seem to be meaningful, in particular with a very low number of observers (5 people).\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to visual planning. The work builds on semi-parametric topological memory (SPTM) and introduces ideas that facilitate zero-shot generalization to new environments. The reviews are split. While the ideas are generally perceived as interesting, there are significant concerns about presentation and experimental evaluation. In particular, the work is evaluated in extremely simple environments and scenarios that do not match the experimental settings of other comparable works in this area. The paper was discussed and all reviewers expressed their views following the authors' responses and revision. In particular, R1 posted a detailed justification of their recommendation to reject the paper. The AC agrees that the paper is not ready for publication in a first-tier venue. The AC recommends that the authors seriously consider R1's recommendations.
Decision: Reject