Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents ELAST, an approach for long-horizon planning through sampling-based motion planning in learned latent spaces. Given a high dimensional observation (images), the approach first learns a latent representation using self-supervised contrastive learning, a latent dynamics model, and a transition density function. This latent space is then searched via expansive space trees to solve motion planning problems from initial states to goal states. Finally, plans are executed via MPC. ELAST allows efficient searching of spaces and long-horizon planning as shown over a number of environments.

strengths_and_weaknesses: The paper is generally well written and clear, as are the figures demonstrating the approach. The area of learning latent representations is useful and important. The minimal assumptions and requirements on data for ELAST makes the method generally applicable. Based on the planning results and plots of the latent spaces the latent space learned is generally robust. The MPC replanner seems to be effective for following the trajectories.

In terms of number of environments and comparisons the experiments section is thorough (though see notes below on complexity of the environments). The detailed ablations in the appendix and ablations in the experiments section are well done.

The two main weaknesses I see are (1) the complexity of the environments (both in terms robot dimensionality and environment variation) and (2) the novelty compared to prior approaches for latent tree searches.

(1) Each of the environments has relatively low dimensional true, important state spaces, which is an area where these latent space methods perform well. It would be useful to see how it performs in spaces such as an SE(2 or 3) rigid body in a complex environment. Furthermore it would be useful to see how it performs with more complex dynamics, such as a kinodynamic system which may force higher dimensional learned latent spaces (a double integrator for instance). 

Finally, as a benefit over prior methods is that it does not require a collision checker (which allows some degree of generalization to new environments) it is imperative the authors demonstrate the approach on more complex obstacle environments, which can be encoded via the context. I’m not sure from the results shown that it would perform well in such cases. In the current implementation, the conditioning on the actual states of parameterization (such as the position of the moved block) does not seem consistent with the rest of the paper: why not condition on the image alone? A 2D environment with significantly more variations and objects than 5c may suffice to show that it can work in such a more complex setting, though scaling to 3d scenes would be more informative. 

(2) The novelty compared to prior works using tree-search in latent space is potentially limited. For example, [15] does use a sampler and a learned collision checker as noted, but removing them essentially reduces that algorithm to ELAST. The state sampler is only used to bias the search and make it more efficient, but also state sampling is likely freely available from your training data. It is not clear to me why one wouldn’t use it. The collision checker is used to allow generalization to novel environments, which in ELAST is handled by the context c. It is not clear how powerfully the context can be used to extend to more complex environments and should be shown by this work. 

One major change between 15 and this work is that the latent space is learned in a potentially more robust way through contrastive learning. It would be useful to investigate the effect this has by comparing to VAE-based latent spaces with enforced dynamics.

A few more minor weaknesses:
- An algorithm box for the tree search and the latent space learning would be appreciated. 
- Adding the network names to the figures in e.g. Fig 3 and 4 would be helpful to quickly see how each fits together.

Minor notes:
- Line 198: list the inptuts and outputs of h_f.
- ELAST + RL is an interesting extension
- Appendix D1 is interesting and would appreciate more detailed

questions: 
See above in strengths and weaknesses as well
- What is the computation time?
- I think an ablation to VAE-based latent spaces with enforced dynamics is missing.
- How robust is the latent space? It would be good to see N random training seeds in the Appendix shown with isomaps. Similarly this would be informative for VAE-based latent spaces and VAE-based spaces with enforced dynamics.
- Why does 19c/d form a loop in the isomap?

limitations: I would appreciate greater discussion of the limitations of ELAST, right now it's focused more on future work and how previous works can be incorporated. A few limitations of at least the current setup I see is it is not clear how robust the approach is to more complex environments (e.g., with more obstacles) and higher dimensional problems (in the true low-level state space or in the controls space). It would also be good to understand how it scales to kinodynamic systems. I feel the computation time may too be a limitation depending on the results.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new algorithm for planning and control for long-horizon goal-reaching tasks in high-dimensional spaces (e.g., control from vision). To do so, the proposed approach performs sampling-based planning in a lower-dimensional latent space. This latent space and its associated transition model are learned using contrastive learning. With these two building blocks, the proposed approach recursively plans a sequence of latent states that are tracked using a learned control policy. The approach outperforms existing baselines in a number of simulation experiments.
strengths_and_weaknesses: This paper makes an interesting contribution for long-horizon planning in high-dimensional spaces, as it combines (long-horizon) sampling-based planning techniques from the robotics community with learned representations from the machine learning community to enable planning in a lower-dimensional latent space. The approach is original and clearly presented, and, in general, the paper is well-written.

** Strengths **
- The learned latent space and density transition model alleviate the need to learn a collision detector, distance metric, and sampling distribution for sampling-based planning.
- By planning and learning a transition model in a latent space, the approach does not require learning high-dimensional transition models (e.g., video generative models), which can be computationally expensive to train and evaluate.

** Weaknesses **:
- The paper does not report computation times. As such, it is unclear if the approach runs in real-time and can be applied to real-world robotic problems, or if it only works in simulation. In particular, more details should be provided regarding scalability of the method with respect to the dimension of the latent space, computation times for the MPC controller, etc.
questions: - Computation times: how long does it take for the approach to compute a plan? How does it compare with baselines? What is the computational bottleneck in the proposed approach? How does it depend on the size of the learned latent space representation?

- Comparisons and feedback control: the role of feedback (from replanning with MPC and tracking the plans with the local policy $\pi$) should be clarified in the comparisons. The following important details are not described in the body of the paper or in the appendix, which hinders reproducibility and interpretability of the results:
--- Is PlaNet [12] implemented with online replanning and/or local feedback or not? Is a new trajectory computed at each iteration (as in the original paper) or every 3-25 time steps as in the CPC baselines?
--- Is HTM [29] implemented with the same local policy $\pi$ as ELAST? Is replanning done as in ELAST for a fair comparison, or not?

- The paper mentions the difficulty of planning through narrow passages in the Hammer environment. Is the corresponding region in the learned latent space also narrow or not necessarily? Indeed, a good learned latent representation for sampling-based planning would assign larger volume to feasible "narrow" passages. Does the proposed approach yield this property or not necessarily? Can the authors add the learned latent space for failure cases in the Hammer environment in Appendix D.8?

Suggestions:
- The discussion in the paragraph: "Does the rewiring affect the quality of generated paths?" is not surprising and follows from results in the sampling-based planning literature (rewiring is a standard step). I recommend making this paragraph shorter.
- The paper and appendix include typos and should be proof-read (e.g., previosuly, exisitng, distriubtion, This baselines applies, etc.).
limitations: Yes, the authors state the limitations of the approach (but, importantly, the paper misses a discussion about the computation times). A discussion of failure cases (e.g., only 22% success rate on the Hammer environment) is included.

If the authors provide a convincing discussion about the computation times for their proposed approach, I will be happy to raise my rating.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose ELAST, a method for model-based long-horizon planning in a low-dimensional latent space of partially-observable systems with high-dimensional observations $s_t$ (e.g. images).
The method can be separated in three conceptual stages.
- First, latent encodings $z_t$ are obtained for each observation $s_t$ (see below).
- Second, a transition model is fit to time-consecutive encodings to capture the system dynamics $p(z_{t+1} | z_t)$ (disregarding context $c$ for brevity here).
- Finally, the transition model is used for latent space planning towards a latent goal.
This is done by expanding a search tree and then using a heuristic to cull expanded nodes based on the transition model's density, such that the tree remains compliant with the learned dynamics (i.e. sampling-based motion planning, inspired by Rapidly-exploring Random Trees (RRT) and Expansive Space Trees (EST)).
The low-dimensional latent space lowers the search cost, which allows the authors to use long planning horizons.
The search produces waypoints for an MPC controller, which uses a local parametric policy to bridge them.

The method is validated over 10 different simulated environments with decent complexity and reasonable success rates.

In terms of model learning, the latent encodings are a byproduct of applying *Contrastive Predictive Coding* (CPC) [1], to estimate the density ratio $p(s_{t+k} | s_t) / p(s_{t + k})$ which implicitly captures the dependence between consecutive observations $s_t$.
The objective of CPC is called InfoNCE and is heavily inspired by *noise contrastive estimation* (NCE) [2] and avoids generative modelling of the observations $s_t$ alltogether.
The encodings alone are not enough for planning, as a latent transition model is also necessary.
The authors estimate the transition's conditional using NCE as well in a second step, based on the latent encodings alone.
There is some redundancy in modelling the transition (see my comments below).

**References**

[1] Oord, A.V.D., Li, Y. and Vinyals, O., 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.

[2] Gutmann, M. and Hyvärinen, A., 2010, March. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics (pp. 297-304). JMLR Workshop and Conference Proceedings.

strengths_and_weaknesses: **Strengths**
- The method seems widely applicable, as little domain knowledge is employed.
- Planning appears to be efficient, the authors consider long search horizons and up to 7500 tree expansion steps during planning.
- Using search for control, be it with tree heuristics like here or through optimal dynamic programming, is generally better than local optimization methods that are prone to getting stuck in local minima.
- The formulation appears novel to me.

**Weaknesses**
- The paper sets out to address planning for partially-observable systems, specifically where the state-space does not match the observations, which for me implies a Partially-Observable Markov Decision Process (POMDP).
However, the methods are geared towards solving an MDP, as lookahead control (MPC) is used and generative observation modelling is omitted alltogether.
I found no clarification of this aspect in the main text, and I believe this is a very important point in terms of positioning the method and making its limitations clear.
For example, an optimal POMDP controller would need to plan in *belief space*, and account for the effect of future observations on the state estimate [1].
State estimates should also generally be uncertain, to account for observation noise and modelling errors, and from what I can tell the obtained latent encodings are a deterministic mapping of the observations.
Could you please comment on how the method fits / differs from the POMDP paradigm? I believe this should be discussed in the manuscript.
- I can see how observation generation can burden model training, but why is that an issue during deployment for control (sec. 1, paragraph 2)? Assuming that we disregard the POMDP aspect (which is currently the case), control would only require the dynamics and rewards / the goal state.
- Contrastive estimation to avoid observation generation has been considered in Dreamer [2] before, where through a single objective dynamics are learned in parallel with shaping the latent space (I'm referring to the mutual-information NCE variant, not the ELBO). What is the justification of learning the transition in two stages (first the encodings through CPC, then the dynamics through NCE)?
- $h_f$ (the forward dynamics model) and $\psi$ (the transition density) seem redundant, and this redundancy raises questions. E.g. why is the transition density learned in a second stage (see comment about Dreamer above) and why isn't the forward dynamics model trained to capture the transition density already?
- It is argued that NCE is better for transition estimation because the noise model provides outliers (end of sec. 4.2). But the noise model should be as close to the data distribution as possible for unbiased NCE gradients--can you elaborate?
- Thresholding based on the transition density when rejecting samples from the search tree does not properly capture the probabilistic mass of the transition (sec. 4.3). Have you considered rejection sampling / importance sampling?
- The experimental section ablates the planner and shows it is a core requirement for success, but ablations of model learning (encoding, transition) would also be helpful to assess the method. E.g. what would happen if the planning scheme is used with dynamics learned through generative modelling, e.g. PlaNet or Dreamer? Currently one cannot tell if the contrastive modelling is a major factor in terms of success, or generative modelling could work just as well.

**Further Remarks**
- I believe the choice of the similarity used in CPC should be motivated further, with more details about why it enforces proximity preservation. This is not immediately clear from the ratio estimation of $p(s_{t+k} | s_t) / p(s_{t+k})$ alone.
- I did not find details about the training of the ensemble $H_f$, this ties into my comment about the redundant transition modelling.
- Sequences of observations are generally not guaranteed to be Markovian (sec. 4, first sentence).

Despite the above weaknesses, the method appears pragmatic and seems to work on the considered tasks.
I find that search in a low-dimensional latent state-space is generally a good idea, albeit driven by heuristics.
This makes me lean towards a slightly positive assessment.
However, I have reservations about the methodology and the authors' input would be appreciated.

**References**

[1] Bertsekas, D., 2012. Dynamic programming and optimal control: Volume I (Vol. 1). Athena scientific.

[2] Hafner, D., Lillicrap, T., Ba, J. and Norouzi, M., 2019. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.
questions: - Using RRT / EST trees for expansion leads to heuristics, e.g. the rewiring or the heuristic bias towards unexplored regions. Have you considered search that is better aligned with dynamic programming?

Also see my remarks above.

limitations: I don't see any major negative social impact. Some of the method's limitations are acknowledged in a dedicated section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method to perform planning in latent space for robotic applications. The latent space is created for visual observations. The planning procedure follows a methodology similar to sampling-based motion planning in robotics: sampling new configurations (here, in latent space), evaluation of the feasibility of transitioning, re-evaluating the connectivity of this new node to existing nodes. The method is demonstrated in several tasks in simulation, including non-rigid object manipulation, a domain outside of what classical sampling-based planners could solve.

strengths_and_weaknesses: - Significance: the method has the potential to open new avenues of research in planning. It tries to extend a well-studied solution from robotics (sampling-based planning) into domains where they couldn’t be applied, and also reducing the necessary information by integrating it with learning based methods.
- Quality: the paper is well written and easy to read. Minor: starting the paper with “Over the past decade, machine learning has significantly improved the state-of-the-art in image-based robotics and perception” but then citing a single paper feels a bit limited. Consider citing several seminal works, or a survey. Also, maybe there is a better citation than Minsky for the limitations of RL (credit assignment in sparse reward setup). Finally, maybe “unexplored” is better word than “undiscovered” in some places of the text? The planner “explores” these areas to check for feasibility. Also, it should be clear what “undiscovered/unexplored” means in this context. Are these areas where the latents are generated before observing images (extrapolation)?
One notable lack are theoretical warranties. Sampling-based motion planning is a field where these warranties were carefully studied. I wonder about the effect on completeness of sampling using a sampler based on an ensemble of observed dynamics. This seems to steer search towards observed transitions, which indicates that all paths need to be randomly explored at the beginning? Experiments on this would be necessary.
- Originality: the main idea (trajectory planning through sampling in the latent space) has been presented before, as the authors review. However, the presented method is original as it does not require a collision checker and can be used for environments with changes that can be parameterized.
- Clarity: the paper and the concepts are mostly clear. The section about NCE could be improved. For example, the equations are copied from the original NCE paper but the terms are not explained. Some sentences are a bit convoluted, with multiple unclear subordinates (e.g., L151). The use of random trajectories to train is first indirectly introduced in L191; this methodology for training should be explained before, maybe at the beginning of Section 4. In general, the use of demonstrations is not clear. Consider indicating with bold text the best solution in Table 1.

There is a couple of typos, e.g., l117, l207, l220, caption Fig.4,...
questions: - Why is the state a sequence of images? What are those sequences for the initial and final states (e.g., s_start and s_goal in Fig.2)?
- What is f in line 197? Please, explain.
- What is the distance metric for the neighbor search in Rewiring? L2 in latent space?
- As I understand it, in line L263 there is a “hidden” implicit assumption: the distance from the tree node in latent space to the goal can be computed in Euclidean space. Or in other words, there is a transformation from latent, to a state where the distance to goal state can be computed by Euclidean distance. Is this true? This seems to strongly restrict the application domains for this method. Also, the effect of this trick is not evaluated in the experiments, but I suspect it provides a large boost. Seems to be however strongly against the idea of planning in latent space? Also, the sentence is weirdly written. The appendix explains another use of Euclidean distance in sampling.
- The experiments are restricted to an MPC setup. Why? As the authors argue, previous approaches could suffer from local minima entrapment. One of the advantages (the main one?) of sampling-based planning techniques such as RRT is that they are able to find a full path to the goal. Why use it in an MPC methodology? Why not evaluate the execution of such a path. Are there issues with it?

limitations: I indicated some of my doubts about the limitations in the questions. I think some of those could be added to the (currently very brief) discussion of limitations in the paper.
review_title: null

Meta-review: metareview: This work proposes a method to first learn a lower dimensional embedding via contrastive learning, then learns a transition model and then utilizes a planner inspired from sampling-based motion planner literature to plan in this latent space from start to goal states. A model-predictive controller is harnessed to follow the desired trajectories in latent space. Overall, the work is well presented and has been well received by reviewers. It should inspire techniques in a lot of related areas like vision-language navigation. During author-reviewer discussion period there was rich interaction and various additional clarifications, and experiments were added by the authors. The authors are encouraged to incorporate them into camera-ready version and release reproducible source-code to accompany the paper.  
confidence: Certain
Decision: Accept