Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduced Trajectory Translation (TR2) framework that seeks to train low-level policies by translating an abstract trajectory into executable actions. TR2 is designed to solve tasks in  three steps:  1) build a paired abstract environment by simplifying geometry and physics, generating abstract trajectories, and solve the original task by an abstract-to-executable trajectory translator. TR2 borrowed the idea of seq2seq model in machine translation to overcome the none one-one mapping issue between abstract and executable trajectories, so that to enable low-level policy to follow the abstract trajectory. Experiments are conducted on various unseen tasks with different robot embodiments.
strength_and_weaknesses: Pros:
This paper is well motivated by solving one-shot task generalization. It propose to achieve the goal by decoupling plan generation and plan execution, such disentanglement is demonstrated useful in deployment. Borrowing the idea of seq2seq model in machine translation, TR2 solve the original tasks with abstract-to-executable trajectory translation. TR2 is also extensively evaluated on various settings and tasks.

Cons:
1) unclear of the rational of abstract-to-execution policy translation. The key technical novelty of this work is to formulate the one-shot task generalization problem as a abstract-to-execution policy translation problem. However, I am not clear about the definition of 'abstract (high-level) trajectory' and 'execution (low-level) policy'. Based on box pusher example, seems the abstract trajectory is some trajectories with none-defined (no explicit) actions or dynamics, while these terminologies are not well defined. Seeing from the pipeline of TR2, the utilization of the abstract trajectory is more like producing a context-level prompt to perform execution. However, when using prompt to perform generalization, the assumption is that there are underlying correlation between the prompt and the target generation sequences. However, based on the illustration of the abstract and execution trajectories, seems they are lie in different dynamics distribution. In this case, I am confused how trajectories under different policies can help for generalization. Unless it is only used for generalize perception space (assuming they are in the same environment).
2) maybe a incorrectness of causal Transformer utilization, and lack of implementation details in transformer (e.g. tokenization, positional embeddings, attention masking, etc.). As I cannot find implementation details of the transformer, I can only figure out from Figure2 and Sec. 3.3. From these, seems the abstract trajectories and execution trajectories are first encoded into a sequence of token embeddings by separate encoders (i.e. x1, x2, ... xn; xn+1, xn+2... xn+k), and then they are fed into a GPT-2 Transformer to perform auto-regressive prediction. However, one thing strange here: as GPT2 is a causal transformer with masked attention, so the input sequence (x1, x2. ... xn, xn+1, xn+2, ... xn+k) is computed with sequence dependencies, i.e. x2 attn (x1, x2), xn attn(x1, x2, .. xn-1), ... xn+k attn(x1, x2, ... xn+k-1).  On the contrary, the encoder takes sequences parallelly. Then it is wired to me, e.g. you are predicting x2 by seeing x1 and x2, however, the x1 and x2 token embedding already seeing the future. The utilization of the Transformer needs to be further clarified. 
3) lack of comparison with the other state-of-the-arts approaches. 
clarity,_quality,_novelty_and_reproducibility: This paper is well motivated by solving one-shot task generalization. It propose to achieve the goal by decoupling plan generation and plan execution, such disentanglement is demonstrated useful in deployment. Borrowing the idea of seq2seq model in machine translation, TR2 solve the original tasks with abstract-to-executable trajectory translation. TR2 is also extensively evaluated on various settings and tasks.

However, there are a couple of technical things need to be further clarified. 1) unclear of the rational of abstract-to-execution policy translation. The key technical novelty of this work is to formulate the one-shot task generalization problem as a abstract-to-execution policy translation problem. However, I am not clear about the definition of 'abstract (high-level) trajectory' and 'execution (low-level) policy'. Based on box pusher example, seems the abstract trajectory is some trajectories with none-defined (no explicit) actions or dynamics, while these terminologies are not well defined. Seeing from the pipeline of TR2, the utilization of the abstract trajectory is more like producing a context-level prompt to perform execution. However, when using prompt to perform generalization, the assumption is that there are underlying correlation between the prompt and the target generation sequences. However, based on the illustration of the abstract and execution trajectories, seems they are lie in different dynamics distribution. In this case, I am confused how trajectories under different policies can help for generalization. Unless it is only used for generalize perception space (assuming they are in the same environment).
2) maybe a incorrectness of causal Transformer utilization, and lack of implementation details in transformer (e.g. tokenization, positional embeddings, attention masking, etc.). As I cannot find implementation details of the transformer, I can only figure out from Figure2 and Sec. 3.3. From these, seems the abstract trajectories and execution trajectories are first encoded into a sequence of token embeddings by separate encoders (i.e. x1, x2, ... xn; xn+1, xn+2... xn+k), and then they are fed into a GPT-2 Transformer to perform auto-regressive prediction. However, one thing strange here: as GPT2 is a causal transformer with masked attention, so the input sequence (x1, x2. ... xn, xn+1, xn+2, ... xn+k) is computed with sequence dependencies, i.e. x2 attn (x1, x2), xn attn(x1, x2, .. xn-1), ... xn+k attn(x1, x2, ... xn+k-1).  On the contrary, the encoder takes sequences parallelly. Then it is wired to me, e.g. you are predicting x2 by seeing x1 and x2, however, the x1 and x2 token embedding already seeing the future. The utilization of the Transformer needs to be further clarified. 
3) lack of comparison with the other state-of-the-arts approaches. 
summary_of_the_review: This paper is well motivated by solving one-shot task generalization. It propose to achieve the goal by decoupling plan generation and plan execution, such disentanglement is demonstrated useful in deployment. Borrowing the idea of seq2seq model in machine translation, TR2 solve the original tasks with abstract-to-executable trajectory translation. TR2 is also extensively evaluated on various settings and tasks.
However, there are a couple of technical things need to be further clarified. 1) unclear of the rational of abstract-to-execution policy translation. The key technical novelty of this work is to formulate the one-shot task generalization problem as a abstract-to-execution policy translation problem. However, I am not clear about the definition of 'abstract (high-level) trajectory' and 'execution (low-level) policy'. Based on box pusher example, seems the abstract trajectory is some trajectories with none-defined (no explicit) actions or dynamics, while these terminologies are not well defined. Seeing from the pipeline of TR2, the utilization of the abstract trajectory is more like producing a context-level prompt to perform execution. However, when using prompt to perform generalization, the assumption is that there are underlying correlation between the prompt and the target generation sequences. However, based on the illustration of the abstract and execution trajectories, seems they are lie in different dynamics distribution. In this case, I am confused how trajectories under different policies can help for generalization. Unless it is only used for generalize perception space (assuming they are in the same environment).
2) maybe a incorrectness of causal Transformer utilization, and lack of implementation details in transformer (e.g. tokenization, positional embeddings, attention masking, etc.). As I cannot find implementation details of the transformer, I can only figure out from Figure2 and Sec. 3.3. From these, seems the abstract trajectories and execution trajectories are first encoded into a sequence of token embeddings by separate encoders (i.e. x1, x2, ... xn; xn+1, xn+2... xn+k), and then they are fed into a GPT-2 Transformer to perform auto-regressive prediction. However, one thing strange here: as GPT2 is a causal transformer with masked attention, so the input sequence (x1, x2. ... xn, xn+1, xn+2, ... xn+k) is computed with sequence dependencies, i.e. x2 attn (x1, x2), xn attn(x1, x2, .. xn-1), ... xn+k attn(x1, x2, ... xn+k-1).  On the contrary, the encoder takes sequences parallelly. Then it is wired to me, e.g. you are predicting x2 by seeing x1 and x2, however, the x1 and x2 token embedding already seeing the future. The utilization of the Transformer needs to be further clarified. 
This paper presents extensive evaluations and demo of deployments, I appreciated. While, it lacks of comparison with the other state-of-the-arts approaches. It is not convincing enough with such comparisons.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to tackle a long-horizon task by (1) designing a simplified version of the environment, (2) solving the task in the simplified environment, and (3) translating the solution trajectory in the simplified environment into the trajectory in the original environment. By decomposing high-level planning and low-level task completion, the proposed method could solve four long-horizon tasks.
strength_and_weaknesses: ### Strengths

* The proposed method can solve very long horizon tasks and demonstrate impressive results, including stacking 26 blocks out of 28 blocks in the real world.
* The experiments are exhaustive and the proposed method outperforms baselines in many experimental setups.
* The proposed method generalizes to unseen tasks.

### Weaknesses

* The proposed method requires a significant amount of domain knowledge about the "environment" and "task", including an abstract environment, a mapping from low-level states to high-level states, a distance function between two high-level states, and a high-level solution trajectory in the abstract environment. The reviewer is wondering whether these are practical assumptions to make. 
* Also, the proposed method requires almost similar efforts with providing sub-goals for the task directly in the original environments, which can make the problem much easier.
* The reward formulation considers the first similar state from the current state (or previous states) to measure the progress of the agent. This may not work when there are multiple similar states in the high-level trajectory, such as periodic tasks. This may require some temporal information in the state space, which makes the mapping to the abstracted environment infeasible. Discussion on this limitation would be appreciated.
* The training curves in Figure 16-20 show lower training success rates compared to the numbers presented in Table 1. This needs some explanation.
* Comparison to SILO on the main experiments would be appreciated with the properly tuned window size depending on the high-level plan's granularity.
* The real-world experiments show very impressive results in 28-block stacking. However, the simulated experiment shows much lower success rates in 5-block and 6-block stacking, which is likely a lot easier than the real-world 28-block stacking task. The authors can explain the discrepancy between the two results in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The experiments seem through. Most implementation details can be found in the appendix.

Figure 20 seems to be rendered with only 10 runs, which is supposed to be 12 runs.

summary_of_the_review: The paper proposes to replace high-level planning with a manually-designed simplified environment and a solution from the simplified environment. Although the paper shows the benefits of the proposed approach, given that it highly relies on the hand-designed high-level planner, it seems to have limited scalability. Therefore, I would be leaning toward a weak rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors tackle the problem of motion planning for robotics. Given a motion planning problem, the authors first simplify the problem to an "abstract" MDP that classical planning algorithms can solve. The authors then use a policy parameterized by a transformer that conditions on the solution to the "abstract" problem while taking action in the "real" MDP. The authors demonstrate that by choosing different "abstract" trajectories for the policy to condition on at test time, the authors can direct the robot to perform impressive novel tasks not seen during training.
strength_and_weaknesses: I found this paper and the accompanying website and supplementary material easy and interesting to read. My personal pitch for the method is that goal-conditioning and reward-densification are both important for training a useful motion planner, and the authors use a high-level trajectory to enable both of these capabilities. The baselines also seem to demonstrate the importance of both of these features of the method, e.g. my interpretation is that SGC uses just reward-densification and GC/LSTM ablate the goal-conditioning mechanism to some extent. Additionally, I appreciate the section of the appendix that covers failure modes.

The main weakness that I see in this paper is I feel it would be good to get some intuition on what is and isn't a good abstract MDP to use for conditioning. Is there a way the authors can adjust the extent to which the abstract MDP is similar to the real MDP and show that the "closer" the abstract MDP is to the real MDP, the better the policy performs? What happens when the abstract MDP is equivalent to the actual MDP - is that oracle performance? Why do the authors condition on abstract states instead of abstract actions? If the model conditioned on actions, the case where the abstract MDP is equivalent to the real MDP would just be a copying task but since it's states, it's less obvious to me whether or not using the real MDP would produce the best results.
clarity,_quality,_novelty_and_reproducibility: The authors provide a flexible framework for integrating classical path planning algorithms into policies parameterized by neural networks. In doing so, the authors show that they provide high-level control by adjusting the "abstract" trajectories that the policy conditions on. I found the paper well-written and clear. I don't feel completely confident commenting on the originality of the work, but I do feel that this paper primarily uses standard tools (e.g. Seq2Seq models) but combines them in a way to get impressive task-generalization that I haven't seen from other papers.
summary_of_the_review: This paper proposes a principled strategy for leveraging classical path planning algorithms to improve neural controllers. I found the numerous empirical results quite compelling. With additional analysis of which kinds of "abstract MDPs" are best for conditioning on, I think this paper will be a good contribution to the machine learning community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel online RL framework, Trajectory Translation, that enables one-shot task generalization in robotics with abstracted trajectories. Trajectory Translation takes an abstract trajectory that is obtained on the simplified simulator (pointmass, magnetic manipulation, etc) as an input combined with the current execution trajectory, and then policy outputs the executable actions. Such translation is learned with PPO and trajectory-aware state-distance-based reward function. Combined with GPT-2 architecture, the proposed method provides strong one-shot performances on Box-Pusher/Couch Moving/Block Stacking/Open Drawer. In difficult or some intervention tasks, the replanning with the abstracted instruction improves the performances.
strength_and_weaknesses: ### Strength
- The proposed framework, trajectory translation, seems to be a novel paradigm for "demonstration-guided" reinforcement learning. Borrowing the success of causal Transformer architecture, the policy can "translate" an ideal/simplified trajectory to an actionable one in unseen configurations.
- The experimental results show TR$^2$-GPT2 significantly outperforms other baselines (TR$^2$-LSTM, SGC, GC) and could solve complex tasks, such as Block Stacking or Open Drawer.

### Weaknesses
- It seems unclear how we can obtain abstract trajectories.  If we collect abstract trajectories with some manually-designed waypoints, some robotics planners could solve the tasks even in executable settings.
- It is also unclear that how the proposed methods handle different dimensionality of the state between abstract and executable trajectory. The dimension of $s^H$ and $s^L$ seems very different (pointmass vs robot).
clarity,_quality,_novelty_and_reproducibility: ### Quality
This paper is well-written and easy to follow.

### Clarity
There are some unclear points in the paper:
- In figure 4 (a), the policy pays attention to the current and the next next chamber, but not to the next chamber where the agent needs to rotate the couch. In contrast, in (b), the policy pays attention to the next chamber, although the agent doesn't need to rotate the couch anymore. Also, I guess the reason why the policy doesn't pay attention to the past paths might be simply because of the context length of the executable policy. I'm not sure whether these patterns describe why TR$^2$-GPT2 works well.

### Originality
The framework that leverages abstract trajectory in the simplified simulator and translates it with a causal transformer to the executable actions seems a novel approach for task generalization to unseen configurations.

summary_of_the_review: While there are some unclear points listed above, the proposed method seems a decent contribution to the community (novelty, good demonstration video, etc). In addition, the empirical evaluation is persuasive enough. Considering those aspects, I lean toward acceptance.

---
Update: I thank the authors for addressing my concerns and questions. Even after the discussion period, I still think my evaluation is fair for your contributions for now. So, I'd like to keep my score as it is.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the problem of generalizable long-horizon planning for embodied tasks such as robot manipulation. The paper proposes an approach that decouples the identification of a suitable long-horizon plan from its execution in the given environment. More specifically, the framework first establishes an abstract representation of the environment (e.g., where objects become point masses) where dynamics are ignored; solves for a high-level solution (trajectory) in this simplified representation of the environment; and then uses a sequence-to-sequence model to "translate" the high-level trajectories to their corresponding low-level trajectories in the actual environment. Experiments on both simulated and real-world robot manipulation tasks demonstrate that the proposed framework outperforms contemporary baselines.

The ability to perform long-horizon embodied (manipulation) tasks remains an open problem that is of significant interest to the robot learning community. As the reviewers point out, the idea of decomposing long-horizon tasks in terms of separate problems of plan generation and plan execution is sensible, as is the use of some amount of domain knowledge as a means of abstracting the environment and task. The specific way in which the proposed Trajectory Translation (TR2) framework achieves this decomposition, via reward densification and goal-conditioning, is principled and shown to be effective on real-world and simulated manipulation tasks.

The reviewers raised a few questions/issues with the initial submission and the authors clearly made a significant effort to address these concerns, through detailed responses to each of the reviewers, additional results, and updates to the paper. Each of the reviewers commented that this effort clarified several of their questions and addressed many of their initial concerns. However, the reviewers remain unconvinced that the amount of domain knowledge assumed for the proposed approach with regards to the environment abstraction and the plan generation is consistent with existing one-shot learning methods. They also find that it is not clear how one would apply the framework to other domains. The AC acknowledges the authors' comparisons to the assumptions made by contemporary one-shot learning methods (e.g., the use of human demonstrations), but finds the arguments not entirely convincing in their current form. This is a promising line of work with good potential, and a revised version of the paper that resolving these remaining issues would provide a solid contribution.
justification_for_why_not_higher_score: Each reviewer acknowledged the authors' response. They point out that while the response and updates to the paper address some of the issues raised in the initial reviews, their primary concerns with the paper still hold.
Decision: Reject