Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies planning with language models (LM) using iterative energy minimization. The authors utilize a bidirectional LM with the masked language model (MLM) objective. The model is trained by optimizing for MLM using expert trajectories. For inference, they iteratively mask actions in a trajectory, including future actions, and use Gibbs sampling to iteratively predict and refine actions. The authors use a decision transformer (DT) for the encoding of a trajectory and explain benefits of this iterative planning approach. Empirical results on BabyAI and Atari games show that LEAP outperforms previous offline RL methods on BabyAI and it is in par with DT on Atari. 
strength_and_weaknesses: **Strengths** The paper is written well and easy to follow through. I found the iterative planning with bidirectional LMs interesting and broadly applicable to other settings including multi-agent learning (not mentioned in the paper).

**Weaknesses**

1. There are no model-based or autoregressive planning baselines. For example, *PlaTe* and *MuZero Unplugged* are two baselines that need to be cited and compared against.

2. Baselines for the Lava experiment are used as is while a large energy is added for those states in LEAP; which a bit unfair in my opinion. Could you add baselines where you just mask an action for DT/IQL if the action is leading to a lava?

3. At the beginning of section 3, in both argmax and argmin, you don't have $s_{2:T}$. Was this intentional or just a mistake in writing? If it was intentional, could you explain why you only condition on the first state in your formulation? You use only the first action in Atari, so I am assuming this doesn't apply to it.

4. Please add confidence intervals to Table-3. DT results have wide intervals and it is difficult to compare LEAP to DT without error bounds.

5. Can you discuss if LEAP would work for non-deterministic dynamics? 

6. In Appendix A.2. you mention causal attention while in main text you mention bidirectional attention with future. Please clarify if you use bidirectional or causal attention.

7. You mention that $\tau^*$ is derived by minimizing $E_\theta(\tau)$ but this is not true as your model doesn't predict states. Please clarify.

8. "baseslines" --> "baselines",
"adaptaiton" --> "adaptation"


PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks. Jiankai Sun, De-An Huang, Bo Lu, Yun-Hui Liu, Bolei Zhou, Animesh Garg.

Online and Offline Reinforcement Learning by Planning with a Learned Model. Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, David Silver.
clarity,_quality,_novelty_and_reproducibility: I found the paper well written and easy to follow. I think the planning with bidirectional language models for offline RL is interesting. While most implementation details are given, some pieces are unclear to reproduce results such as transformer attention mechanism.
summary_of_the_review: I think bidirectional LM based planning with Gibbs sampling is interesting. I found some baselines missing, especially relevant model-based and planning baselines.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper looks at the use of masked-language models trained on a given set of trajectories to generate action sequences that minimize a specified energy function. If the energy function captures the objectives of a sequential decision-making problem, then the procedure could be utilized to generate the course of action to be performed. The proposed system is then tested in various domains including, BabyAI and ATARI and shows a number of desirable properties such as online adaptation,  novel environment generalization and task compositionality.
strength_and_weaknesses: One of the main strengths of the approach is the fact that it presents a straightforward application of existing learning methods. 

In terms of weaknesses, one of the central problems is the clarity of exposition. In fact, the current description of the method presented in the paper is not sufficient to clearly evaluate the properties of the method. First off, the method is referred to as planning. In the common usage of the term, planning usually involves reasoning with some form of task model. It isn’t completely clear if there is a task model being maintained in this case. Is it referred to as planning because you are using an energy function or is the probabilistic model meant to stand for the world model? 

Also assuming access to a set of near optimal training data brings the setting closer to a behavior cloning set up than traditional RL set up, where the agent has to learn through interaction with the environment. The fact that for the experiments, you had to initially run an RL agent (at least in atari) kind of speaks against the usefulness of these methods on their own. Also the energy function itself makes no reference to total reward associated with each trajectory. If the original trajectory set contained trajectories of varying quality how does the model differentiate them during the testing time? Would it just have to assume the better action showed up with higher frequency? 

It is also unclear to me if the method was only designed for deterministic settings? For, stochastic settings solutions can’t take the form of a sequence of actions, rather it needs to take the form of policies. Which means after each action, it needs to check what state it leads to and decide the next action based on this information. While in theory the model can support such execution, it is unclear if one really benefits from the use of such a sequential model in stochastic settings. After all the sequence predicted by the model, even if it is the highest likelihood one, may end up not being followed.  Instead at each step, one needs to consider the expected value associated with following a given action. However as discussed above, the model doesn’t really seem capable of associating any kind of value with a specific action.

Finally on a smaller note, while the learning methods itself uses tools from NLP that were used to learn language models, they are not used to learn language models in this context. So I might advise the authors from referring to the method as planning using language models and maybe just refer to the fact that you are using sequential models.

In terms of experiments, as discussed the generation of action trajectories are a bit unrealistic, but apart from that the experiments presented seem like a reasonable starting point. One point I would like to know is whether the model presented here can perform better than the traces that were present originally. In other words, if it was presented a state and context that correspond to an example trace part of the initial set, could it have come up with a better course of action by leveraging other traces it saw.

Post Rebuttal Comment: After the new set of experiments and the detailed discussion with the authors. I am happy to recommend the paper for acceptance.

clarity,_quality,_novelty_and_reproducibility: As discussed above there are quite a few details that are unclear which makes it hard to completely evaluate the method. In terms of novelty, there are quite a few works that have looked at using a transformer based model for predicting action sequences. Some of which are mentioned in the paper, a few relevant ones like [1] are overlooked (I would definitely urge the authors to look at [1] as it does touch up on the issue of handling stochastic setting). However to the best of my knowledge, their specific approach to sampling sequence is novel.

[1] Paster, Keiran, Sheila McIlraith, and Jimmy Ba. "You Can't Count on Luck: Why Decision Transformers Fail in Stochastic Environments." arXiv preprint arXiv:2205.15967 (2022).
summary_of_the_review: While I think the paper presents an interesting direction and formulation, I don’t believe the paper in its current form is ready to be published yet.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to leveraging a language modeling objective to train an energy function which can be used together with a sampling procedure to generate trajectories for discrete-control tasks. The approach uses a masked language modeling loss to train a bi-directional language model to fit expert demonstrations; the likelihood of this model is used to choose actions (given a context of states and actions) via an iterative Gibbs sampling procedure. The method is tested on a subset of BabyAI and Atari tasks and compared to several baselines; performance is significantly better than baselines on the BabyAI tasks and about the same as baselines on Atari tasks. Extensions of the approach to enable solutions to compositional problems are also propose and show good results on the tested tasks.
strength_and_weaknesses: Strengths:
1. This paper is very well written. The central message of the paper is clearly conveyed and the figures nicely illustrate the different potential applications of the proposed approach.
2. The idea of using a bi-directional language model for energy estimation and using a sampling based approach for computing actions leveraging this energy is interesting and shows good results compared to baselines on a subset of the tested tasks. 
3. The approach is extensible and can easily lend itself for compositional objectives which prior methods seem to struggle on.

Weaknesses:
1. A key point mentioned in the motivation of the approach is that "auto-regressive" modeling approaches can struggle to provide good action selections for long-horizon problems due to their uni-directional nature. This was used as a motivation for bi-directional reasoning but it is not clear if this makes a difference as there are no baselines that ablate this selection (I presume a different planning method would also be needed for this ablation).
2. The key novelty (in my opinion) of the presented approach is the choice of masked language modeling style losses for modeling the expert data distribution, and subsequent use of this likelihood function for planning. Which of these two matter more? Would any generic planning approach (e.g. random shooting vs Gibbs sampling vs cross-entropy style planning) work with the proposed energy function? And vice-versa would other ways of approximating the energy work with the proposed planning approach? This is unclear from the current results, and an ablation for this would be very illuminating.
3. The results on Atari are quite weak; in fact, CQL performs better than the proposed approach on 3/4 tasks, and only on a single task the approach gets a high final score which makes the overall score be higher than CQL. Additionally, the results in Fig 7 indicate that while there is some correlation between the achieved rewards and the estimated energy there are a significant number of outliers. Why is this the case?
4. There are three specific extensions of the proposed approach mentioned: online adaptation, novel environment generalization and task compositionality. It is not clear if these are unique to the presented approach. Specifically, the proposed online adaptation schema can work for any sampling based planning approach if rejection sampling is used, and the task compositionality only works if the energy functions can be linearly combined (and even then there can be significant local optimas from the planning perspective). These should be highlighted in the presentation.
5. A potential limitation of the proposed approach is its applicability to continuous action spaces; several, if not all of the presented baselines can directly be applied to continuous action spaces whereas the proposed approach is limited to discrete tokens. This can potentially be relaxed by binning/discretizing continuous actions but nonetheless it would be useful to discuss this in the main text.
6. It would useful to understand the computational/time complexity of the proposed planning approach as it requires several sequential iterations with a potentially large model for successful planning.
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is well written and the approach and results are clearly described and illustrated. The results clearly highlight the strengths of the proposed approach, but some further improvements to the ablations and discussions on potential limitations of the proposed method could add further clarity.

Quality & Novelty: The idea of using a language modeling loss to estimate the energy of the data distribution & its use for planning action sequences is interesting and novel. The experiments show promising results, albeit on a limited set of tasks. There are several baseline comparisons and some ablations provided but further ablations can add value.

Reproducibility: The code for the paper is made available on Github and makes it potentially easy to reproduce the results.
summary_of_the_review: Overall, this paper presents an interesting new approach to utilizing a language modeling loss to estimate an energy for planning action sequences and is tested on a small set of tasks in two different domains. The results are promising and show improvements compared to the presented baselines. I would suggest a borderline accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a novel approach (LEAP) for modelling RL trajectories and planning with the model. Unlike GPT-2/Trajectory Transformer style of autoregressive model, it applies BERT style modelling where the actions can be masked in any position of the trajectory. With such a model, planning does not have to follow temporal order. The empirical results show LEAP can do flexible planning and surpass model-free offline RL methods in terms of success rate or return.
strength_and_weaknesses: Strength:
* The proposed method is rather novel for RL planning as it goes beyond step-by-step planning which gives extra flexibility and might lead to a new thread of future work.
* Empirical results show LEAP can fit multiple scenarios of planning, eg. goal-oriented, reward maximization and online adaptation by a constraint function.

Weakness:
* Empirical evaluations did not include any other planning-based (actually any model-based) methods. It's then hard to compare the proposed method with conventional planning.
* some of the notations are confusing. 
  * The trajectory length is first defined as $n$ and then it redefined as $T$.
  * I'm also not sure what does eq (3) mean. Why all the padded actions correspond to state $s_n$?
  * eq(2) defines energy function as pseudolikelihood but then in the Atari experiment, it seems like a different function is used which is associated with reward. Also, if the optimization objective can be anything, why it's called energy?

Questions:
1. Does $\pmb s$ in eq (1) and eq (2) mean $\pmb{s}_{1}$? So the "model" is not predicting states?
2. Following question 1, if that's the case, how can you define constraint based on future states in eq (4)? Are you using the simulator for planning?
3. How did you get the value and reward estimation for Atari experiments?
4. What did you get the goal state for BABYAI tasks?

clarity,_quality,_novelty_and_reproducibility: * The quality of the paper is in general OK but there seem to be some notation problems.
* The clarity is not good. I'm still a bit confused about a lot of technical details after reading through it.
* The paper is quite original for RL community.
* Code is not provided, due to the low clearance of the methodology, it can be very difficult to reproduce the results even if the hyperparameters are provided.
summary_of_the_review: I think the idea of the paper is interesting and the empirical studies also show great potential for this work. However, the write-up of the paper still needs to be improved to improve clarity. Current I still have a lot of questions about the method, which may affect my final recommendation but in general, I'm leaning to accept the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an approach to using a masked language model trained on expert trajectories to generate sequences of actions that minimize a specified energy function. The method is tested on a subset of BabyAI and Atari tasks and compared to several baselines, and shows improved performance on the BabyAI tasks and comparable performance on Atari tasks. The paper also explores the use of the approach to solve compositional problems. Strengths of the paper include its novel approach to integrating language models with planning, and its ability to generate sequences of actions that minimize energy. The paper's weaknesses include its limited experimental evaluation and the lack of comparison to state-of-the-art methods, however many of these weaknesses were address during the discussion period, with all the reviewers suggesting that the paper is accepted.


justification_for_why_not_higher_score: While the authors managed to address many of the reviewers' concerns, some reservations still remain about the general applicability of the method.
justification_for_why_not_lower_score: All of the reviewers agree that the paper should be accepted.
Decision: Accept: poster