Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling

## Abstract

Abstract In the endeavor to make autonomous robots take actions, task planning is a major challenge that requires translating high-level task descriptions into long-horizon action sequences. Despite recent advances in language model agents, they remain prone to planning errors and limited in their ability to plan ahead. To address these limitations in robotic planning, we advocate a self-refining scheme that iteratively refines a draft plan until an equilibrium is reached. Remarkably, this process can be optimized end-to-end from an analytical perspective without the need to curate additional verifiers or reward models, allowing us to train self-refining planners in a simple supervised learning fashion. Meanwhile, a nested equilibrium sequence modeling procedure is devised for efficient closed-loop planning that incorporates useful feedback from the environment (or an internal world model). Our method is evaluated on the VirtualHome-Env benchmark, showing advanced performance with better scaling for inference computation. Code is available at https://github.com/Singularity0104/equilibrium-planner .

## 1 Introduction

Recent advances in large language models (LLMs) have spurred tremendous progress in robotic planning (Huang et al., ; Li et al., ; Singh et al., ; Driess et al., ; Ahn et al., ; Huang et al., ; Zhao et al., ; Hu et al., ). Based on their extensive world knowledge, LLM agents seem close to autonomously performing robotic tasks, such as in household scenarios. However, growing evidence shows that existing LLM agents struggle with task planning (Kaelbling & Lozano-Pérez, ) that decomposes a high-level task into mid-level actions. While this problem requires long-horizon planning as well as consideration of environmental feedback, LLMs are often limited by: (1) unidirectional dependency: due to autoregressive generation, previous tokens cannot attend to future tokens, resulting in limited ability to plan ahead (Wu et al., ); (2) lack of error correction for existing outputs, unless with a heavy system 2; (3) fixed forward process hindering the allocation of more inference computation to further improve planning performance. These inherent limitations of LLMs lead to inefficiency in the closed-loop long-horizon robotic planning.

To address above challenges of LLM planners in closed-loop long-horizon planning, we advocate the approach of self-refinement (Welleck et al., ; Shinn et al., ; Kim et al., ; Madaan et al., ) that iteratively improves a previously generated plan. The reasons behind are threefold: (1) bidirectional dependency: since the output is conditioned on a previous draft plan, it can attend to all tokens in the plan (from an old version), thus improving its ability to plan ahead; (2) internal error correction which allows implicit self-correction in a forward pass without an explicit, heavy system 2; (3) dynamic computation allocation by iterating through a self-refinement process until convergence. However, such a self-refining strategy imposes significant training difficulties because it requires backpropagation through infinite self-refining steps (Werbos, ). This may be seen as an extreme case that reflects some of the general challenges in teaching LLMs to plan and reason. Existing solutions include curating process supervision (Uesato et al., ; Lightman et al., ) or applying reinforcement learning (Zelikman et al., ; OpenAI, ; Kumar et al., ), but they are considerably more complex than supervised training and difficult to implement.

This work proposes an simple learning framework for planning via self-refinement. Specifically, we formulate the self-refining process as a fixed-point problem that recursively refines the plan until the equilibrium point, as illustrated in Fig. 1. While the forward process of this fixed-point problem could be solved efficiently using root-finding methods, more interestingly, its backpropagation can be skipped since its gradient is explicated by the implicit function theorem (Krantz & Parks, ) as in deep equilibrium models (Bai et al., ; Geng & Kolter, ).
It is noted that the derived gradient term may be further simplified through a Jacobian-free approximation (Fung et al., ) to facilitate training. These analytical techniques allow end-to-end supervised training of the LLM planner to accomplish self-refinement without the need for additional verifiers or reward models in reinforcement learning-based counterparts, greatly enhancing simplicity and practicality. And after training, our equilibrium model-based planner is capable of dynamically allocating more inference computation in the equilibrium solving process to achieve better planning performance.

Another important cue for self-refinement in robotic tasks is closed-loop feedback from the environment. To efficiently incorporate environmental feedback, we devise a nested equilibrium sequence modeling procedure consisting of inner and outer loops, where the inner loop iteratively refines a plan using previous feedback, while the outer loop updates the feedback by interacting with the environment. This enables closed-loop planning from even a few environmental interactions. Moreover, the nested equilibrium solving process is accelerated by reusing the previously derived equilibrium. We further implement the above design within an LLM agent framework, seamlessly integrating the equilibrium model-based planner, an experience memory buffer containing past plans and feedback, and a world model to estimate feedback in the absence of environmental interactions,
thus enabling it to operate effectively in robotic planning.
The contributions of our work are summarize as follows:

- •
We present equilibrium sequence modeling, a simple training approach for self-refining LLMs based on equilibrium models, allowing for end-to-end learning in a supervised manner.
- •
A nested equilibrium solving process is proposed to efficiently incorporate closed-loop feedback into the equilibrium sequence modeling, reusing previous equilibrium solutions to alleviate inference computation. It is further implemented with a world model to improve practicality.
- •
Our method is evaluated on the VirtualHome-Env benchmark (Puig et al., ; Liao et al., ), demonstrating its advantageous performance with better scaling w.r.t. inference computation.

Figure: (a) Equilibrium in planning
Refer to caption: x1.png

## 2 Related Work

LLMs for Planning. Scaling up inference computation to improve LLMs’ performance on planning and reasoning tasks has received increasing attention (Brown et al., ; Snell et al., ; Wu et al., ; OpenAI, ). Existing techniques involving chain-of-thought (Wei et al., ; Zelikman et al., ; ), repeated sampling (Wang et al., ; Brown et al., ) and tree search (Yao et al., ; Zhao et al., ) showed preliminary results with handcraft system 2. Alternatively, a method called self-refinement (Welleck et al., ; Shinn et al., ; Kim et al., ; Madaan et al., ) suggests recursively refining the existing LLM output in an autonomous manner, but it relies heavily on prompting or intricate training procedures. To fully exploit its potential, we propose an end-to-end optimization method for self-refinement through deep equilibrium models.

Deep Equilibrium Models (Bai et al., ) are infinite-depth neural networks specified by fixed-point problems $x^{*}=f_{\theta}(x^{*})$, where $f_{\theta}$ is an equilibrium layer. While their inference can take infinite steps by the fixed-point iteration, their gradients are estimated using implicit differentiation (Krantz & Parks, ) without backpropagating through all layers, thus enabling memory-efficient training. They have been extensively applied to tasks such as visual understanding (Bai et al., ; ) and image generation (Pokle et al., ; Geng et al., ; Bai & Melas-Kyriazi, ). In this paper, we apply the fixed-point formulation of deep equilibrium models to the self-refinement process in LLM planners, allowing for simple supervised training of LLMs to refine themselves.

## 3 Method

We study the problem of robot task planning which aims to decompose a high-level task description into long-horizon mid-level action sequences (Kaelbling & Lozano-Pérez, ). In the following, we first discuss the limitations of LLM planners in self-refinement (Section 3.1) and address them with a novel equilibrium sequence modeling scheme (Section 3.2), which is compatible with various feedback from the designs in Section 3.3. Lastly, practical implementations are given in Section 3.4.

### 3.1 Preliminaries on Self-Refinement

The prevailing LLMs are intrinsically limited in planning, as their unidirectional dependency results in limited capability to plan ahead (Wu et al., ), and the lack of error correction hinders closed-loop planning. These reasons call for alternative mechanisms to address robot task planning.

Recently, Welleck et al. (); Shinn et al. (); Kim et al. (); Madaan et al. () proposed self-refinement, which employs an LLM to iteratively refine the previous LLM output. This strategy naturally addresses the above limitations, since it introduces bidirectional token dependency and a dynamic error correction mechanism. Formally, let $x_{t}$ denote a draft plan and $c_{t}$ denote context (*e.g*. environmental feedback), then planning may be viewed as a self-refinement process as follows:

$$ $x_{t+1}=f_{\theta}(x_{t},c_{t}).$ (1) $$

However, self-refinement via prompting (Shinn et al., ; Kim et al., ; Madaan et al., ) has been found to be very limited by Huang et al. (). Alternative training-based methods require careful curation of training sequences (Welleck et al., ; Havrilla et al., ) or reinforcement learning (Qu et al., ; Kumar et al., ) and are therefore difficult to train. Overall, they remain deficient for robotic planning compared to system 2-based alternatives, as shown in Hu et al. ().

### 3.2 Self-Refinement as An Equilibrium Model

To address the training inefficiency of self-refinement approaches, this section proposes equilibrium sequence modeling, a simple supervised training scheme for teaching LLM planners to self-refine through the lens of deep equilibrium models (Bai et al., ; Geng & Kolter, ).

Let us first consider a simplified scenario of self-refinement without environmental feedback, namely that the context $c_{t}$ is fixed, *e.g*. to a predefined system message $c$. Then, the self-refinement process in Eq. 1 reduces to a fixed-point problem concerning only the plan $x_{t}$. Denote the initial plan by $x_{0}=\varnothing$ and the equilibrium plan, *i.e*. the endpoint, by $x^{*}$, then its trajectory can be expressed as:

$$ $(x_{0},c)\rightarrow\ldots\rightarrow(x_{t},c)\rightarrow\ldots\rightarrow(x^{ *},c).$ (2) $$

Although its forward process is tractable with existing root-solving techniques, such as the classic fixed-point iteration or alternative numerical methods (Broyden, ; Anderson, ), its training requires recurrent backpropagation through multiple self-refining steps (Werbos, ). This results in an extremely inefficient and unstable computational process where end-to-end training fails.

Instead, we approach it directly from an analytical perspective. Assuming access only to an outcome supervision $L(\cdot,y)$ on the plan, *e.g*. its distance to the ground truth plan $y$, then self-refinement is formulated as an optimization problem minimizing the loss function of the equilibrium plan:

$$ $\displaystyle\min_{\theta}$ $\displaystyle L(x^{*},y)$ (3) s.t. $\displaystyle x^{*}=f_{\theta}(x^{*},c).$ $$

Interestingly, the above optimization problem can be solved without backpropagating over the entire inference process. As the following theorem indicates, we can directly differentiate through its fixed point regardless of the solution path, with only a constant computational and memory cost.

###### Theorem 1

(Implicit Function Theorem (Bai et al., ; Krantz & Parks, ))
Assuming that $\big{(}I-\frac{\partial f_{\theta}}{\partial x^{*}}\big{)}$ is invertible, then the loss gradient of Eq. 3 w.r.t. $\theta$ is given by:

$$ $\frac{\partial L}{\partial\theta}=\frac{\partial L}{\partial x^{*}}\left(I- \frac{\partial f_{\theta}}{\partial x^{*}}\right)^{-1}\frac{\partial f_{\theta }}{\partial\theta}.$ (4) $$

Its proof is provided in Appendix A. It is noteworthy that the inverse Jacobian term $A=(I-\frac{\partial f_{\theta}}{\partial x^{*}})^{-1}$ within the above gradient is difficult to compute exactly, for which existing work often approximates through the damped fixed-point unrolling or the Neumann series (Geng et al., ). For simplicity, we follow Fung et al. (); Geng et al. () to drop the inverse Jacobian term using $A\approx I$ (which is justified by Theorem 3.1 in Fung et al. () under appropriate assumptions):

$$ $\frac{\partial L}{\partial\theta}=\frac{\partial L}{\partial x^{*}}\frac{ \partial f_{\theta}}{\partial\theta}.$ (5) $$

Figure: (a) Inference: solving equilibrium via fixed-point iteration
Refer to caption: x2.png

Equilibrium Sequence Modeling.
Based on the simplified gradient estimation, we reformulate its training into a supervised learning problem. According to the chain rule, the derived gradient is exactly the gradient of the following optimization problem associated with the equilibrium $x^{*}$:

$$ $\min_{\theta}\quad L(f_{\theta}(x^{*},c),y).$ (6) $$

This new formula represents a new equilibrium sequence modeling scheme that can be implemented in two stages as in Fig. 2. In the first stage, we iterate over the fixed-point problem within Eq. 3 to solve the equilibrium plan $x^{*}$. Then, it is paired with the ground truth plan $y$ as a training sequence, which is used as in the standard supervised finetuning pipeline to teach the LLM to self-refine.

It features two intuitive advantages: (1) instead of directly regressing the ground truth, it only adjusts the equilibrium point, which reduces overfitting compared to the vanilla supervised finetuning; (2) it teaches self-refinement by a simple supervised loss, without requiring additional value functions or reward models (Welleck et al., ; Havrilla et al., ; Qu et al., ; Kumar et al., ).

###### Theorem 1

### 3.3 Equilibrium Models with Feedback

This section extends the derived equilibrium sequence modeling to a more practical scenario where the environment may provide some closed-loop feedback, *e.g*. failure details, during plan execution. Such auxiliary information would be an effective cue for planners to further refine their plan.

To take into account environmental feedback, we consider an adaptive context $c_{t}$ that is influenced by the plan $x_{t}$ rather than fixed. Then, the previously concerned equilibrium solving process of Eq. 2 should be revised as an iterative process that couples plan and feedback, starting from $x_{0}=c_{0}=\varnothing$:

$$ $(x_{0},c_{0})\rightarrow\ldots\rightarrow(x_{t},c_{t})\rightarrow\ldots \rightarrow(x^{*},c^{*}).$ (7) $$

After the modification, the existing derivations only hold when we neglect the derivatives related to $c^{*}$. Fortunately, this is a natural choice due to the non-differentiability of most feedbacks. Therefore, the equilibrium planner can be trained in a similar supervised way as in Eqs. 6 and 2, and after training it would be able to self-refine based on the latest feedback just by forward passes. However, iteratively interacting with the environment to obtain feedback is costly and may not be recoverable. In response, we devise a nested equilibrium solving scheme for more efficient closed-loop planning.

Figure: Algorithm 1 Inference of Equilibrium Planner

Nested Equilibrium Solving. Inspired by the introspection process in human daily life, we propose to divide equilibrium solving into a nested loop process. The inner loop introspects on the existing plan and feedback and takes no action, while the outer loop interacts with the environment to update the feedback. Formally, each inner loop is an equilibrium solving process with fixed feedback $c_{t}$:

$$ $\begin{cases}\;x_{t}^{1}=f_{\theta}(x_{t}^{0},c_{t})\\ \;\cdots\\ \;x_{t}^{*}=f_{\theta}(x_{t}^{*},c_{t}).\end{cases}$ (8) $$

Thanks to this inner-loop introspection mechanism, our equilibrium model can be more efficient in closed-loop planning, achieving superior performance with fewer environmental interactions.

Reusing Equilibrium Solution. Another efficiency bottleneck is the equilibrium solving. Considering that its speed depends largely on the initial plan, it is unnecessary to restart from $\varnothing$ every time. Therefore, we accelerate equilibrium solving by reusing the previously derived equilibrium plan as the starting point of the next iteration, similar to Bai et al. (); Bai & Melas-Kyriazi ():

$$ $x_{t+1}=x_{t}^{*}.$ (9) $$

which corresponds to the starting point $x_{t+1}^{0}$ of the inner loop. Similarly, history feedback could be reused across different inner loops. This is achieved by initializing the context of the next inner loop by concatenating the previous feedback with the latest feedback (paired with its original plan):

$$ $c_{t+1}=(x_{t+1},\text{Env}(x_{t+1}))\parallel c_{t},$ (10) $$

where $\parallel$ denotes concatenation. The whole nested inference procedure is described in Algorithm 1.

### 3.4 Practical Implementation

This section discusses the implementation of the proposed equilibrium planner. To enable effective training while interacting with the environment, the following two modules are carefully devised to complement the planner: an experience memory that caches all equilibrium plans and their feedback during equilibrium solving, and a world model to estimate the closed-loop feedback in the absence of environmental interactions. The planning framework is illustrated in Fig. 3.

Equilibrium Experience Memory. During the training process, our equilibrium model interacts with the environment only when the inner loop reaches the equilibrium point. This results in a small number of equilibrium points, which may not be sufficient for supervised training. To improve our training efficiency and stability, we opt to cache all previously obtained equilibrium points, along with their environmental feedback, in an experience memory. Thereafter, these equilibrium points can be sampled repeatedly for versatile training purposes. For example, for the planner, we randomly sample a batch of equilibrium points at each training epoch, which are paired with the ground truth for supervised training. In particular, the most recent equilibrium points are sampled more frequently to reduce distribution shift. Next, we describe another crucial component of our framework.

Internal Feedback from World Model. Due to inefficiency of interacting with the environment, we maintain a world model (Ha & Schmidhuber, ) to provide the necessary feedback in closed-loop planning. Our world model takes the environmental context, task instruction and current plan as inputs and predicts some basic types of feedback. This definition is slightly simpler than the commonly used world model, which requires simulation of the environmental states, and therefore may to easier to train. Concretely, we implement the world model with an LLM and finetune it on the planner’s equilibrium feedback over all iterations for better generalizability. And during inference, we alternate between using external and internal feedback in closed-loop robotic planning.

Figure: Figure 3: Illustration of our proposed framework. It incorporates (1) a memory containing all the equilibrium experience during inference, (2) a self-refining planner trained on equilibrium plans with the ground truth, and (3) a world model trained on the memory to simulate environmental feedback.
Refer to caption: x3.png

## 4 Experiments

### 4.1 Experimental Settings

Benchmark.
The VirtualHome-Env benhmark (Puig et al., ; Liao et al., ) is adopted during the experiments. It contains 1360 long-horizon tasks with ground truth action sequence annotations (average length 10.8) and provides updated scene graphs after each action, allowing simulation of closed-loop feedback. To analyze the generalizability, we divided the dataset into a training set and three test subsets, including the novel scene set, the novel task set, and the novel scene and task set. More statistics and examples about VirtualHome-Env can be found in Section B.1.

Metrics.
We use executability (Exec.), success rate (SR), goal conditions recall (GCR) following Hu et al. (). Exec. evaluates whether the plan can be executed in given the environment, SR refers to whether the goal is accomplished, and GCR measures the proportion of goal conditions achieved. To examine the closed-loop planning capabilities, we study two test setups, without error correction or with up to 10 corrections, the latter allowing for self-correction based on environmental feedback. We also evaluate the computational efficiency by measuring TFLOPS at inference time.

Baselines.
Our method is compared with Tree-Planner (Hu et al., ), SELF-REFINE (Madaan et al., ), and a supervised finetuned planner. They are all reproduced using Llama 3 8B (Dubey et al., ). In addition, we consider several baseline methods that call the GPT-3.5 API, including ProgPrompt (Singh et al., ), Zero-shot Planner (Huang et al., ) and two self-refining planners, Local Replan (Raman et al., ; Guo et al., ) and Global Replan (Shinn et al., ). Their results are presented for reference only. See Section B.2 for more details.

Implementation Details.
Our implementation is consistent with the baseline methods by finetuning from Llama 3 8B (Dubey et al., ) on the VirtualHome-Env training set (paired with the equilibrium points). The number of finetuning epochs is set to 6, and the learning rate is set to 0.0002. The world model is finetuned on all planner interactions for 5 epochs using the same learning rate. A greedy LLM sampling strategy is used in later refinement steps until convergence. Moreover, we implement the KV cache to speed up inference. Further details are provided in Section B.3.

**Table 1: Performance on VirtualHome-Env without correction. Our planner achieves state-of-the-art performance in most evaluations. Note that the Exec. metrics are marked in gray because they are already high and can easily exceed 99% with simple automated rules (by truncating illegal output).**
|  | Novel Scene and Task | Novel Scene | Novel Task |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Exec. | SR | GCR | Exec. | SR | GCR | Exec. | SR | GCR |
| GPT-3.5 API: |  |  |  |  |  |  |  |  |  |
| Zero-shot Planner | 16.49 | 1.07 | 1.52 | - | - | - | - | - | - |
| ProgPrompt | 35.04 | 12.54 | 19.99 | - | - | - | - | - | - |
| Iterative-Planner | 44.54 | 27.04 | 33.25 | - | - | - | - | - | - |
| Tree-PlannerN=25 | 55.74 | 28.33 | 39.96 | - | - | - | - | - | - |
| Tree-PlannerN=50 | 49.01 | 28.14 | 35.84 | - | - | - | - | - | - |
| Finetuned Llama 3 8B: |  |  |  |  |  |  |  |  |  |
| Supervised | 93.55 | 24.19 | 32.55 | 96.84 | 41.05 | 49.81 | 95.94 | 26.07 | 35.53 |
| Tree-PlannerN=25 | 95.16 | 38.71 | 63.18 | 96.08 | 51.58 | 69.45 | 95.50 | 40.38 | 63.75 |
| Tree-PlannerN=50 | 94.94 | 38.71 | 63.50 | 96.06 | 51.58 | 69.54 | 95.40 | 39.74 | 63.29 |
| Ours | 90.32 | 40.32 | 65.40 | 95.79 | 65.26 | 79.47 | 93.38 | 41.88 | 62.76 |

**Table 2: Performance on VirtualHome-Env with up to 10 corrections. Our planner consistently leads in SR and GCR performance. The Exec. metrics are shown in gray for the same reasons in Table 1.**
|  | Novel Scene and Task | Novel Scene | Novel Task |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Exec. | SR | GCR | Exec. | SR | GCR | Exec. | SR | GCR |
| GPT-3.5 API: |  |  |  |  |  |  |  |  |  |
| Local Replan | 79.66 | 37.46 | 51.90 | - | - | - | - | - | - |
| Global Replan | 82.09 | 37.93 | 52.46 | - | - | - | - | - | - |
| Tree-PlannerN=25 | 89.13 | 35.30 | 56.65 | - | - | - | - | - | - |
| Tree-PlannerN=50 | 88.26 | 41.58 | 59.55 | - | - | - | - | - | - |
| Finetuned Llama 3 8B: |  |  |  |  |  |  |  |  |  |
| SELF-REFINE | 96.77 | 43.55 | 65.18 | 92.63 | 54.74 | 70.24 | 94.44 | 39.96 | 62.37 |
| Tree-PlannerN=25 | 95.16 | 41.94 | 56.49 | 96.08 | 55.79 | 68.82 | 95.50 | 42.09 | 57.83 |
| Tree-PlannerN=50 | 94.94 | 43.55 | 58.91 | 96.06 | 58.95 | 70.00 | 95.40 | 43.38 | 59.79 |
| Ours | 91.94 | 56.45 | 76.63 | 97.89 | 77.89 | 87.07 | 92.31 | 54.91 | 74.18 |

### 4.2 Main Results

The experimental results in the two planning setups, without correction or with up to 10 corrections, are summarized in Tables 1 and 2.
Overall, our method achieves the leading performance on the majority of metrics. Specifically, the experimental results show that: (1) Even without error correction, our self-refining process still brings a significant improvement of 14% on SR in the novel scene subset, with other metrics superior or comparable to the previous leading method. (2) By incorporating environmental feedback, our approach improves all metrics by more than 11% and up to 19%, showing clear advantages. (3) Similar to the existing finetuning-based methods, our generated plans exhibit a high executability of over 90%, which can be improved to 99% by simply truncating illegal overlength outputs. These results clearly confirm the advantages of our approach.

Figure: (a) SELF-REFINE
Refer to caption: x4.png

Figure 4 further compares our self-correction process with the baseline methods. As can be seen, our method is better at incorporating environmental feedback to improve the plan, while the baselines fail by simply repeating the previous plan, or by making only local adjustments that are insufficient. This exemplifies the effectiveness of equilibrium sequence modeling for self-correction.

**Table 3: Effectiveness of different types of feedback. They are measured under the constraint of up to 10 rounds of internal or external feedback. The last two rows are reported in the main tables.**
|  |  | Novel Scene and Task | Novel Scene | Novel Task |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| World model | Env. | Exec. | SR | GCR | Exec. | SR | GCR | Exec. | SR | GCR |
|  |  | 88.71 | 33.87 | 59.98 | 96.79 | 49.47 | 66.60 | 93.80 | 34.62 | 59.06 |
|  | $\checkmark$ | 83.87 | 51.61 | 75.13 | 96.84 | 75.79 | 85.79 | 92.31 | 56.62 | 75.53 |
| $\checkmark$ |  | 90.32 | 40.32 | 65.40 | 95.79 | 65.26 | 79.47 | 93.38 | 41.88 | 62.76 |
| $\checkmark$ | $\checkmark$ | 91.94 | 56.45 | 76.63 | 97.89 | 77.89 | 87.07 | 92.31 | 54.91 | 74.18 |

Figure: (a) Scaling w.r.t. inference computation
Refer to caption: x7.png

Figure: (a) Effectiveness of reusing equilibrium
Refer to caption: x9.png

### 4.3 Ablation Study

Effectiveness of equilibrium sequence modeling is quantitatively verified in Table 2. Compared to alternative methods, our approach shows more than 11% improvement over the prompting-based self-refinement method SELF-REFINE (Madaan et al., ) and more than 12% improvement over the system 2-based Tree-Planner (Hu et al., ). Meanwhile, we maintain a simplistic supervised finetuning fashion similar to the compared methods, without intricate reinforcement learning. This validates the effectiveness of equilibrium sequence modeling in our robot task planning problem.

Effectiveness of various feedback. As can be observed in Table 3, incorporating external feedback from the environment or internal feedback from the world model consistently improves performance. Even though the world model does not provide as much improvement as the real environment, it also increases performance by over 3%. In particular, the synergy of both types of feedback yields the highest performance on most of the metrics, further confirming their effectiveness. In the following analysis, we will focus on our method using only environmental feedback for simplicity.

Scaling of performance. Here, we follow Brown et al. (); Snell et al. (); Wu et al. (); OpenAI () in considering the scaling w.r.t. inference computation. The results in Fig. 5(a) show that our method achieves better performance-computation tradeoff along with leading scaling w.r.t. inference computation. Thus, more inference budget can be allocated to improve its performance. Furthermore, in Fig. 5(b), we show that its performance advantage is largely due to better long-horizon planning capabilities, achieving more than twice the success rate for extremely long plans.

Computational efficiency. Although our training is slower than the baselines (36h vs. 12h) due to the equilibrium solving process for synthesizing training pairs ($\approx$24h), it exhibits a competitive inference efficiency. For example, our method takes 16h to evaluate, while Tree-Planner takes 24h. This can be attributed to our design of reusing equilibrium in nested equilibrium solving, As illustrates in Fig. 6(a), it accelerates the convergence of feedback iterations, achieving better performance ($>$55%) with significantly fewer interactions. Furthermore, we show in Fig. 6(b) that our method is able to dynamically allocate inference computation for planning tasks of different complexity.

## 5 Conclusion

This work proposes an equilibrium model-based LLM planner that is capable of self-refining plans from external and internal feedback. Unlike existing self-refinement methods based on prompting or sophisticated reinforcement learning, our proposed equilibrium sequence modeling allows simple supervised training of the self-refining planners. Moreover, it also enables the planner to efficiently incorporate environmental feedback or a world model for closed-loop planning. We implement the proposed approach on the VirtualHome-Env benchmark, and the experimental results suggest that it can dynamically allocate inference computation to achieve state-of-the-art planning performance.
Code: [https://github.com/Singularity0104/equilibrium-planner](https://github.com/Singularity0104/equilibrium-planner).

Acknowledgements.
We thank Xingjian Bai and Liyuan Wang for their helpful discussions.

## Appendix A Proof of Implicit Function Theorem

###### Theorem 1

(Implicit Function Theorem (Bai et al., ; Krantz & Parks, ))
Let $L:\mathbb{R}^{n}\times\mathbb{R}^{n}\to\mathbb{R}$ be a differentiable loss function, and let $f_{\theta}:\mathbb{R}^{n}\times\mathbb{R}^{p}\to\mathbb{R}^{n}$ be a differentiable function parameterized by $\theta\in\mathbb{R}^{q}$. Consider the following optimization problem:

$$ $\displaystyle\min_{\theta}$ $\displaystyle L(x^{*},y)$ (11) s.t. $\displaystyle x^{*}=f_{\theta}(x^{*},c).$ $$

where $x^{*},y\in\mathbb{R}^{n}$, and $c\in\mathbb{R}^{p}$.
If $\big{(}I-\frac{\partial f_{\theta}}{\partial x^{*}}\big{)}$ is invertible, then the loss gradient w.r.t. $\theta$ is given by:

$$ $\frac{\partial L}{\partial\theta}=\frac{\partial L}{\partial x^{*}}\left(I- \frac{\partial f_{\theta}}{\partial x^{*}}\right)^{-1}\frac{\partial f_{\theta }}{\partial\theta}.$ (12) $$

Proof of Theorem 1. To derive the loss gradient w.r.t. $\theta$, we begin by differentiating the equilibrium condition $x^{*}=f_{\theta}(x^{*},c)$ with respect to $\theta$. Applying the chain rule, we have:

$$ $\frac{\partial x^{*}}{\partial\theta}=\frac{\partial f}{\partial\theta}+\frac{ \partial f}{\partial x^{*}}\frac{\partial x^{*}}{\partial\theta}.$ (13) $$

Given that $\big{(}I-\frac{\partial f_{\theta}}{\partial x^{*}}\big{)}$ is invertible, we can rearrange the above equation and solve for $\frac{\partial x^{*}}{\partial\theta}$:

$$ $\frac{\partial x^{*}}{\partial\theta}=\left(I-\frac{\partial f_{\theta}}{ \partial x^{*}}\right)^{-1}\frac{\partial f_{\theta}}{\partial\theta}.$ (14) $$

The chain rule implies $\frac{\partial L}{\partial\theta}=\frac{\partial L}{\partial x^{*}}\frac{
\partial x^{*}}{\partial\theta}$. Substituting the expression for $\frac{\partial x^{*}}{\partial\theta}$, we obtain:

$$ $\frac{\partial L}{\partial\theta}=\frac{\partial L}{\partial x^{*}}\left(I- \frac{\partial f_{\theta}}{\partial x^{*}}\right)^{-1}\frac{\partial f_{\theta }}{\partial\theta}.$ (15) $$

$\square$

###### Theorem 1

## Appendix B Experimental Settings

Figure: (a)
Refer to caption: extracted/5899050/figs/env_exp_a.jpg

Figure: (a) Distribution of plan length
Refer to caption: x11.png

### B.1 Benchmark

Environment.
We adopt the robotic planning benchmark VirtualHome-Env (Liao et al., ) based on VirtualHome (Puig et al., ). It consists of a complex set of 292 planning tasks in 7 different indoor scenes, provided with 1360 mid-level action trajectories as ground truth annotations. These action trajectories are typically very long, with an average execution length of 10.8, highlighting its long-horizon characteristic. Moreover, the VirtualHome environment provides detailed feedback after performing each mid-level action, making it an ideal testbed for closed-loop planning.

Figure 7 visualizes a few examples sampled from the VirtualHome-Env benchmark. In each example, the planner is placed in an environment that spans a few indoor rooms and is given a detailed description of the environment. The description is originally in the form of a scene graph with objects as nodes and spatial relationships as edges, but for simplicity we present the planner with only the object nodes. The planner is then asked to generate a semantic action sequence based on a short task description. For instance, after receiving an instruction “turn on TV …”, the robot agent must first walk to the table and grab the remote control, and then point at the TV to turn it on. As can be seen, these planning tasks usually involve a rather complex scene setup, and the ground truth action sequences are quite long. We provide more detailed statistics for this benchmark in Fig. 8.

Compared to alternative embodied planning benchmarks, VirtualHome-Env features long time horizons as well as more diverse closed-loop feedback. For example, ALFRED (Shridhar et al., ) and ReALFRED (Kim et al., ) are two common embodied instruction following benchmarks, but their plan lengths are relatively short and can be determined by a few templates, making them unsuitable for long-horizon planning. PlanBench (Valmeekam et al., ) and TravelPlanner (Xie et al., ) are recent benchmarks designed specifically for LLM planning, but they do not provide closed-loop feedback during execution, which is an essential element of robotic planning. Therefore, we adopt VirtualHome-Env during the experiments, with further details presented below.

Action.
The VirtualHome environment (Puig et al., ) originally supported animating 12 atomic actions based on the Unity simulator, with the followup work VirtualHome-Env (Liao et al., ) adding support for more actions using a graph simulator. It currently supports 40 atomic actions, in which 21 actions can be animated through Unity. Each action is defined by an action name and some object arguments, and is implemented by prewritten code executors. In our experiments, we use a full set of 40 actions included in the VirtualHome-Env dataset, summarized as follows:

- 1.
Actions without object association: SLEEP, STANDUP, WAKEUP.
- 2.
Actions associated with one object: WALK, FIND, GRAB, WASH, WIPE, PULL, PUSH, POUR, TURNTO, POINTAT, WATCH, TOUCH, OPEN, CLOSE, RUN, SIT, READ, PUTON, PUTOFF, DROP, LIE, SWITCHON, SWITCHOFF, DRINK, LOOKAT, TYPE, CUT, PUTOBJBACK, EAT, RINSE, PLUGIN, PLUGOUT, GREET, SCRUB, SQUEEZE.
- 3.
Actions associated with two objects: PUTIN, PUTBACK.

Feedback.
Because the environment includes a graph simulator of the scene graph, it can respond quickly to actions, *e.g*. changing object attributes, and provide the updated scene graph at each step. In our experiments, we curate several types of closed-loop feedback based on these scene graphs, simulating coarse feedback that may be received in real-world situations. Specifically, we consider the following four categories of environmental feedback associated with task failure:

- 1.
Program format feedback:“Your output does not conform to the required format”, meaning that the generated action sequence does not conform to the required format.
- 2.
Invalid command feedback: “Your output has an invalid command: …”, meaning that the generated action sequence has an illegal command line.
- 3.
Execution feedback: “Your output is executed incorrectly in the environment.”, meaning that the generated action sequence cannot be executed in the environment.
- 4.
Task completion feedback: “You have not completed this task. The following objects and corresponding states do not meet the goals: … The following objects have wrong relative position: …”, meaning that the generated action sequence cannot complete the task, with more details about the task failure.

Dataset Split.
We randomly divide the VirtualHome-Env dataset into training set and test set in a 50:50 ratio. To analyze the generalizability of our method, we mainly study the following three subsets of the test set: novel scene set, novel task set, and novel scene and task set. For instance, the novel scene set consists of seen planning tasks on unseen scenes. Overall, the dataset contains 735 training trajectories, 468 trajectories within the novel task set, 95 trajectories within the novel scene set, 62 trajectories within the novel scene and task set. Our models are first trained on the training set for a fixed number of epochs and then evaluated on the three test subsets above.

### B.2 Baselines

Our method is mainly compared with Tree-Planner (Hu et al., ) and SELF-REFINE (Madaan et al., ), both reproduced using Llama 3 8B Instruct (Dubey et al., ) in line with ours. The former traverses an action tree that is built by repeated plan sampling, while the latter relies on self-refinement. To reproduce them, we perform supervised finetuning of Llama 3 on the training split of VirtualHome-Env for the same number of epochs as our method, and then follow their original procedures for inference. For instance, Tree-Planner is reproduced with both settings $N\in\{25,50\}$ in action tree construction. The system prompts they use are similar to ours in Fig. 9.

We also report the results summarized by Hu et al. () for reference. They additionally considered ProgPrompt (Singh et al., ), Zero-shot Planner (Huang et al., ) and two self-refinement planners, Local Replan (Raman et al., ; Guo et al., ) and Global Replan (Shinn et al., ). Since these baselines were implemented by calling the GPT-3.5 API instead of finetuning Llama 3, we report them in the novel scene and task track for a relatively fair comparison. It is worth noting that they adopted a smaller subset of actions and feedback, and differed in the curation of partial observations of the environment. Therefore, their results are for reference only.

There are several robotic planning baselines that we have not compared due to large environmental differences. For example, LLM-MCTS (Zhao et al., ) is a representative tree-search (Yao et al., ) based planner. It followed Watch-and-help (Puig et al., ) to generate a dataset of simple embodied tasks (mostly object rearrangement tasks), while our work considers a more complex set of planning tasks. Alternative planners based on symbolic scene graph (Zhu et al., ; Rana et al., ), code (Liang et al., ; Sun et al., ), or PDDL (McDermott, ; Liu et al., ; Guan et al., ) are less flexible and difficult to implement in our environment.

Figure: Figure 9: Example of the prompt used by our equilibrium planner.

### B.3 Implementation details

Prompt.
Our approach involves two LLMs, one LLM as the equilibrium planner and an additional LLM as an optional world model. The planner’s prompt is illustrated in Fig. 9. As can be seen, it consists of system prompt, task definition, environment description, history feedback, and draft plan. Notably, the system prompt is modified from Hu et al. (), and the environment section describes the initial environment sorted by rooms, including the object names with their IDs within each room. For the additional world model, we adopt a similar prompt, except that it receives more information about the initial environment, including edges in the scene graph that correspond to spatial relations The world model is then asked to predict the environmental feedback given a generated plan.

Finetuning.
Both our equilibrium planner and the world model are finetuned in a supervised manner. The equilibrium planner is finetuned for 6 iterations with a learning rate of 0.0002. At each iteration, we curate training data by pairing fixed points and ground truths, where the fixed points are sampled from an equilibrium memory of past iterations. To prevent overfitting to the history equilibrium, a decay ratio of 0.5 is used when sampling from the fixed points of previous iterations. Thereafter, we update the model parameters using gradient descent according to Eq. 6 for one epoch per iteration. For the world model, we collect all interacting experiences between the planner and the environment, including plans and feedback, and finetune it for 5 epochs using the same learning rate of 0.0002.

Inference.
The inference procedure of our planner was described in Algorithm 1, which involves a nested equilibrium solving process. Given the environment and the task instruction, we initialize the draft plan $x_{0}$ and the feedback $c_{0}$ as null and iterate through a nested loop. Each inner loop reuses the feedback from the outer loop to self-refine the draft plan, and after it converges, we update the feedback by interacting with the environment or world model. This process continues until it reaches an upper bound on the outer loop, which we set to 10 to match Tree-Planner (Hu et al., ). We adopt greedy sampling for the LLM, except that the first refinement step uses top-k sampling with $k=10$ for higher diversity. Since most of the prompt remains unchanged during equilibrium solving, we implement KV cache to accelerate the inference process, which can be further improved with parallel decoding techniques (Santilli et al., ; Cai et al., ; Kou et al., ).

Figure: (a) SELF-REFINE
Refer to caption: x13.png

Figure: (a) Ours w/o external feedback
Refer to caption: x16.png

## Appendix C Additional Results

To further illustrate the effectiveness of equilibrium sequence modeling in closed-loop planning, we additionally exemplify our self-correction process on a long-horizon robotic planning task in Fig. 10. Compared to SELF-REFINE (Madaan et al., ) and Tree-Planner (Hu et al., ), our approach is more competent in revising a long plan through few forward passes without additional system 2. This is attributed to our efficient training scheme for teaching planners to self-refine.

We also compare different types of feedback utilized by our planner in Fig. 11. As can be observed, internal feedback alone cannot enable successful replanning, but it can reduce environmental interactions prior to convergence. This confirms the effectiveness of both internal and external feedback in closed-loop planning and thus justifies the framework design of our equilibrium planner.

## Appendix D Limitations

While our equilibrium sequence modeling improves the planning capability of LLMs, we identify the following failure scenarios during the experiments: (1) hallucination of the equilibrium planner and the world model as in vanilla LLMs; (2) lack of awareness of history context such as previously grabbed objects. The latter may be resolved with reasoning techniques as in Yao et al. ().

In a broader sense, our method may be limited in generalizing to new domains because it requires the ground truth and environmental feedback during training. These procedures with the equilibrium solving process results in lower training efficiency. Also, the current formulation only considers the explicit output plan without implicit reasoning steps. Furthermore, our model has only text input and no visual input, which limits its applicability in the real world. This can be resolved by introducing video-based planners (Du et al., ) and world models (Yang et al., ; Brooks et al., ).

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. The authors cite [1] as evidence that self-refinement via prompting tends to not lead to successful error corrections. Figure 4 shows an example of error correction by the proposed method with environment feedback (i.e., with both the inner and outer loop). Does self-refinement via fixed point iteration (i.e., only running the inner loop, making it more similar to the setting of [1]) lead to better error correction performance than the approaches studied in [1]? If so, then it would be nice to see some empirical evidence of this or a few sample inference traces.
2. For the experimental results presented in table 2, what is the ratio of the number of calls to the synthetic world model to the number of environment interactions?
3. How does the dynamic allocation of inference compute presented in figure 6b work? What decides how many inference forward passes to take?
4. Is the dataset used for training constructed once, or adaptively generated from the newest solutions to the equilibrium problem?
5. Why can the proposed approach improve better from environment feedback than baselines?

[1] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In International Conference on Learning Representations, 2024.
soundness: 3
strengths: 1. As the authors mention in their introduction, there is indeed a natural match between the problems faced by task planning (and more generally long-horizon reasoning) and the benefits posed by deep equilibrium models. There appears to be substantial originality in the authors' identification of this match, and their subsequent application of deep equilibrium models to long-horizon task planning.
2. The experimental evaluations are fairly thorough, with many ablation studies I wanted to see present.
3. The manuscript is clear and well written.
summary: This paper proposes formulating long-horizon task planning with LLMs as a fixed point problem, arguing that such a formulation leads to a natural realization of many desirable properties of task planning, including global context when deciding the plan, the ability to recover from errors in drafts of the plan, and the ability to scale inference time compute to yield better plans. The authors draw from the literature of deep equilibrium models [1] to instantiate their idea, proposing a training framework based off of the implicit differentiation algorithm proposed by [1]. The full framework proposed by the authors includes a nested inference algorithm, with an inner loop that solves the fixed point problem for a static context, and an outer loop that sends the plan to the environment for feedback. In the case that the environment cannot be queried online, the authors propose training a world model to simulate environment feedback. In comparisons with baselines, the proposed approach performs the best on the VirtualHome-Env benchmark. 

[1] Bai, Shaojie, J. Zico Kolter, and Vladlen Koltun. "Deep equilibrium models." Advances in neural information processing systems 32 (2019).
weaknesses: 1. The first row in table 3 seems to correspond to the setting in which no correction is allowed and the synthetic world model is not used (so only the inner loop is run). The results obtained by the proposed algorithm in this setting are however lower than the Tree-Planner baseline results from table 1. What explains this low performance? More generally, why is the synthetic world model needed, and could baseline approaches benefit as well from such a world model?
2. Although prior work has solely evaluated on the VirtualHome-Env benchmark, the results would be more solidified if the authors had chosen an additional benchmark to evaluate on.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: How practical it is to run an LM to reach a fixed point? How sensitive is convergence with respect to the initial sequence?

How would your method perform without fine-tuning Llama model?

The authors suggested, "they divided the dataset into a training set and three test subsets, including the novel scene set, the novel task set, and the novel scene and task set." Since VirtualHome is a popular benchmark, have you surveyed prior papers about their partitioning of the tasks? Have you attempted using a more conventional partitioning used by prior works?
soundness: 1
strengths: The paper is well written. The proposed concept is relatively simple and straightforward. The math is easy to follow. The intuition of self-refinement is grounded.
summary: The paper proposes a new LLM planning framework via equilibrium models. The goal is to build a long-horizon task planner that can incorporate feedback from the environment as well. The authors start with the limitations of prior LLM planners, such as the lack of bidirectional dependency, and lack of internal error correction. Built on these insights, the authors propose to make LLM planners iterative like equilibrium models. The direct training objective of the equilibrium model is intractable, so the authors proposed a simplified objective built on prior works in theory. The simplified objective involves running the LLM iteratively to a fixed point and imposing supervised training loss only on the fixed point. An associated planning framework is then proposed using the fine-tuned LLM. The authors present empirical results on VirtualHome and compared against previous LM planning frameworks.
weaknesses: My strongest criticism of of paper is the core math it builds on: the authors say one can simplify Eqn(4) to Eqn (5) per Theorem 3.1 in Fung et al. (2022) under appropriate assumption. Yet the "appropriate assumption" is completely unsatisfied here which makes the entire math perspective unfounded. Specifically, Theorem 3.1 in Fung et al. (2022) on their Assumptions 2.1, 3.1, 3.2, and 3.3; Let's take Assumption 2.1 for example, the mapping used in equilibrium model ($T_\theta$ in Eqn (6) of Fung et al. (2022) and $f_\theta$ in the reviewing manuscript), must be Lipschitz with respect to input. This is absolutely ungrounded for transformer-based LLMs. In fact, ML theory papers always devote a great amount of effort to circumvent this problem just because of how unacceptable it is to assume this transformer. This is not to mention assumptions 3.1, 3.2, and 3.3 in Fung et al. (2022). Therefore, you are not even making a strong assumption, but an assumption that's widely considered wrong for LLMs.

Therefore, I urge all reviewers to review the paper by ignoring the flawed math explanation and examining its merit from its empirical merit. 

The technical contribution becomes very limited if we proceed to ignore flaws in the training objective. The feedback, iterative planner seems to be an obvious solution even considering the environmental feedback perspective. 

I think one straightforward way for the authors to address the flawed training objective is to conduct ablations that uses your planning framework but with original Llama3 8B that's not fine-tuned. This is no way making the math correct but can inform us about the performance of your method without a unfounded training objective. 

I also wonder whether the training procedures in line 204 and 205 are grounded. Specifically, how practical it is to run an LM to reach a fixed point? It seems to me that in a high-dim space like language token sequences, it's really hard to achieve convergence. Is it because the authors specifically prompted it to do so?

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 1
questions: Please refer to the weaknesses section.
soundness: 2
strengths: If I have understood correctly, the core problem tackled by the paper is really crucial. A lot of approaches are using LLMs as high-level planners. However, multiple evaluations have shown that LLMs are unable to reason over long horizon. With the proposed approach of self introspection, LLMs can be used to aide the planning while *verifying* the LLM generated plan until the equilibrium is reached.
summary: This paper introduces a novel approach to long-horizon robotic task planning by addressing limitations in current large language model (LLM) planning agents, which often struggle with foresight and error correction. The authors propose an equilibrium sequence modeling technique that iteratively refines a plan until it reaches a stable state, or equilibrium, without requiring additional reward models or verifiers. This approach allows for efficient closed-loop planning by incorporating environmental feedback, enhancing the robot's ability to plan and adapt in complex tasks. Through self-refinement, the method improves long-horizon action prediction and reduces computation by reusing equilibrium states, resulting in superior performance on the VirtualHome-Env benchmark.

The model integrates a nested feedback system, which combines both internal (world model) and external (environmental) feedback to guide plan refinement, achieving improved executability and success rates compared to existing models. Unlike reinforcement learning-based methods, this equilibrium-based framework is trained in a supervised manner, simplifying training while enabling high computational efficiency. The results demonstrate that the proposed approach outperforms baseline methods on various metrics, indicating its potential for more robust and adaptive robotic task planning in dynamic environments.
weaknesses: I do not have many specific weaknesses of the paper. The major reason for that (which is the biggest weakness and reason for my overall rating) is that the paper is extremely  hard to follow. I was not able to understand the paper properly even after reading it multiple times. As I mentioned in the summary of the paper, I was able to get the overall idea but the details of the approach are still unknown to me. However, here following are some ways in which the paper can be made readable: 

- The paper is not self sufficient. It heavily relies on previous works about equilibrium models and fixed-point iteration methods. Readers who are not familiar with these previous works would fine nearly impossible to read the paper. The paper can be updated by having Background section that discusses these methods. 
-  I did not understand how the self refinement with a fixed $c$ works? If the there is no feedback from the environment, how does the approach know even if the generated plan is correct or not in the first place. 
-  There seems to be some distinction between the "training" and "equilibrium solving via fixed point iteration" but they also seem to be connected. It is not clear from the paper how are they different and how are they connected. 
- How this is exactly connected with LLMs is not clear. What part of the modules are using LLMs are not clear from the paper. Can authors comment on it and make it clear in the paper? 
- How does nested refinement work if the context is similar for the inner loop without any feedback? How does the equilibrium model know what to change between two plans? 
- What are the inputs? What are the assumptions? This needs to be made clear in the paper. 

In general, IMO the writing of the paper needs to be self-sufficient. Which the paper is currently not. Above I have tried to outline some of the points that can be used to update the readability of the paper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Are there any iteration-wise results available to show the subsequent effects of feedback over time compared to the other approaches?
2. How is the world model initialized using an LLM?
3. What is the frequency of switching between real experience and synthetic world-model generated experience?
4. Minor typo: L293: “May to easier to train”
soundness: 2
strengths: - The introspection inspired approach provides a significant benefit over prompting based methods for similar planning tasks.
- The approach is intuitive and provides a unique perspective on solving planning problems via equilibrium methods.
summary: This work proposes a self-refining framework for robotic task planning using language model agents. By iteratively refining draft plans until equilibrium, this method enables efficient closed-loop, long-horizon planning. Utilizing a nested equilibrium process, the approach effectively integrates environmental feedback, enhancing task performance and inference efficiency. Evaluated on the VirtualHome-Env benchmark, it demonstrates increased scalability in robotic planning scenarios, supported by an equilibrium-based training method that allows end-to-end learning in a simple supervised manner.
weaknesses: While an inference study of computation is presented, it is unclear whether the training budgets of the competing approaches are fairly represented in the context of the finetuning costs of the world model for the proposed method.

Meta-review: The paper proposes an approach to closed-loop planning for long-horizon robotic tasks using equilibrium sequence modeling and LLM-based self-refinement. While the empirical results are compelling, the reviewers identified several significant concerns.

Strengths:

- The paper tackles the critical problem of long-horizon planning with LLMs.

- Innovative use of equilibrium models for plan refinement.

Weaknesses:

- Reviewers share valid concerns about flawed mathematical justifications. The reliance on unsupported assumptions diminishes the strength of the theoretical contribution.

- Some reviewers noted that the paper is challenging to follow for those unfamiliar with equilibrium models. Improved explanations and a self-contained presentation are necessary.

- Questions remain about the training cost, dataset partitioning, and the mechanisms for integrating feedback.

The paper shows great promise but requires substantial revisions to address theoretical concerns, improve clarity, and solidify its contributions. I would recommend rejection in its current form.

Decision: Reject