InteractiveCOT: Aligning Dynamic Chain-of-Thought Planning for Embodied Decision-Making

Authors: Kechen Jiao, Zhirui Fang, Jiahao Liu, Bei Li, Zhongjian Qiao, Yaxin Xu, Yifan Zhu, Xinyu Liu, Jingang Wang, Xiu Li
Primary area: applications to robotics, autonomy, planning
Keywords: Embodied Agent, Multi-modal Large Model, Chain-of-Thought, Planning, Reinforcement Learning, Preference Learning

Abstract:
Vision-Language Models (VLMs) are increasingly being employed as the decision-making "brains" of embodied agents. Effectively harnessing their powerful generalization capabilities in dynamic, context-specific tasks remains a significant challenge. Chain-of-Thought (CoT) prompting is often utilized for complex task execution, but existing methods either rely on static strategies that fail to adapt to changing environments or fine-tune on offline datasets, which are insufficient for optimizing agent decision-making through interaction.
In this paper, we propose a novel approach that focuses on optimizing the CoT reasoning process rather than just the final action tokens. By aligning the CoT process through preference-based reinforcement learning, specifically Direct Preference Optimization (DPO), we enhance the agent's ability to make accurate decisions in dynamic environments while mitigating model degradation during fine-tuning. Our method models the environment as a Markov decision process, requiring the agent to reflect on the current state in real time to generate adaptive plans and actions.
By prioritizing the optimization of the CoT process over the final actions, we enhance the agent's reasoning adaptability while effectively mitigating model degradation during fine-tuning.
Experiments in the ALFWorld environment demonstrate an average success rate of \textbf 26.67%, which is a 6\% improvement over RL4VLM, and show that our method effectively mitigates model degradation post fine-tuning. These results highlight the potential of integrating preference-based reinforcement learning techniques with CoT processes to enhance the decision-making capabilities of vision-language models in embodied agents.

Decision: reject
Reviewer scores: 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1730200184755

thread_type:
official_review

<thread_object>
creator: reviewer_fl8f

<content>
summary:
This paper introduces InteractiveCOT, a method for fine-tuning a vision-language model (LLaVA-7B) for embodied AI tasks. The authors aim to enhance RL4VLM by optimizing the chain-of-thought (CoT) process rather than focusing solely on final action decisions. Using the Direct Preference Optimization (DPO) algorithm, they fine-tune the vision-language model with online-collected episodes. The proposed method is evaluated in the ALFWorld environment, demonstrating improvements over the PPO baseline.

strengths:
- Utilizing foundation models as a starting point for embodied agents is an interesting approach, as it can enhance sample efficiency and reduce redundant exploration by leveraging the prior knowledge embedded in these models.
- InteractiveCOT achieves a 6% improvement in the final success rate by focusing more on optimizing the CoT process during VLM fine-tuning.

weaknesses:
- The paper's presentation is difficult to follow, with numerous typos and inconsistencies in notation. For example:
    - Line 84, "In summery" -> "In summary".
    - In Figure 1, "LLaVA as dicision model" -> "LLaVA as decision model."
    - Line 215, "donate" should be "denote"; additionally, $\pi_{ref}$ is duplicated.
    - The definitions of subscripts and superscripts for action (i.e., $a_t^1$ and $a_t^2$) in line 245 and in Equations (4), (6), (7), (8), and (9) are inconsistent.
    - Line 213 references the tuple with $o_t$, but it is unclear where $s_{1 \sim t-1}$ originates.

- The authors should include a background section to introduce the basic RL framework, including elements of the MDP, trajectories, and policy, to clarify the RL context being considered. Without this, it is difficult to follow the subsequent sections. Additionally, a brief overview of the original DPO algorithm should be provided so that modifications proposed in the methods section are clearly distinguishable. 

- In Section 3.1, the authors state that the VLM is used as a planner; however, it is unclear how the plan is generated. It appears that the VLM functions directly as a policy, outputting final actions to step into the environment, as illustrated in Figure 1. Thus, it may be misleading to frame the proposed method as a "re-planning framework" (line 197). Can author clarify this point?

- What type of action space does the paper consider? Is it continuous or discrete? If it is discrete, how is the MSE calculated in Eq. (7)?

- In line 201, what does "well-fine-tuned model" refer to? Is this the VLM fine-tuned by the proposed method?

- Throughout the paper, what does $\tau_t^{t-1}$ represent?

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709997325

thread_type:
official_review

<thread_object>
creator: reviewer_r5ga

<content>
summary:
The paper proposes a preference based fine-tuning method for a VLM on sequential decision making tasks. They look at Alfworld tasks and show a 6% improvement over the baseline RL4VLM (which looks at RL based finetuning of VLMs). They add auxiliary losses (APW and APC) to encourage COT - action consistency in the output form the VLM.

strengths:
How to benefit from and optimise chain of thoughts in LLM-derived models for decision-making is an interesting question which the paper considers.

weaknesses:
The paper is very poorly written:
1) Many implementation details are left to the imagination, there isn’t a code implementation provided either. 
2) The supplementary material only contains a single page derivation and no experimental details. 
3) Lots of typos and confusing wording throughout the paper
4) Lines 214, 215, what is meant by “πref refers to the policy generated by the unrefined output of VLM? Donate → Denote 
5) πθ and πref donate the policy generated by VLM - not clear enough
6) Line186  - typos? - For once(?) planning instance, planner output the …
7) Fig 1 : dicision -> decision
8) In many places there is mention of a “Preference action” - I suppose the authors mean to say “preferred action”
9) Fig 1 is not very clear - what is the circle trying to outline in the “preference action” and “not preference action”? 
10) The subscripts in the equations are not defined anywhere
11) Line 222 typo in the section heading
12) The contributions claim interactiveCOT supports both PPO and DPO- where is this validated? Isn’t the paper only presenting DPO results?
13) Are their pairs action level or trajectory level
14) Point 2 in contributions: “tailored to the interaction characteristics of embodied agents, “ - what does this mean?

Lack of thoroughness in experiments:
1) The paper doesn’t mention what exactly is the reward function used by the PPO baseline? The paper mentions using partial task progress signal (which is much denser than a 0/1 success signal indicating whether the episode was successful) to construct preference labels - did the baseline (PPO) also benefit from this extra information? If not, why? It should be possible to convert the partial progress to rescaled rewards.
2) Fig 6a both variants seem the same in terms of the max performance resached yet the text describes one variant (w action weighting) as better than the other
3) No information about the hyperparameters is provided, no information on how many seeds were used, fig 4b does not even have error bars - was only one seed used per task??
4) How was the checkpoint used to report the final score chosen? Fig 4a has a larger gap between the two methods at 2k steps (6% as claimed by the authors), but the gap would be smaller (by around 2%) if you used the ckpt for PPO at 1k training steps - which would reduce the improvement to 4%. 
5) How exactly were the trajectories collected?
6) No other baselines are studied apart from RL4VLM:  the RL4VLM paper cited studies a CNN+RL baseline, as well as a prompting-only LLM baseline. It would also make sense to compare to the baseline where the authors train a VLM with DPO directly for action selection (as done for the VLM-RL case in [1]).
7) What is the reason/motivation for using the particular preference score definition in eqn 10? Why 50 * progress  specifically?
8) 454 line - unclear, and where is this demonstrated?
9) Line 426 - extremely vague description of scores computation (“In practice, considering the achievement of long-term goals, we calculate preference scores using a method similar to discount factor weighting in reinforcement learning returns”)

[1] Large Language Models as Generalizable Policies for Embodied Tasks

questions:
I have listed the questions in the bullet points of the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731190252779

thread_type:
official_review

<thread_object>
creator: reviewer_mhuu

<content>
summary:
This paper used Direct Preference Optimization (DPO) with a Vision Language model that uses Chain-of-Thought (COT) prompting for task planning of embodied agents. The proposed method optimized the COT process instead of the final action as in PPO. The experiment showed the proposed method outperformed RL4VLM in terms of the average success rate on ALFWorld dataset.

strengths:
- Replacing PPO in RL4VLM with DPO can increase average success rate.

weaknesses:
- The contribution is incremental. The main contribution of this paper is replacing PPO in RL4VLM (which also has COT) [1] by DPO [2] with necessary adaptions.

- The motivation that the authors want to use DPO is that it could be beneficial in the cases of long action sequences, partial occulsion and multi-tasking. However the results do not show any of these scenarios. It's not clear whether DPO performs better thanPPO in these scenarios.

- More analyses are needed to show the performance gain on ALFWorld, details see comments below.


[1] Zhai, Yuexiang, et al. "Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning." arXiv preprint arXiv:2405.10292 (2024).
[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).

questions:
- I'm wondering why the proposed method works much better on the task Heat & Place and Examine in Light, while worse on Pick & Place. Can authors provide more insights on these?

- What will the proposed model perform without COT prompting? Since the proposed method re-plan the task at interactive steps, is COT helpful in this case? I'm interested to see the ablation study with and without COT prompting.

- Intuitively, re-planning should work better when the action sequence is long, and the closer towards the end of the action sequence, the more accurate and less uncertain the planned actions should be. Does the proposed method behave like this? 

- There are typos and grammar mistakes in the text and figure, please check the spelling.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732676681363

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to the discussion

comment:
Dear Reviewers,

We sincerely appreciate the time and effort you've dedicated to reviewing our paper. We understand that you have a busy schedule, and we are truly grateful for your valuable feedback. As the Author-Reviewer discussion phase is coming to a close soon, we wanted to kindly reach out to see if there are any additional questions or points you'd like to discuss. We are keen to ensure that our responses have effectively addressed your concerns, and we believe that further discussion could be beneficial. 

We would greatly appreciate the opportunity to engage in any further discussion. Thank you once again for your thoughtful consideration.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732973129966

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Update the Manuscript

comment:
Thank you once again for all your efforts! We have updated our manuscript accordingly, and the revisions are summarized as follows:

- **Writing Quality**: We made extensive revisions to enhance the clarity and readability of the paper, ensuring all implementation details are thoroughly described and included a link to the code implementation for reference.

- **Typos and Wording**: We conducted a comprehensive review of the manuscript to correct all typos and unclear wording. For example, "donate" has been changed to "denote," and the phrase in line 186 has been clarified. Additionally, "dicision" in Figure 1 is corrected to "decision," and we have clarified the relationship between "preference action" and "not preference action." All instances of $\tau_t^{t-1}$ have also been corrected to $\tau_i^{t-1}$.

- **Definition of Equations and Symbols**: All subscripts in the equations have been defined in the text for better understanding, and the typo in the section heading on line 222 has been corrected.

- **Clarification of Contribution Claims**: We clarified the contributions of the method and the role of the DPO algorithm, and we will open-source our algorithm framework shortly. We also explained the specific meaning of "tailored to the interaction characteristics of embodied agents" to eliminate any ambiguity.

- **Preference Score Definition**: A more detailed explanation of the motivation behind the preference score definition in Equation 10 has been provided, clarifying the choice of 50 * progress specifically.

- **Trajectory Collection and Score Computation**: We elaborated on the process of trajectory collection and clarified the scoring computation description to ensure understanding.

We hope that these revisions address your concern. As the discussion phase is concluding, we kindly ask you to read our response and make a reconsideration of our work. Please feel free to contact us if there are any points you'd like to discuss further. Thank you for your time and attention!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper received ratings 5,3,3, which are all below the acceptance threshold. The reviewers raised various concerns regarding missing implementation details, lack of thoroughness in the experiments, missing details required to understand the paper, incremental contribution, and the need for more analysis. The authors provided a rebuttal, but the reviewers did not find the responses convincing enough (more details below). The AC is in agreement with the reviewers and recommends rejection.