PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation
Authors: Fei Ni
Jianye HAO
Shiguang Wu
Longxin Kou
Yifu Yuan
Zibin Dong
Jinyi Liu
MingZhi Li
Yuzheng Zhuang
YAN ZHENG
Venue: NeurIPS 2024 poster (2024)
Keywords: robotics manipulation
multi-modal LLM
image generation
instruction following
Primary area: generative_models
TL;DR: We propose PERIA, a novel framework that leverages a MLLM to enable effective task decomposition via holistic language planning and vision planning for robotic manipulation in embodied scenarios.

Abstract: Long-horizon manipulation tasks with general instructions often implicitly encapsulate multiple sub-tasks, posing significant challenges in instruction following.
While language planning is a common approach to decompose general instructions into stepwise sub-instructions, text-only guidance may lack expressiveness and lead to potential ambiguity. Considering that humans often imagine and visualize sub-instructions reasoning out before acting, the imagined subgoal images can provide more intuitive guidance and enhance the reliability of decomposition. Inspired by this, we propose **PERIA**(**PE**rceive, **R**eason, **I**magine, **A**ct), a novel framework that integrates holistic language planning and vision planning for long-horizon manipulation tasks with complex instructions, leveraging both logical and intuitive aspects of task decomposition.
Specifically, we first perform a lightweight multimodal alignment on the encoding side to empower the MLLM to perceive visual details and language instructions. 
The MLLM is then jointly instruction-tuned with a pretrained image-editing model to unlock capabilities of simultaneous reasoning of language instructions and generation of imagined subgoals. Furthermore, we introduce a consistency alignment loss to encourage coherent subgoal images and align with their corresponding instructions, mitigating potential hallucinations and semantic conflicts between the two planning manners.
Comprehensive evaluations across three task domains demonstrate that PERIA, benefiting from holistic language and vision planning, significantly outperforms competitive baselines in both instruction following accuracy and task success rate on complex manipulation tasks.

PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation
Fei Ni
Jianye Hao 1.2+
Shiguang Wu
Longxin Kou' Yuzheng Zhuang"
Yifu Yuan'
Zibin Dong'
Jinyi Liu'
Mingzhi Li
Yan Zheng
College of Intelligence and Computing, Tianjin University
Huawei Noah's Ark Lab
Abstract
Long-horizon manipulation tasks with general instructions often implicitly encap- sulate multiple sub-tasks, posing significant challenges in instruction following. While language planning common approach to decompose general instructions into stepwise sub-instructions, text-only guidance may lack expressiveness and lead potential ambiguity. Considering that humans often imagine and visualize sub-instruction reasoning out before acting, the imagined subgoal images can pro- vide more intuitive guidance and enhance the reliability of decomposition. Inspired by this, we propose PERIA(PEreeie that integrates holistic language planning and vision planning for long- horizon manipulation tasks with complex instructions, leveraging both logical and intuitive aspects of task decomposition. Specifically, we first perform a lightweight multi- modal alignment the encoding side to empower the MLLM to perceive visual details and language instructions. The MILMM is then jointly instruction-une -unned with a pretrained image-editing model to unlock capabilities of simultaneous reasoning of language instructions and generation of imagined subgoals. Furthermore, we introduce consistency alignment loss encourage coherent subgoal images and align with their corresponding instructions, mitigating potential hallucinations and semantic conflicts between the planning manners. Comprehensive evalua- tions across three task domains demonstrate that PERIA, benefiting from holistic language and vision planning. significantly outperforms competitive baselines in both instruction following accuracy and task success rate on complex manipulation tasks. The details and visualizations are available the homepage.
Reason, Imagine, Act), a novel framework
Introduction
Recent advances in vision-language models (VLMs), such BLIP [1] and LIV [2], enable open- vocabulary visual recognition and multi-modal alignment, showing promise in robotic manipulation tasks specified human-provided language instructions [3, 4, 5, 6]. For semantically clear and concise instructions, such as "pick the red block on the green one", robotic agents can easily understand and complete the task in single step using action primitives. However, when instructions become more general and complex, such as "stack the blocks as pyramid and each layer in one color", the manipulation task can span long horizons and implicitly encapsulate multiple sub-tasks separated by action primitives, posing a major obstacle in instruction following. Current approaches often resort to decomposing complex instructions into manageable subtasks, either through language planning or vision planning based on the decomposed modality. Language planning, the more common approach, decomposes into progressive stepwise sub-instructions, which can be either predefined skill libraries
"Corresponding authors: Saanyehho@@e @ww..euu..uu..n. edu.cn. www.bennn@uuueuuuccc
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Steck in pyrered each
1. Put Medee betemn
shop fowands succesi
yellow bocko bocks
Perceive
Reason
Imagine
Act
Figure I:
Overview of PERIA (Perceive, Reason, Imagine, Act), inspired by the human cognitive process of following complex instructions, which involves perceiving environment and tasks, reason- ing the required language plans, and imagining the intermediate subgoal images before acting.
in natural language [3, 7] or latent codebooks [8]. On the other hand, vision planning, more recent development, decomposes complex instructions into coherent subgoal images as keyframes [9, 10], serving as visual milestones provide more intuitive and expressive guidance for action execution.
Language planning focuses on "how act" and the sub-instructions outline the necessary procedural action process of the task completion, emphasizing the temporal dependencies and causal relationships between decomposed stepwise sub-instructions. On the other hand, vision planning concentrates on "what to act towards" " and intuitive and grounded subgoal images with rich spatial and contextual information can enable robot agents to more easily understand what intermediate landmarks and visual anchors should achieve towards task completion. From cognitive perspective. humans rely on a symbiotic operation of the brain's ' hemispheres [11]. with the left primarily associated with logical language-basee based reasoning, and the right is linked to intuitive visual-based imagining. For humans, language planning and vision planning are often intertwined and performed simultaneously. involving either imagining the desired intermediate goals and then reasoning about the required plans to achieve them, or first reasoning out necessary stepwise plans and then imagining corresponding resulting images. Inspired by this, a natural question arises: Can we develop a framework that emulates this cognitive synergy by simultaneously performing language planning and vision planning for robolic manipulation lasks involving complex instructions just like humans?
For this, we propose PERIA(PErceive, Reason, Imagine, Act), a novel framework that integrates multi-modal large language model (MLLM) and diffusion model to enable language-based reasoning and visual- based imagining respectively, leveraging holistic language planning and vision planning for long- horizon manipulation tasks with general complex instructions. Specifically, MLLM's perception ability by fine-tuning the encoder side's 's projection layer to align the text and vision modalities in the LLM's hidden layers in lightweight manner, avoid the potential hallucina- tions and enhance the grounding ability. Next, we perform instruction tuning to simultaneously equip PERIA reasoning and imagination capabilities by explicitly adding additional image tokens after the reasoning phase and extracting rich latent image representations from MILMM to guide the generation of corresponding subgoal images. Moreover, We also introduce an alignment loss between reasoned sub-instructions and imagined subgoal images to enhance the consistency and accuracy of vision and language planning, jointly updated with generation and reasoning losses. In this way, vision planning provides a visualization of language planning, offering more intuitive guidance to avoid potential confusion. Language planning, in turn, provides reliable logical guidance at the semantic level for vision planning, preventing semantic conflicts in the generation of coherent image chains. The comprehensive evaluation across three typical long-how horizon manipulation tasks demonstrates that PERIA enjoy the accuracy of instruction following and synergistic combination significantly improves decomposition accuracy and task success rate compared to existing methods that rely solely on either language or vision planning alone. The contributions of this work are as follows:
first train the
We propose PERIA, novel framework that integrates holistic language planning and vision planning, leveraging the logical and intuitive decomposition of general complex instructions.
We encourage MILM to output rich latent visual tokens to guide the diffusion model to generate im- ages and further explicitly align between language instructions and visual subgoals, simultaneously developing the MLLM's reasoning and the diffusion model's 's imagination capabilities.

---

PERIA demonstrates significant improvements in instruction following and task success rate on complex manipulation tasks compared existing methods that rely on either language or vision planning alone, establishing a promising and inspiring paradigm for long-horizon manipulation.
2 Related Work
2.1 Hierarchical Planning for Long-horizon Manipulation
Embodied manipulation tasks with general instructions often span multiple subtasks and long hori- zons, making direct end-to-end action prediction challenging due to compounding errors without intermediate guidance [12, 13, 14, 15]. Recent works adopt hierarchical planning, decomposing complex instructions into sequential sub-tasks to execute. Language planning like LISA [8] and Xskill [16] decompose the general instruction based on the latent skill codebook discovered during training. SayCan [3] and EmbodiedGPT [7] both leverage LLM to enable reasoning into sequen- tial interpretable instructions in natural languages. Vision planning, more recent development, decomposes complex instructions into sequential subgoal images. CoTDiffusion [10] utilizes diffu- sion models translate multi-modal prompts into coherent subgoal images in a chain-of-thought manner, serving as visual milestones that are challenging to describe using language alone. While existing works rely solely on either language or vision planning, our PERIA framework enables simultaneous language and vision planning, harnessing the strengths of both approaches to provide a comprehensive, multi-modal guide that enhances the accuracy of instruction decomposition and following.
2.2 LLM for Robotics Manipulation
With the tremendous success of LLMs, there has been a surge in research exploring their capabilities for robotics manipulation, such as SayCan [3]. Inner Monologue [17]. PAR [18] leverages a vision language (LMc)(VLMM as a captioner for visual observations and the generated captions are fed into
LLM for language planning. ViLA [19] and CoPA [20] follow a similar pipeline but replace LLM and VLM with more advanced GPT4V [21] with stronger visual reasoning capabilities. EmbodiedGPT [7] employs pre-trained open-sourced LLaMA model [22] as the language model for instruction tuning on collected robotics data, enhancing reasoning and planning capability specifically for embodied scenarios. PERIA introduces image generation as additional supervision signal to encourage the MLLM to perceive more detailed visual details, reducing hallucinations and errors in reasoning. The generated images in vision planning also provide more intuitive guide that further enhances the accuracy of instruction decomposition and improves instruction following performance.
2.3 Image Generation for Robotics Manipulation
Inspired by recent development of recent text-to-image models [23, 24, 25], many works have begun to explore the visualization of manipulation tasks to guide robot action execution. LfVoid [26] enables the editing of original observations to obtain goal images based on natural language instructions to provide reward signals. SuSIE [9] similarly leverages an image-editing diffusion model to act high-level planner by proposing intermediate subgoals that low-level controller can accomplish. LfVoid and SuSIE are limited to single-step sub-instructions, while CoDfifuusion [10] supports various instruction modalities and generates coherent subgoal image chains using semantic alignment module. These works demonstrate that subgoal images can provide more detailed and intuitive guidance than language-only instructions. However, they do not incorporate LLMs for reasoning and are prone to failure and semantic conflicts without logical guidance. Our PERIA framework leverages the prior knowledge in MLLMs assist in generating promising sequential images, enhancing consistency with complex task instructions and improving instruction following.
3 Method
By leveraging MILM and diffusion--ased image editing models, PERIA enables holistic language planning and vision planning for stepwise language instructions and visual subgoal images, serving as language milestones and visual anchors to guide action execution long-horizon tasks. We first introduce the lightweight alignment of language and vision modalities on the encoding side of
3

---

Give brief description of the given observation subsequent keyfsomes / short demonstrations,
Observation
Task instruction exee.e Spell a word using letters on the table for the name of top
Prompt
de... Please give brief stepwise instruction to do next and the
Image
Text Encoder
00000000000000000000000000000000
CS conference
w............................... ..
corresponding gool image:
Projection
Multi-modal LLM
00000000000000000
<BOS> Move orange letter EOS
[IMG]]IMMG]
[IM6]]
Multi-modal L.LM
*
Output Projection
<Scene Description
On the table, there are two bigger green blocks, one smaller red bowls the left bottom ...
Laesson
Limagine
AAction Recognitions:
Diffusion Model
Move the smaller blue block into the block bowl.
<Video Underst derstanding::
Move orange letter to the bottom left area
LAligmant
The given demonstration is to first put smaller blue block ento the smaller red block and then...
Action
Mutti-modal L.L.M
Recognition
Figure 2: Overview of PERIA. PERIA first learns align the vision and language encoding side of MLLM for perceiving. Then PERIA performs instruction tuning to MLLM jointly with diffusion model in an end-to-end manner to unlock the reasoning and generation ability for holistic language planning and vision planning. A and * show the module is trainable and frozen, respectively.
the MILM achieve precise Perceive ability in Section 3.1. We then illustrate how to perform instruction tuning on the MLLM to enable Reason for language planning in Section 3.2 and how to jointly train with a diffusion model to Imagine coherent subgoal images aligned with corresponding instructions in Section 3.3. Moreover, we leverage an explicit alignment between instructions and images to achieve a synergistic effect between language and vision Section 3.4. Since our focus is not on the low-level policy, please refer to Appendix E for the implementation details of Act.
3.1 Perceive: Encoding-side LLM-centric Multimodal Alignment
To enable embodied robot agents to effectively perceive and comprehend visual scenes, straight- forward approach is to use an off-the-shell Vision-Language Model (VLM) as an image captioner. However, the information bottleneck between the I.LM and VIM limited by language modality, results in missing visual details, which is particular problematic in robotics manipulation tasks that require precise visual understanding. To address this limitation, we leverage image captioning as a training task for LLM- multimodal alignment encourage visual representations compatible with the text feature space within the LLM, extending it to a MLLM to allow for a more precise and detailed comprehension of visual scenes. Specifically, we utilize the privileged information available in simulation environments to create a large-scale dataset of pairwise ground-trum -truth data through three types of captioning tasks. First, the single-frame scene description scenario focuses understanding a single observation frame, where the MLLMM is tasked with providing a brief description covering aspects such as object recognition, size identification, number counting, color understanding, and spatial relationships. Second, action recognition between consecutive keyframes scenario presents the MILMM with two consecutive keyframes, requiring it to understand the visual difference between them and recognize the executed action, enhancing the perception of spatial relationships and action dynamics at the instruction level. Moreover, short demonstration understanding scenario involves processing a given short demonstration of frame sequences, strengthening the MLLM's temporal relationship understanding and grounding ability. These carefully designed captioning tasks with the high-quality training data, enable the MLLM to develop a strong foundation in visual perception and understanding for embodied manipulation tasks.
Specifically, initialized from a pre-trained LLM, the MLLM contains a visual encoder V(e.g., CLIP- I. [27]) to extract the visual features f, and projection layer W to project into the language modality. We follow the training of L.LVVA [28] with cross-entropy loss (CELoss) as:
C = {I1, .... 31), I fui, U2,..., Unb.
sf. MLLM ... -. (2 1llprommt W(J (L))))
(1)
LPerceive
CEL.
t=1

---

where C can be the image caption for features alignment and | is number of word tokens. is the numbers of the images Z fed in MILM, which can be differ in different captioning tasks. To perform the lightweight alignment, we freeze the weights of both the vision encoder V and LLM,, and only update the parameters of W that encourage to map the image features into shared latent space that is compatible with the MILM's hidden representations. The alignment of visual and language modalities on the encoding side can effectively alleviates hallucinations and lays the foundational perception abilities for generating more grounded language and vision planning. For more detailed categorical analysis of improvement benefiting from captioning task, please refer to Appendix F.I.
3.2 Reason: Instruction Tuning for Language Planning
With the initial coarse alignment of visual and language on the encoding side, we proceed to instruction tuning to encourage the MLLM to learn how to decompose complex instructions for language planning. These general task instructions T can be categorized into two types based on the modalities involved: 1) text-only instructions, such as "sort blocks into bowls according to the matching colors" which can be directly processed by the language encoder; and 2) multi-modal instructions that consist of interleaved language and images of a single object or whole observation, as suggested by VIMA-BENCE [13], which are more expressive and challenging to understand. For instance, consider an interleaved multi-modal prompt such as "Stack objects <img> in this order <img> <img>", where <img> serves placeholder for the corresponding images, which can be images of blocks or other objects in the observation. With the benefit of encoding-side alignment via several captioning scenarios across frames and videos, MLLM equipped with the input projection layer can handle multi-modal instructions including interleaved text, images, and even video frames.
Then we design an instruction prompt P template such as: "Given the current observation <img> and the general task instruction [7], can you provide a brief and concise sub- cub-instruction about how to act next?". We collect the stepwise language instructions & as the groundtruth response for the language planning task specified with the observation o, the prompt P and the general instruction T. The instruction tuning loss for language planning is defined as follows:
& = fel, e2,..., at. I == f0, U1, U2 ..., vnh: e = MLLMMLMM (fe..,,.
..., e- (P, W(F V(D)D,
CReason
CELoss(e(,,e) t=1
where are the word token of stepwise instruction, and are the possible images from multi-modal instruction. The text instruction with the <img> token and the corresponding image are processed by the aligned language encoder and image encoder respectively and then are unified fed into LLM for reasoning. The MLLM follows the standard auto-regressive training for the next token prediction and then can be regarded as visual assistant for various tasks such as visual question answering. To perform instruction tuning, we fine-tune the MLLM using the LoRA technique [29] while keeping the encoding side frozen, including the visual encoder and its projection layer. Additionally, we employ two kinds of prompts to require MI.LM to generate the next sub-instruction for the single step or all the stepwise instructions in order respectively. Two modes can be randomly switched during the instruction tuning and can effectively encourage MLLM to perform single-step and multi-step sequential language planning for closed-looped and open-looped control respectively.
3.3 Imagine: Decoding-side Synergistic Training for Vision Planning
Considering the phrase a picture is worth a thousand words, subgoal images could provide higher expressive capabilities for conveying subtasks compared to sub-instructions with complex language only. Inspired by this, we integrated pre-trained conditional diffusion models to convert decomposed sub-instructions into coherent visual subgoal plans. While a natural approach would be directly use the text instructions or captions as prompts for the image editing model, shown in Figure 3, relying solely on decoded text instructions as conditions may lead to an information bottleneck. The expressiveness of the instructions can be limited, and information loss may occur, as it is confined to the language modality. Inspired by [30, 31J, to bridge the gap between the language and vision modalities, we introduce N special [TMG] tokens in the vocabulary codebook of the MIL.M. These special tokens have trainable word embeddings and should predicted after the generated language instructions jointly during the reasoning phase, shown in Figure 2. These appended visual tokens
5

---

Image Obs
Text Ins
Visual Obs
Text Ins
Visual Obs
Text Ins
+
+
+
VLM
LLM
ViT
MLLM
ViT
MLLM
Visuel Token
Text Taken
Caption
Instructions
Projection
Instructions
Projection
Diffusion Model
Diffusion Model
Diffusion Model
Subgoal Images
Subgoal Images
Subgoal Images
Instructions
Three pipelines of MILM for generation images. PERIA (c) leverage visual tokens extracted from the MLLM during language planning serve as more expressive guidance for subgoal imagination compared to captions @a) or decomposed instructions (0) in language only.
Figure 3:
[IMG] are treated as latent imagination of subgoal image from the MLLM and we employ an output image projection module R. to transform them into actual visual guidance U for diffusion model:
u (R((([aa)))) + p }.9)
where w is the word embedding of language instructions and h is the hidden state from the last layer of MLLM before image projection layer of [IMG], conditioned on learnable query embeddings {9......... JL), where L is the token numbers setting from the pre-trained diffusion model. The transformation over w can be seen as a general representation from language modality, while
represents more grounded visual imagination that aligns with the language planning within the MLLM's reasoning. To simultaneously fine-tune the diffusion model and the MILM, we employ the generation loss between the generated image and the groundtruth image. Our image editing model is based on latent diffusion, which learns the noise latent z at the denoising timestamp to reconstruct the groundtruth goal image. The generation loss is to learn the UNet eo that predicts the added noise based on the input image v and the visual imagination guidance U from the MLLM, formulated as:
Limagine No,.. [Ile (ile-eazze,
es(z,, v,, v,,1
3.4 Enhancing Consistency between Vision and Language Planning
To further enhance the consistency between vision and language planning, we introduce additional alignment objective between generated language instructions and visual images, as illustrated Fig- ure 2. Specifically, we feed both the generated image 12+1 and the current observation o planning step into the MLLM and prompt it with understanding the differences between the two frames, which is exactly the action recognition captioning task in the perceive phase of PERIA Section 3.1.
The response output &, generated by the MLLM is compared with the groundtruth stepwise language instruction & for consistency, and can be formulated as the alignment consistency loss:
C (E:)T- I ((o,,,+++)))
E, = MLLM(prompt,WWL = (V(o)), ((e++1)))
(5)
Connsistency
CEL.s)(&
The additional alignment task reinforces the synergy between vision and language planning, ensuring that generated subgoal images and text instructions are consistent and mutually informative, alleviating the compounding errors that may arise in long-horizo tasks due to inconsistencies. Vision planning provides a visualization of language planning. offering more intuitive guidance and reducing potential confusion or ambiguity. Conversely, language planning provides logical guidance at the semantic level for vision planning, preventing semantic conflicts during the generation of coherent image chains. This synergistic approach leverages the complementary strengths of vision and language, enabling PERIA to produce plans that are both visually grounded and semantically meaningful.

---

Visual
the bladke
Language Planning
Pur snalkorbiue Mack
Pat Magergge by block onto
amalle
Put analersreen block
Put Aggenbrren blokk
amateryal
Pugeer gnay
Vision Planning
Put the blecke bowil motchhed colorn.
Longuage Plarning
Pick the yello block plcce on The yallion
Prck the gnay slock piace gracy
Pakk the oyon block and place
Pick-p the blue block
Pick the red black place on bavt
Vision Planning
Neu
Hour F
HOU
Spel word about the Top CS cenference.
Language Plonning
Mone oronge letter N bottinn onea
Mewn pirk letter right aronge
Move eyen lether right
Move hiln latter right wii
Move perple letteer right to letter
Figure 4: The illustrating examples of holistic language and vision planning for general instructions, with stepwise sub-instie cob-instructions and coherent subgoal images enhancing the instruction following.
Experiments
4.1 Experiment Setup
Benchmark & Tasks To provide comprehensive evaluations, we conduct experiments across three typical long-horizon manipulation environments. More benchmark and task details are in Appendix A.
LoHoRavens [18]: a Ravens-based benchmark consisting of 11 long-horizon language- conditioned tasks categorized into Stacks, Sort, and Matching. Original tasks all involve ma- nipulating Bowls&Blocks and we additionally develop more complex Letters scenario including 9 tasks of Shape, Orders, and Spell further diversify the instruction and increase task difficulties. VIMA-BENCH [13], a benchmark for long-horizon manipulation, contains diverse tasks guided by multi-modal prompts. We choose 8 tasks from three representative categories Rearrange, Constraints, and Follows, specified by interleaved language and images of object or ultimate goal.
Baselines To more clearly and comprehensively evaluate the effectiveness of different approaches, we categorize the baselines into three types based on their specific planning methods as follows:
End-to-end: We choose CLIPort [12], one of the most widely used end-to-end language- conditioned imitation learning framework in Ravens-like manipulation benchmarks. CLIPort directly take the high-level language instructions as input to predict the action without planner. Language Planning: We select several representative language planning methods that decompose general high-level instructions into stepwise instructions. LISA [8] trains skill predictor to combine the discovered implicit skill codebooks for complex instructions. PAR [18] (Planner- Retor-Reporter) replaces the latent skill planner with an LLM, using the VLM as reporter for visual observations. The instruction and the generated captions are then fed into the LLM for language planning. EmbodiedGPT [7] follows similar pipeline but replaces LLM and VLM with more advanced MLLM with stronger visual reasoning capabilities after instrution tuning.
Vision Planning: SuSIE [9] incorporating pretrained image-editing models to generate goal images for action prediction but only support simple single-step instructions. CoTDiffuison [10] leverage semantic alignment module within the diffusion model to enable the sequential subgoal image generation for complex general instructions. For more details, please refer to Appendix C.
4.2 Main Quantitative Results of Success Rate
We begin by comparing the performance PERIA and baselines in solving long-horizon horizon tasks across three typical task domains. The baselines can be categorized into three types of planners: e2e planner, language planner, and visual planner. outperforms other baselines in terms of success rate. As expected, the end-to-end learning method performs the worst due to the lack of intermediate guidance, making difficult for the policy to follow general instructions for long-hon horizon tasks. In contrast, the visual planner paradigm, which explicitly
shown in Table 1, PERIA significantly

---

Table 1: The evaluation of success rate between baselines and we report the mean and variance across 5 seeds.
BlockstBowls
Letters Orders
VIMA-BENCH
Model
Stacking
Sort
Matching
Shape
Spell
Rearrange
Follow
Constraint
CLIPont
18.4
19.2
17.8,2a
9.8
8.1g27
2.3g0
5.8
2.4g00
8.3
LISA
26.6118
22.
23.0
18. 11.4
16.11s9
10.2g17
8.9.2.
6.3g15
11.9
PAR
34.7
32.8263
31.1244
31.5258
30.7249
27.3.72
24.425.
16.1
26.5445
EmbodiedGiPT
48.0
49.1
43.4g78
40.926.
48.247.5
52.7
38.345.3
37.2+47
43.5g05
SuSIE
24.1
82.6241
33.2257
37.826.
35.2+4.3
34.1274
37.945.8
40.2as.4
51.2477
CoTDiffusion
47.9
44.3g7.0
56.626.2
46.1
53.9
44.827.
51.2a6.3
54.5g7.3
76.1
PERIA (ours) 63.9_5.8
65. 65.020.4
72.3
60.6g5.2
65.2
71.1=7.6
74.826.0
67. 67.2=7.
89.3.4.0
decomposes tasks into stepwise instructions and employs hierarchical framework consisting of language planner and a language-condingg conditioned policy, shows more promise and demonstrates a clear advantage over the end-to-end approach. Within the visual planner category, PAR and EmbodiedGPT both leverage the common sense knowledge from L.LM and significantly outperform LISA, which uses skill predictor for latent skill codebook rather than LLM. Furthermore, although both PAR and EmbodiedGPT are based on LLaMA, EmbodiedGPT employs visual projector to expand the LLM an MLLM for more precise perception and reasoning capabilities, while PAR applies a captioner to convert visual images into the language modality for reasoning, which may impact the accuracy of reasoning and task success rate to some extent. The visual planner paradigm, which generates intermediate keyframes, offers more intuitive guidance compared to language planning, and its advantage is more evident in VIMA-BENCH, where sub-tasks are challenging to describe sufficiently using language-only instructions. CoTDiffusion supports generating coherent subgoal images for complex instructions, resulting in performance gains compared to SuSIE. But CoTDiffusion does not explicitly reason about the instructions, which can lead to semantic inconsistencies in the generated subgoal images, causing still underperform compared to our algorithm. In contrast, our PERIA algorithm introduces an MLLM for explicit reasoning and generation, providing more sufficient and reliable intermediate guidance for instruction following in long-horizon tasks.
4.3 Further Analysis
Accuracy of Language Planning We compare the
Table 2: Evaluation of reasoning accuracy between methods two metrics.
accuracy of language planning with two evaluation met- rics: the token accuracy which directly calculates the token-level matching rate between decomposed stepwise instructions and the groundtruth instructions, and the se- mantic similarity by calculating the embeddings distance of two instructions from pre-trained text encoder like CLIP. Our focus here is on generative language planning using LLMs and exclude LISA from this comparison. As illustrated in Table 2. PERIA demonstrates the highest accuracy in both token-level and semantic- level comparisons. Although PAR introduces LLMs for language planning, it relies on an isolated, out-of-the-shhe VLM as a captioner to convert visual observations into language descriptions, which may cause details missing during hard captioning. EmbodiedGPT further introduces a projection layer to bridge the gap between vision and language in the latent space, gaining more advantages in perception which is critical in language planning. Compared to EmbodiedGPT, PERIA'S superior performance can be attributed to the explicit incorporation of vision planning. By jointly fine-tuning MLLM using the additional image generation loss, the supervision from visual aspects encourages promoting attention to visual details and spatial information for more grounded reasoning. When we remove the joint training of vision planning, we observe the more frequent hallucinations and errors in language planning, such as generating unseen objects with wrong colors, sizes, locations, which significantly decreases the accuracy of language planning. Moreover, we also ablate the encoding-side multimodal alignment and the degradation in accuracy highlights the importance of enhancing the foundational perception capabilities through our carefully designed dataset, which includes various perception- -reltted data such as spatial relationships. temporal relationships, size recognition, and color identification. To further investigate the improvement in foundational perception abilities, we conduct a detailed categorical analysis, which can be found in Appendix F.1.
Method
Token t Semantic 1
PAR
58.2
0.63
EmboddiedGPT
65.9
0.68
PERIA (ours)
6
0.98
wlo perceive pretrain
80.2
0.83
w/o vision planning 83.7
0.79
8

---

REA mo Connsterrrco
Conaatecy Sour
1.
(b)
(C)
Figure 5: More detailed quantitative analysis. (a) The ablation studies consistency loss and [IMG]
token numbers. (b) The comparisons of three planning paradigms in tasks with various horizon lengths. (c) The evaluation of generalization ability of three levels. See text for further discussion.
Table 3: Comparisons of FID (L) between methods on three task domains.
Fidelity of Vision Planning We further compare the fidelity of generated goal images against groundtruth keyframes using the Fréchet Inception Distance (FID) [32] as the evaluation metric. Although SuSIE not strict vision planning method for long-horizon manipulation due to its limitation to simple single-step instructions, we grant it relaxed privilege by provid- ing oracle stepwise instructions to enable comparison. However, shown in Table PERIA still demonstrates superiority, primarily due to the implicit gen- eration of latent image tokens during language planning. The extracted image latent embeddings from MLLM retain more details and provide more sufficient guidance beyond language for subgoal image generation. Ciffifuusion supports general instruction inputs and can sequentially generate multiple images. However, the absence of explicit language planning CoTDiffusion makes challenging to ensure the semantic coherence of the generated images, potentially leading dilemmas such as semantic repetition, jumping, or regression within the generated image sequences. In contrast, PERIA incorporates MLLM for reliable instruction decomposition and leverages the extracted image latent embeddings to achieve superior fidelity in vision planning compared to existing methods. Moreover, the performance drop in the ablation study without consistency loss highlights the importance of alignment between reasoned stepwise instructions and generated subgoal images, attributed to the synergistic combination of language planning and vision planning in our framework.
Methodology
Blocks
Letters
VIMA
SuSTE (+oracle)
18.9
18.1
19.4
CoTDiffusion
13.1
15.8
17.6
PERIA (ours)
10.2
13.5
11.4
w/o alignment
12.3
14.2
15.9
Consistency between Reasoning and Imagining We leverage CLIP [27] to measure the image- language similarity between generated instructions and images, with results presented in Figure 5a. The additional consistency aligmment loss explicitly constraints and encourages semantic alignment between the imagined images from vision planning and the reasoned stepwise instructions from language planning, significantly enhancing the collaboration and consistency between the two modalities. Furthermore, increasing the number of [IMG] tokens provides more expressive and sufficient guidance, facilitating the MLLM in producing semantically coherent language and image tokens. However, the benefit of adding more tokens becomes marginal beyond a certain threshold.
Effectiveness of Holistic Planning We modify the low-level policy model into several variants, including ones that simultaneously utilize stepwise instructions and subgoal images, well those that rely on each modality individually. As shown in Figure 5b, the holistic planning approach achieves the highest success rate than single planning, with the benefit of the increased amount of information available and rich multi-modal guidance for decision-making. which reduces the training difficulty of low-level policy and enhance the accuracy of action prediction. Moreover, the advantage holistic planning becomes more evident as the horizon length increases, demonstrating its scalability and effectiveness handling complex, long-horizon manipulation tasks.
Generalization across Tasks We evaluate the generalization ability in three levels with increasing difficulty: placement generalization with novel placement of objects (L1), object generalization with novel objects (L2),, and combinatorial generalization with extra novel instructions (L3). The results in Figure 5c demonstrate that PERIA enjoys substantial advantage over other baselines, highlighting the importance of the common knowledge prior within the MLLM and diffusion model and holistic planning, which enhance the generalization and robustness for unseen challenging tasks.

---

5 Conclusion
We propose PERIA (SEe, Reason, Imagine, Act), a novel framework that integrates MLLM and diffussion-basee image editing models enable holistic language and vision planning for long-horize manipulation tasks with complex instructions. We first perform a lightweight multi-modal alignment to enhance the MLLM's fundamental perception capabilities of visual details for manipulation, alleviating potential hallucinations. Then, we encourage MLLM to output rich latent visual tokens to guide diffusion model in generating images and explicitly align language instructions with visual subgoals to simultameously unlock MLLM's 's reasoning and diffusion model's imagination capabilities. Extensive evaluations across three challenging benchmarks demonstrate that PERIA significantly outperforms competitive baselines in both instruction following accuracy and task success rate, while also enjoying better generalization ability across tasks. We believe PERIA highlights the potential of holistic language and vision planning and we hope this novel paradigm can provide some insights to robotics manipulation research of long-horizon tasks with complex instructions in free-form, towards more open embodied scenarios. One current bottleneck is the relatively high time cost of training and inference. Improving the joint training efficiency of MLLMs and diffusion models in a lightweight manner and accelerating image generation sampling are interesting directions for future work.
6 Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132, 62106172), the National Key R&D Program of China (Grant No. 2022ZD01 16402) and the Xiaomi Young Talents Program of Xiaomi Foundation.
References
[1] Dongxu Li, Jumnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for control- lable text-lo-image generation and editing. arXiv preprint arXiv:2305. /4720, 147.0000 2023.
[2] Yecheng Jason Ma, William Liang, Vaidehi Som, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: I Language-image representations and rewards for robotic control. arXiv preprint 2rXiv:2.366.0099. 2023.
[3] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsca Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et Do as can, not as say: Grounding language in robotic affordances. arXiv preprint 22X1v22004.01691, 2022.
[4] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian lehter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint Xivi: w.x.vvv:30.000 2.303.0338,, 2023.
[5] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, al. Rt-2: Vision- Aision-lagguage-cttinn models transfer web knowledge to robotic control. In 7th Annual Conference on Robot Learning. 2023.
[6] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models, arXiv preprint arXiv:2307.0583 059773, 2023.
[7] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2023.
[8] Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learning interpretable skill abstractions from language. Advances in Neural Information Processing Systems, 35:21711-23 1-21724, 2022.
[9] Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained imago-editing diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[10] Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Liu Jiashun, Yan Zheng, Bin Wang, and Yuzheng Zhuang. Generate subgoal images before act: Unlocking the chain of thought reasoning in diffusion model for robot manipulation with multimodal prompts. Computer Vision and Pattern Recognition, 2024.
10

---

[11] Michael C Corballis. Left brain, right brain: facts and fantasies. PLoS biology. ee 2014.
[12] Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on robot learning, pages 894-906. PMLR, 2022.
[13] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Fei-Fei. Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
[14] Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning language- condige-o robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303 1315. PMLR, 2022.
[15] Longxin Kou, Fei Ni, Yan Zheng, Jinyi Liu, Yifu Yuan. Zibin Dong, and HAO Jianye. Kisa: unified keyframe identifier and skill annotator for long- horgzhorzzon robotics demonstrations. In Forty-first International Conference on Machine Learning.
[16] Mengda Xu, Zhenjia Xu, Cheng Chi, Manuela Veloso, and Shuran Song. Xskill: Cross embodiment skill discovery. In Conference on Robot Learning, pages 5536-3555 PMLR, 2023.
[17] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et Inner monologue: Embodied reasoning through planning with language models. arXiv preprint urXiv:2207, 056008, 2022.
[18] Shengqiang Zhang, Philipp Wicke, Lütfi Kerem Şenel, Luis Figueredo, Abdeldjallil Naceri, Sami Haddadin, Barbara Plank, and Hinrich Schütze. Lohoravens: A long-horizon annutag--oonditioned benchmark for robotic tabletop manipulation. arXiv preprint arXiv:2310. 12.220000.0200,. 2023.
[19] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao. Look hefore you leap: Unveiling the power of gpt-4v in robotic vision-language planning. arXiv preprint arXiv:2311. 2023.
[20] Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manip- ulation through spatial constraints of parts with foundation models. arXiv preprint arXiv:2403.0888 08248, 2024.
[21] OpenAL Gpt-4v(ision) system card. htttps://ed..opee http:///eeed.eeeeeaaaaaaaaaaappe wttp:/ppppppp////.......... Card. Card.pdff -df, 2023.
[22] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 092888 2023.
[23] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220..06125. 2022.
[24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic texi-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11487, 2022.
[25] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555. 2023.
[26] Jialu Gao, Kaizhe Hu, Guowei Xu, and Huazhe Xu. Can pre-trained text-to-image models generate visual goals for reinforcement learning? Advances in Neural Information Processing Systems. 36, 2023.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning (ICMI.), 2021.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual Instruction Tuning. In arXiv:2304.00 084455 2023.
[29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0944 09685, 2021.
11

---

[30] Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan, Guiding instruction- based image editing via multimodal large language models. arXiv preprint 17110 2023.
[31] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2023.
[32] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[34] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong. Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for rohotic manipulation. In Conference on Robot Learning, pages 726-747, PMLR, 2021.
[35] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openftamingo: An source framework for training large autoregressive vision- langugg models. arXiv preprint arXiv:2308.01 01330, 2023.
[36] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13392-18402, 2023.
[37] Oier Mees, Lukas Hermann, Erick Rosete- Bosete-Bees, and Wolfram Burgard. Calvin: benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3):33777333, 2022.
[38] Ilya Loshchilov and Frank Hutter. 1XXv:1771.05/00,2201 2017.
Decoupled weight decay regularization. arXiv preprint
[39] Jiaming Song, Chenlin Meng, and Stefamo Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:001..02002, 2020.
[40] Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F., Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 902023.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the CVPR, pages 16674-10685, 2022.
12

---

A Details of Benchmarks and Tasks
To provide comprehensive evaluations, conduct experiments across three typical long-horizon manipulation environments covering diverse instruction types and task types.
A.1 Bowls&Block
LoHoRavens [18] is a benchmark dataset built upon the Ravens robot simulator, comprising ten long-horizon, Lunguage-condttoons and Matching. In Stacks tasks, the objective is place blocks in absolute or relative areas. Sorr tasks require sorting blocks or bowls with similar specified attributes together. Matching tasks involve placing corresponding blocks into matching bowls. These tasks encompass various aspects of long-horizon reasoning, including color, size, space, arithmetic, and reference. To successfully complete each task, robot must effectively combine multiple reasoning capabilities and develop an appropriate long-horizon plan.
tasks. The tasks are categorized into three types: Stacks, Sort,
Move Move the blocks with some specified attributes like colors, sizes or locations to the specified area. We design tasks as follows:
MoveBlocktoArea Move all the blocks to the labs_area).
MoveColorBlocktoorren Move all the [color) blocks to the (abs_area):
MoveBlockinAArettoAArro Move all blocks in (abs_area) to {abs_area). MovesizeBlektooooneer Move all {size) blocks to (position) corner.
right
Figure 6: The example in MoveelockinArreatoAree. in Blocks&Bowls Move,
Stack Stack blocks or bowls with specified attributes and put some area together. We design
4 tasks as follows:
StackAllBlocks: Stack all the blocks together.
Blacssiie Stack all the blocks of the same size. Stack Block Stack all the blocks the same color.
Stack ColoroooooocttoAArrc
Stack all blocks of primary color on left
side.
Stock ALL Blocks of the together.
Figure The example StackBlocksORSameCClor Blocks&Bowls Stack.
Matching Placing corresponding blocks into matching bowls zones. We design 3 tasks as follows:
PuBBoeeknMatchingBow:: Stack all the blocks together. PutBlokInniimmtthingBooo/ Stack all the blocks the same size.
PatBlokknZooeeviiMaachhhniCCC Put blocks of the same color in the zone with matching color.
13

---

Put the blocks the bowls with matched colors
Figure 8: The example in PutBlockImMatchingBoy in Blocks& Matching.
A.2 Letters
In addition to the original LoHoRavens simulator, which consists of long-horizon tasks involving only Bowls&Blocks, we have developed a novel Letter scenario further diversify the range of long-horizon reasoning tasks. This scenario randomly generates various letters with different colors and cases. We have designed three new task types: Shapes, which select the letters with specified shapes and arrange it together; Orders, which involves arranging letters in specific order to test the robot's understanding of sequence and position; and Spell, which assesses the robot's 's capacity for letter combination and word spelling by requiring the robot to spell words that meet specific requirements.
Shape Move the blocks with some specified attributes like colors, sizes or locations to the specified
area. We design tasks as follows:
SorrVeriiaSymmmllektto.cc
Sort the vertically symmetrical letters
to the bottom side.
BortHorrzottaSymmBlllsttt..o Sort the horizontal symmetrical letters to the blank space
Sor(ConralSymmlloccttoo Sort the central symnetrical letters to the corner.
AHOXX
Sort the vertically synmetrical letters to the battom side.
Figure 9: The example in SortVerricakyymmBloossttkkk in Letters Shape.
Orders Stack blocks or bowls with specified attributes and put some area together. We design 3 tasks follows:
Putlecteerrllpaaeetcalorerrr::::
Put the letters on the tables in Wutleetteeseerllpaaattiollrrrr Put the letters on the tables in reverse alphabetical order.
alphabetical order.
SortConsLLettersOrdeer: Sort the consonants from all letters in orders.
W
3
accop
accopu
accapuu
a accopuux
all letters alphabetical order.
Figure 10: The example in PutLetteessAlpaaaeticallrrrr in Letters Orders.
14

---

Spell Placing corresponding blocks into matching bowls or zones. We design 3 tasks as follows:
Spelll.oggWords: Spell words that are long as possible.
SpellCSCon/Name: Spell out the name of top CS conference.
Name: Spell out the name common transportation.
Airp
Airp!
Airpia
Airpian
Airplane
Spell a word about the transportation.
Figure 11: The example in SpellTransName in Letters Spell.
A.3 VIMA-BENCH
VIMA-BENCE [13]: benchmark for long- horizon manipulation with general instruction specified by multi-modal prompts, containing various tasks ranging from simple object manipulation to multi- object manipulation. We select three kinds of representative long-horizon manipulation tasks Rearrange, Constraints, and Follows, in which general instructions specified by interleaved language and images of object or ultimate goal.
Rearrange Move the blocks with some specified attributes like colors, sizes or locations to the specified area. We design 2 tasks as follows:
RearrangeOhitoGGaal: Rearrange ob to this setup <img>.
Rearrange objects to this setup <img>
then restore.
F
Rearrange objects this setup
and then
Figure 12: The example in RaaraangOhiicGilltheeeeeerr in VIMA-BENCH Rearrange.
Constraints
Stack blocks or bowls with specified attributes and put to some area together. We
design tasks as follows:
SweepNoExceedCons::
Sweep all <obj> into <container> without exceeding <constraint>,
SweepNoTouuhhons:: Sweep all <obj> into <container> without touching <constraint::
PutSameTeetfrromooall: Put all objects with same texture as <IMG> into it
PutSame@ShpefrromGaal: Put all objects with same shape as <IMG> into it.
15

---

Sweep all
into
without touching
Figure 13: The example in SweepNoTouchhons VIMA-BENCH Constraints.
Follow
Placing corresponding objects following orders specified by several relevant images. We
design tasks follows:
Follow.MotionOjj: Follow this motion for <obj>: <img 1>
<img
N>
FtollOw Stack objects this order: <img 1>
<img N
Stock objects this arder
Figure 14: The example in StackObjFollow in VIMA-BENCH Follow.
B Details of Datasets
B.1 The Collection of Expert Demonstrations
For the LoHoRavens and VIMA-Bendi datasets, we utilize the provided oracle engines collect expert demonstrations. is worth noting that if there are multiple correct answers or multiple orders to complete the task, only focus whether the custruction-specife in the end and include correct demonstrations as training data. Across all designed 28 tasks, we collect 2k demonstrations per task, gathering total of 56k long-hone mor-hooizoon demonstrations with horizons ranging from 2 to 10+ sub-tasks. For all the collected data, we divide it into an 80% proportion for the training dataset Derain and 20% for the testing dataset. Among all the tasks, the Spell task in the Letters dataset requires additional explanation. Unlike other tasks where the instructions are relatively fixed, the Spell task's instructions to spell the name of a top computer science conference, we can solve this by maintaining list of all top CS conferences in advance and checking all possible permutations of the given letters find the answer, which is then executed by the oracle engine to render the expert data. However, for more diverse instructions, such as spelling the name of food or city, the list may be extremely large, requiring the introduction of LLMs like GPT to assist in finding the answer for the oracle engine, which is good direction to expand the Letters domain and enrich the task types, future work.
complex task is completed
more flexible and diverse. For example, if the instruction asks
B.2
The Privileged Information Annotation of Datasets
During the initialization process, maintain a record of the assets used and annotate their corre- sponding attributes, enabling accurate identification of the color, size, and spatial relationships of the manipulated objects during each subtask's 's aack-and-llace operation without the need for additional manual annotations reliance on VLMs for captioning, which often have low accuracy without fine-tuning, even for models like GPT-4V. This privileged information from the simulator's underly- ing data allows us to construct series of captioning tasks that help improve PERIA's foundational capabilities in visual perception and reasoning. During testing, access to the underlying environment information, such as the exact number of ground truth blocks or letters and their various attributes, is
16

---

not possible. The model must directly perceive and ground these crucial visual details from the visual observations and perform subsequent reasoning, significantly increasing the task's difficulty.
Table Overview of three main task types, including Block&Bowls, Letters and VIMA-BENCH.
Task Type
Deseription
Horizon
Color
Size
Spatial
Instruction
Blocks & Bowls
Move all the blocks to the [ABS POS] area
4-115
x
x
Text-only
blocks color
2~15
Text-only
Move
Move all the blocks in the [ABS POS] area the [ABS POS] aren
2-15
x
X
Text-only
Move
blocks on comeersiide
4-115
x
Text-oaly
Stack all the blocks
4-115
x
Text-only
Stack
Stack blocks the same size.
4-115
x
Text only
Stack blocks alternate colors.
2-15
Text-only
Stack only the primary color blocks on the left side.
2~-12
X
Text-only
blocks the howls matching colors
Text-only
Matching
Put the blocks in the bowls with mismatching colors
2-112
Text-only
Put blocks same color in the zone with matching color
2~-12
Text-only
Letters
the vertically symmetrical letters
Text-only
Shape
Sort the horizontal symmetrical letters to the blank space
2-15
x
Text-only
Sort the central symmetrical letters to the corner
2-115
x
Text-only
Put the letters the tables alphabetical order
2-15
x
Text-only
Orders
Put the letters on the tables in reverse alphabetical order
2~15
x
Text-only
the consenants
all letters
2-15
Text-only
Spell words that are as long possible
4-15
Text only
Spell
Spell out name top conference
4-10
Text-only
Spell out the name of common transportation
4-15
Text-only
VIMA-BENCH
Rearrange objects
2-5
Multi-modal
Rearrange
Rearrange the objects to this <IMG> then restore
3~100
Multi-modal
Sweep all <ubj> into <ooatainer> without exceeding <comstraint>
2~6
Multi-modal
Sweep <obj> into crontainer> without louching <constraint>
2-.9
Multi modal
Constraints
Put all objects with same texture as <IMG> inoo it
2--8
Multi-modal
objects with same shape as <IMG>
2-88
Multi
Follow this motion for <nbjo: <IMG 1sM.<IMG No
2-88
Multi modal
Follow
Stack objects in this order:
<IM.
2-8
Multi-modal
B.3 The Wordcloud of Language Instructions
To visually summarize and showcase the frequency all instructions, including object nouns, colors, sizes, and verbs, we create word cloud visualization in Figure 15. We tokenize each instruction and record all the tokens from the language instruction for each skill code used in the trajectory. Once we generate heat maps and word clouds. These word
have this mapping from skills tokens, distributions effectively visualize the scope of the benchmarks, which focus on manipulating objects in human spaces by following general complex instructions in unpredictable scenarios.
bow MoVe
Stack zone
Sort t Puu t
order ymmmet rical Spell
rightIMe without Sochiie
Sweep tobs
trainr
ABS
Color
P
Steed left
Side
right
O
blocs
letters
left
top
00 ects
container
Put
ltft
area
mweolonabecico tables
Rearrange
(a) Boolk&Bowls
(b) Letters
(c) VIMA-BENCH
containermatching
Containe aettannemmatcion
lettersi
00
Light
DOWLS
Lo
Ob ects
O
K
IMG
Rear Dearranga range
order
side
PPu
eeft
Spell
1
wi thout
o
POS
ABSO
Sort
tables S symmetric
Sweep
color
r Stack Move
obj
areacoone corner 2one
Sighabstick
(d) All Three Task Types
Figure 15: World Cloud:
We ereated the word cloud to visually summarize the key aspocts covered the
diverse manipulation instructions three tasks types.
17

---

C Details of Baselines
CLIPort CLIPort [12] is popular end-to-end algorithm functioning language-conditioned imitation learning agent that directly takes in high-level language instructions without planner. It combines the broad semantic understanding of CLIP [33] with the spatial precision of Transporter [34]. As an end-to-end baseline, we make no modifications to CLIPort, as its native SE(2) action space well-suited for benchmarks like Ravens, which is one of the key factors contributing the high data efficiency of Transporter and CLIPort. We train CLIPort by matching general instructions with pairwise actions for Block&Bowls and Letters. To accommodate the multi-modal instructions in
we make additional adaptations by directly borrowing the prompt tokenization mechanism from VIMA without further modification, the same with other baselines. Specifically, instead of operating on raw RGB images, VIMA adopts an object-centric representation by cropping objects from both prompt and observation images as object tokens sequences with pixel coordinate information.
We also compare with LISA [8], hierarchical imitation learning framework that discovers implicit skills and learns to combine them for complex tasks. LISA learns diverse, interpretable primitive behaviors or skills from language-condingg conditione demonstrations to better generalize unseen instructions. It employs vector quantization to learn discrete skill codes that are highly correlated with language instructions and the behavior of the learned policy. LISA can be considered form of language planning, where the predicted instructions are in the form of skill codes. The low-level foundation model in LISA uses a decision transformer as its backbone and we retain the original implementation without any additional modifications.
LISA
PAR PAR [18] (Planner-Actor Aettr-Actoorreeee is paradigm that replaces the skill predictor with an LLM, using a VLM as a reporter for visual observations. The instruction and the generated captions are then fed into the LLM for language planning. In PAR, Llama 2 13B [22] and VLM OpenFlamingo [35] with few-shot prompting are employed as the Planner and Reporter, respectively. It is important to note that the Actor, or the low-level foundation model, is precisely the language-conditioned CLIPort trained by stepwise sub-instructions as mentioned earlier. To ensure fair comparisons, we make no modifications and keep the low-level foundation model consistent across all other baselines with CLIPort as backbones, the same with PAR.
EMT EmbodiedGPT [7] is standard paradigm that incorporates an MLLM for language planning. The main difference between EmbodiedGPT and PAR lies in the replacement of the LLM+VLM combination with a more advanced MILM, which possesses stronger visual reasoning capabilities. EmbodiedGPT trains the MLLM with the constructed embodied chain-of-thought dataset to enable the MILMM to perceive visual details in its hidden layers, similar to LLaVA [28]. To ensure fair comparisons, we make no modifications to the planning module and keep the low-level foundation model consistent across all other baselines, using CLIPort as the backbone, identical to the approach in PAR.
SuSIE SuSIE [9] proposes hicrarchical framework that leverages an image-editing diffusion model to act as high-level planner by proposing intermediate subgoals that a low-level controller can accomplish. It is worth noting that SuSIE is not a strict vision planning method for long- nog-horizon manipulation, as it can only support relatively simple single-step instructions and falls short when it comes to complex general instructions. To enable a comparison, we grant SuSIE relaxed privilege by providing oracle stepwise instructions, as it is limited to handling sub-instructions of single step and cannot generate image chains for complex general instructions. SuSIE chooses InstructPix2Pix [36] as the pre-trained image-editing model and fine-tunes it with dataset of language-labeled video clips and robot trajectories from CALVIN [37]. Since the image editing model is sensitive to training data, we find that its generation performance on the Ravens domain is limited. To address this, we perform addi- tional fine-tuning, keeping the number of training iterations and dataset exactly the same with PERIA.
CoTDiffusion CoTDiffusion [10] is a standard vision planning paradigm that supports translating general complex instructions, including text-only or multi-modal prompts, into visual subgoal images in chain-of-though! manner. Compared to SuSIE, the most significant difference lies in CoTDiffusion's 'ss explicit design of semantic alignment module within the diffusion model to capture the correspondence and semantic completion between the generated images and the general
18

---

instruction, enabling chain-of of ofothttttt generation. Similar to SuSIE, we fine-tune CoTDiffusion on our collected dataset and employ the same low-level conee-condiiooeed policy as SuSIE, which is the image-conditioned variant of CI.PPor. However, since CoTDiffusio does not explicitly introduce an LLLM for planning. it may still encounter semantic conflicts during the vision planning process, such as repetition, backtracking, or skipping steps.
D Details of High-level Planner Learning
D.1 Pretraining of Perceiving Stage
In the initial pretraining stage, PERIA aims to acquire vision-language knowledge and alignment between vision and LLM from large collection of aligned image-text pairs, The designed captioning task for alignment of visual and language modalities in the encoding side is crucial for effective understanding and reasoning about visual scenes, bridging the gap between perception and reasoning in manipulation tasks and laying the foundation for the subsequent development of reasoning and imagination abilities in PERIA. We choose ViT-B-32 as the visual encoder and Vicuna-7B as our LLM backbone. For the input visual projection, we opt for a simple linear projection module, as we found that more complex architectures like Q-former from BLII2 [1] yield similar performance. The linear projection consists of three layers with a hidden size of 4096.
We regard the output from the injected projection layer as a soft prompt for the LLM, prompting it to generate the corresponding ground-trute truth texts. Throughout the entire pretraining process, both the pre-trained vision encoder and the LLM remain frozen, with only the linear projection layer being fine-tuned. To perform the lightweight alignment, we freeze the weights of both the vision encoder and LLM, and only update the parameters of the projection module to encourage mapping the image features into shared latent space that is compatible with the MLLM's hidden representations. Specifically, we train with batch size of 64, using 8 V 100 Nvidia GPUs for parallel training in 8 hours. We employ the AdamW optimizer [38] with a learning rate of 2e-4, a linear warmup of 1 steps, and weight decay of 0.01.
D.2 Joint Training of Reasoning and Imagining
The output projection layer adopts a transformer-based architecture characterized by a hidden size of 512, 4 attention heads, 4 encoder layers, and 4 decoder layers. The latent image token embeddings and the word embedding are fed into the output projection layer and map the latent image representation into the latent space of the diffusion model. We train the MLLM and diffusion model jointly using eestruction-oolloiing datasets and incorporate LoRA [29] to fine-tune the weights of the LLM to achieve lightweight supervised fine-tuning. For the diffusion- based image editing model, we choose the finetuning pipeline borrowed from Instruct-Pix2Pin [36], the most widely used pipeline for conditional image editing tasks. We train for 50k steps with batch size of 16, using 8 V100 Nvidia GPUs for parallel training over 42 hours. We employ the AdamW optimizer [38] with a learning rate of le-4, a linear warmup of 1k steps, and weight decay of 0.01. We track an exponential moving average (EMA) of the model parameters with a decay rate of 0.999 and use the EMA parameters at test time. The strength of classifier-free guidance w is set 2.0, and we use the DDIM sampler [39] with 50 sampling steps.
Empirically, we find that the pretraining stage of perception is crucial. Without the encoding pretraining in the perception stage, the convergence time of the subsequent decoding side significantly increases, and the performance deteriorates. During the joint training of MILM and the diffusion model, we observe that the loss convergence of the reasoning stage is often faster than that of the generation stage. One major reason could be that image editing more challenging, requiring more details compared to language planning. We did not extensively tune the ratios between the image loss, generation loss, and consistency loss. Instead, we simply added them together considering the similar scaling ranges among them. Investigating the optimal weighting of these loss components could potentially further improve the synergy between language and vision planning, but we leave this exploration for future work. Furthermore, we notice that when the image loss of the diffusion model approaches convergence, continued training, although not resulting in a significant decrease in loss,
3 Pattps:/ http:://///ggnnaacc... www.//o///nnc//eeeeeeeeea /viianaevvaaaaaaaaaa 7b-v1.5. 5, 7B, version
19

---

notably improves the fidelity of the generated images during evaluation. Therefore, after reaching certain level, we turn off the gradients of the LLM and keep only the gradients of the diffusion model enabled, which can further accelerate the convergence speed of the generation loss without affecting the overall reasoning quality. However, we have not further investigated the specífic relationship, is not our main focus, but it presents an interesting research direction that we will explore in future work. The summary of architecture and the parameters are listed in Table as follows:
Table 5: The overall configuration and training pipeline of two training phases for MLLM and diffusion model.
Vision Encoder
Input Projection
LLM
Output Projection
Diffusion
Training Stage
Name
Param
Name
Param
Name
Param
Name
Param
Name
Param
Perceiving
ViT-B-32
87M
Linear 18M A
Vicuna [40]
7B
Reason& Imagine
ViT-B-32
87M*
Linear
18M
Vicuna [40]
7B
Transformer 31M SD [41]
1.3B
E
Details of Low-level Policy Learning
In the final phase of the PERIA framework, we focus on training the low-level policy model to develop its capability act, enabling the effective execution of the generated language and vision plans from the previous phases. We adopt CLIPort, a widely used end-to-end learning algorithm for Ravens, as its native SE(2) action space well-suited for benchmarks like Ravens, which is one of the key factors contributing to the high data efficiency of Transporter and CLIPort. CLIPort has two variants: a language-conding conditioned policy and an image coal-conditioned policy. We train these variants with stepwise language sub-instructions and coherent keyframes as inputs, respectively, allowing them to serve as the low-level foundation policy models for language planning and vision planning. For fair comparison, our planning- maaning-basedd methods, whether language planning or vision planning, use CLIPort as the backbone for the low-level foundation model.
To accommodate the simultaneous presence of stepwise language sub-instructions and coherent keyframes from vision planning and language planning, design variant that is conditioned on both image and sub- inbbissssuctoon simultaneously. We combine the two representations through cross-attention block with a 4-layer lightweight attention layer, 4 cross-attention heads, and an embedding dimension of 768 as the fusion layer. We sample the oracle action trajectories a, current observation o, stepwise instruction e, and pair-wise wise subgoal images v from the Dirain as mini-batcl B. The action à is predicted with the instruction and corresponding e simultaneously. The low-level policy & is updated on mini-batch B according to the following loss:
Lact Σ |rr - ps (e, et, #t)l
(6)
We use the AdamW optimizer [38] with learning rate of Ic-4, linear warmup of 500 steps, and weight decay of 0.01. We train with batch size of 64 for 10k steps on single V100 Nvidia GPU, which takes 12 hours. The policy model is conditioned on both the generated subgoal images and the reasoned language instructions, reducing the training difficulty from two perspectives: first, by providing more decision-makinn action sequences. Thanks to the explicit subgoal generation from the high-level visual planner, the low-level policy model is not required to master complex multi-step manipulation skills over long horizons. Furthermore, the stepwise instructions from language planning and the subgoal image chains from vision planning enable the policy model to be trained without directly conditioning the general instruction to predict the entire long-horizon action sequence. Instead, the policy model can predict action sequences in segments, effectively reducing the complexity of the policy learning task by leveraging the structured guidance provided by the language and vision planning components.
information, and second, by shortening the prediction horizon for
20

---

F
More Results and Analysis of Additional Experiments
F1
The Effect of Encoding-side Alignment
To effectively improve the perceiving capabilities of the MLLM and lay solid foundation for grounded reasoning and imagination, we introduce encoding-side alignment. We design a series of visual question answering (VQA) tasks to categorize the perception capabilities into five fundamental sub-skills: object recognition, color recogmition, size identification, number counting, and spatial relationship understanding. For each fundamental perception capability, we design several targeted questions. The question templates are detailed in Table 6, and can be divided into two types based on the answer format: Yes/No questions and open-ended questions. We feed the questions and corresponding visual images into the MLLM for evaluation, prompting the MILM to generate language-only answers. In this study, we use GPT-4 and compare its generated responses with the ground-truth trrth answers and give a evaluation of the semantic similarity and correctness ranging from 1-5, with I being the lowest score, indicating incorrect answer, and 5 representing the highest similarity, indicating the most accurate and contextually appropriate answer. Notably, we find that GPT-4's similarity assessments are highly accurate and closely match expert human evaluations, allowing us to directly employ it as scoring mechanism. This approach enables automated large-scale evaluation without the need for extensive crowdsourced annotation resources, making the assessment process more efficient and cost-effective. We attempted to use GPT-4 for more fine-grained scoring (for example, scoring from 100), but found that the consistency with human evaluations was not as good as the 1-5 scale. For each term of fundamental perception capability, we randomly select 100 cases from the 28 tasks dataset used in the paper for evaluation. We calculate the total score and normalize it by the maximum possible score (28 task 100 cases 5 score) to obtain percentage score, which presented in Figure 16.
Table 6: Overview of five types of fundamental perception capabilities. The question templates illustrated and can be categorized into two types: open-ended and Yes/No questions.
Foundation Capability
Questions
Answer Type
Can you identify all the objects on the table?
Open-ended
Are there any objects that are movable the table?
Yes/No
Object Recognition
How many different types of objects are table?
Open-ended
Does the letter appear on the table?
Yes/No
What are colors the blocks? Is the robot manipulating any of the red blocks?
Open-ended
Yes/No
Color Recognition
How many colors are there in blocks on the table? Are the colors of the blocks on the desktop duplicated?
Open-ended
Yes/No
Can you tell the size of all the blocks on the table?
Open-ended
Are the blocks the same size?
Yes/No
Size Identification
How many different-sizer blocks?
Open-ended
Are the blocks relocated by the robot identical in size?
Yes/No
How many blocks
the table?
Open-ended
Which item has the highest number different objects?
Open-ended
Numher Counting
Can you identify the number of blocks bowls?
Open-ended
How many objects were moved in lotal in demonstration?
Open-ended
Which corners of the table have no objects?
Open-ended
Are there objects stacked on top of each other?
Yes/No
Spatial Relationship
Which area of the table has the most objects?
Open-ended
How many layers is the highest stack of objects?
Open-ended
Our results in Figure 16 demonstrate that carefully designed captioning tasks can significantly enhance the MILM's performance across various foundational perceiving capabilities. When we remove the perception pretraining with encoding side alignment, we observe more frequent hallucinations and emror in the designed VQA evaluations. It is worth noting that even the ablated version of perception pretraining still shows some advantage over EmbodiedGPT, which can be attributed to the additional image generation loss. The supervision from visual aspects encourages attention to visual details and spatial information for more grounded reasoning.
side
21

---

Foundation Perception Capabilities Evaluation
Object
EmbodiedGPT
PERIA wo Perception Pretrain
Spatial
100
40
Color
Size
Number
Figure 16: The evaluation of fundamental perception capabilities between language planning methods with MLLMs.
F.2 The Effect of Joint Training
To verify the effectiveness of language planning and vision planning, conduct an ablation study by decoupling the training of reasoning and imaging. First, we train the MLLM solely for reasoning. generating text tokens to predict stepwise instructions. Subsequently, we use only the text tokens from the stepwise instructions as conditional information to train the image editing model. It is important note that this differs from the visual fidelity experiment mentioned the main text, as the <IMG> tokens are entirely set to zero, and the absence of joint training eliminates the consistency loss. We introduce semantic similarity metric to evaluate instruction following accuracy. Specifically, we calculate the CLIP similarity between generated subgoal images and general prompts, normalized by the CLIP score between the ground truth ultimate goal image and prompts. This metric reflects the progress of generated subgoal images throughout the entire chain, tracking the instruction following and gradual advancement towards ultimate goals specified by complex instructions.
Blocks
&Bowls Blocks&Bowls
Letters
VIMA-BENCH
Normallzed Clip Scores
0.0
0.9
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
Horizon Length Steps
Figure 17: The evaluation of the normalized CLIP scores between instructions and generated subgoal images for each generation step. reflecting the stepwise accuracy of instruction following and the incremental progress towards ultimate goals specified by complex instructions.
To ensure consistent comparison, we select tasks with horizon length across all task types. The results in Figure 17 show that CoTDiffusion has the worst semantic alignment due to the lack of explicit incorporation of LLMs for logically reliable reasoning. The results reveal that the generated images from the decoupled training version exhibit relatively poor instruction following compared to the jointly trained version. We attribute this two main reasons. First, using only text instructions as conditioned information fails to provide sufficient guidance, which can be considered version
22

---

with <IMG> tokens. Second, the absence of a consistency loss constrains and encourages semantic alignment between the generated subgoal images and instructions. In summary, the performance drop caused by decoupled training highlights the benefits of joint training, which enables synergistic effect, more fine-grained and consistent image generation, and instruction reasoning. noting that the performance drop caused by decoupled training is more significant the VIMA- BENCH, highlighting the importance latent image token embeddings in providing guidance beyond language, especially in task environments where text-only instructions are challenging to describe sufficiently and completely.
worth
F.3 The Flexibility of LLM Backbones
To comprehensively compare the impact of different L.MMs backbones on the capabilities of the PERIA framework, we experiment with various LLMs backbones for fine-tuning, including Vieuna- 7B 1, Vicuna-13B LLaMA-2-7B 6 and LLaMA-3-8B The evaluation results for each model are presented in Figure 18. The results show that Vicuna- 13B outperforms Vicuna-7B, indicating that larger model sizes can bring performance gains. However, the more recent and powerful LLaMA- 3-8B surpasses both Vieuna models, demonstrating that our framework can achieve substantial improvements and enhancements by leveraging stronger LLM backbones.
80
PERIA Vicuna
PERIA Vicuna 139
PERIA L'ama2-78
PERIA Llama3-8B
75
Success Rate (%)
70
65
60
55
Blocks&Bowls
Letters Task Type
VIMA-BENCH
Figure 18: The evaluation of PERIA with different LLM backbones across three task types.
G Quick Guideline of Usage
From PERIA import load_peria
1lm_backbone ['vicuna-7B', 'LLaMA2-7B', 'Vicuna-133', 'LLaMA3-8B") peria Icad_peria (11__aabone))))
Low_levee_foee
load fdm fdm fdm_path')
obssevvttinn = _oad_obss ('obs_s_____paaa""
cask_instruction = load_ins
prompt _cad__rommpt_ ("ad_pprompppprrrottt_aaaa"
while not denc:
stepwise_instructte = htw.o/ollaammioooeeetttiiiioo
(observatinn,
task_instruction, prompt)
subgoal_image p prompt)
(observation,
action low Level fdm (observation,
subgoal image)
observation, done env www.pptppaccinn
3b-v1.5, 13B, version 1.5
httrps:/hhuginnggocccco 1aaettt-llmmm/tte hammm//taaaa-----hatttt 7B
"attps://uuggiggee wwt.ccc...cc.a.....o/ooaaa...... Llama 3-8B-Instruct,
23

---

H Pesudocodes of Framework
Algorithm 1 The training of PERIA for robotics manipulation.
Input: Training dataset with pair-wised keyframesstepwise sub- iub-instruction and action trajectories,
MLLM, conditional diffusion model and low-level foundation model for action planning.
Perceive: Encoding-side Alignment between Vision and Language
for each iteration do
Sample images I {v1, u2,..., 2n3 and pairwised caption {21, 22,
22, ... 21 from the
D as mini-batch B
Calculate the projected visual tokens W(f V(Z))) by projection layer W after the visual encoder V the encoding side.
Fed the text token from prompt and the projected visual tokens into L.L jointly and infer caption di in an autoregressive way:
ML.LM (11, - --.-Fe-1), ) prompt W(S (V(m))==))
Update parameters of W on mini-batch B according the following loss:
Peeceive
E 1 CELoss(Ér,
end for
Reasoning and Imagine: Decoding-side Joint training for MLLM and Diffusion Model
for each iteration do Sample general task instructions T, initial observation image o, prompt P, pairwised sub-instructions & {e1, ...... e from the Dtrain as mini-batch
Et) and groundtruth subgoal images I {U1, U2,,... vn
Fed the text token from general task instructions T and prompt P and the projected visual tokens from o into LLM jointly and reason the stepwise sub-instruction d;: of [P, IP.T,W W(f = V(o))),, calculate the reasoning loss as follows: CReason = E CELoss(e(,
(II,,,
Extract the word embedding Winss] from append [IMG] token after the reasoning
phase, and extract the hidden state [ix]] from the last layer within MILM.
Transform w (MC) and [( into actual visual guidance II via image projector
Generate imagined subgoal images via image editing diffusion model with conditional
guidance 14 and calculate the imagine loss as follows:
Limagine = V., (0..)))) [||e ea Ez(z,, t, v, [
Fed the generated image and the original visual observation back MLLM and
perform the same captioning task like Perceive Stage
action recognition of consequent images between and o Infer the caption
Calculate the consistency between inferred instruction E and groundtruth instruction
LConsistency = El= CELoss( Update MILM,, diffusion model and corresponding projector R on the decoding side: Liotal = LReason + Limagine + CCensistency end for
Act: Training of Goal-conditioned Low-level Policy
for each iteration do
Sample the oracle action trajectories a, current observation o, stepwise instruction and pair-wised subgoal images from the Dirain mini-batch B
Predict the action with the instruction e and corresponding e simultancously
Update low-level policy v on mini-batch B according the following loss: CAst Σx ||å
end for
24

---

I Limitation & Future Work
While PERIA demonstrates significant improvements in long-horizon manipulation tasks with com- plex instructions, there are still some limitations that need to be addressed in future work.
The current implementation of PERIA relies a pre-collected dataset for training the MLLM and diffusion model. Although this allows for effective learning of perception, reasoning, and imagination capabilities, may limit the framework's adaptability novel environments tasks that deviate significantly from the training data. Future work could explore methods for online learning or adaptation enable PERIA to generalize new situations more effectively.
The joint training of the MILM and diffusion model can be computationally intensive and time- consuming, particularly when generating high-quality images. While we have demonstrated the effectiveness of this approach, further research is needed to optimize the training process and improve its efficiency. This could involve the development of more lightweight architectures, advanced training techniques, or parallelization strategies.
While PERIA has shown promising results in simulated environments, its performance in real- world scenarios remains be explored. Real-world manipulation tasks may introduce additional challenges, such as noisy sensory inputs, dynamic environments, and physical constraints, which could affect the framework's performance. Future work should investigate the deployment of PERIA on physical robotic systems and assess its robustness and effectiveness in real-world settings.
Despite these limitations, PERIA introduces a novel and promising paradigm towards enabling robots perform complex manipulation tasks with general instructions. By addressing these challenges and continuing to refine the framework, we hope that PERIA can provide some insights to the robotics manipulation research towards long-horizon tasks with more complex instructions in free-form, paving the way for more intelligent and versatile robotic systems that can effectively operate in wide range of environments and applications.
Social Impact
By enabling robots to understand and follow more natural and diverse human instructions. PERIA can facilitate seamless human-robot collaboration industries such manufacturing, healthcare, and household assistance. This could lead to increased productivity, improved quality of life, and the creation of new job opportunities. For instance, an educational robot equipped with the PERIA framework could help children engage in constructive play activities, such as building block games or puzzles. The robot could provide step-by-step guidance and demonstrations, adapting to the child's skill level and learning pace. This interactive and personalized approach to learning could enhance children's cognitive development, problem-solving skills, and creativity.
In conclusion, the advancements in long-horizon manipulation tasks presented in this work have the potential advance the progress in the field of intelligent embodied robots, but responsible development and deployment practices must be adopted to ensure the safe, ethical, and beneficial integration of robots in educational or industrial settings.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please see the abstract's last four sentences and the introduction's 's last paragraph in Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: See the conclusion section in Section 5. Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what
implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
26

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have the code and model checkpoints ready for release. Besides, we provide sufficient implementation details for researchers to reproduce the results Appendix D and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the puper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and model checkpoints is ready and will be released after the conference decision made.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: //tp//ips....// cc/
publ ic gui des /CodeSubmi as i onPol cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
27

---

including code, unless this is central to the contribution (e.g., for new open open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips ps..ps... cc/ hocc/pll.tt puublii. / gu des CodeSubmi ss i onPol o ccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
reproducible, they
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide full details of benchmarks in Appendix A, baselines in Appendix C, implementation details in Appendix 1) and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provided the mean and standard error over several random seeds in the experimental results to demonstrate statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
28

---

Justification: We detailed the compute resources used for the experiments in Appendix D and Appendix E. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: hwwripsu.....ee. ttt.:c/cpppbbcc httpp///.ccccccccccccceeeeeeelll
Answer:
[Yes]
Justification: Authors conducted in the paper conform, in every respect, with the NeurIPS
Code of Ethics https: puttps://neutt httpp//sccccpppcccccccccctcotoot
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See the conclusion in Section 5 and the social impact in Appendix J. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: The paper poses no such risks. Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For all the datasets and algorithm baselines used in the paper, we have cited the original papers and provided the license, copyright information, and terms of use in the package our code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swit thoode com/ dataaaaaaaaa aatasets s has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
30

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "KqvZedn6p7", "reviewer": "Reviewer_HJPE", "summary": "This paper focuses on robotic manipulation with complex instructions. It proposes PERIA, a framework that integrates MLLM and diffusion models to incorporate both language planning and visual planning for long-horizon language-instructed manipulation tasks. Specifically, PERIA first performs a lightweight multi-modal alignment to consolidate the multi-modal perception capabilities. Then, PERIA performs multi-modal instruction tuning, where it outputs both subgoal language descriptions and visual tokens, both of which are fed to a diffusion model to generate subgoal images. PERIA introduces an additional consistency loss between and generated subgoal image and language descriptions. Experimental results demonstrate that PERIA significantly outperforms competitive baselines.", "questions": "•\tYou generate subgoal images with subgoal descriptions and generate visual tokens. Why not use 1) subgoal descriptions and observation or 2) generated visual tokens alone? The former resembles a world model, and the latter sounds like a decoding of an imagined visual subgoal, both of which sound more natural. I guess you have tried the latter but found it was not as good as adding subgoal language.\n\n•\tWhat LLM do you use? It is possible that a powerful LLM accounts for superior performance to some extent. Have you compared the LLMs of different works?", "limitations": "Yes, the authors address the limitations at the end of the conclusion.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThis work follows a natural and reasonable pipeline to tackle the manipulation tasks with complex language instructions. Combining language planning and visual generation for manipulation is a sound approach.\n\n•\tThe alignment stage empowers the overall capabilities, as demonstrated in the experimental part.\n\n•\tPERIA achieves convincing experimental results compared with previous works. The authors also conduct extensive ablative study to mine more insights.", "weaknesses": "•\tEnd-to-end learning for such a large system requires considerable cost. Such a comprehensive framework may lead to powerful performances but the resources may be a limitation. This paper does not present how much resources PERIA uses or related experiments to address such potential concerns.\n\n•\tOne of my concerns is that the consistency objective, which forces the MLLM to output subgoal language descriptions, may suffer from accumulative error. This is because when the generated subgoal image is not the desired image but is a natural image that can be reached within one-step action, the MLLM would learn an incorrect subgoal description.\n\n•\tMore literature references and related baselines should be incorporated.\n\n•\tThe ablation in visual planning lacks an experiment where PERIA generates subgoal images with either subgoal descriptions or generated visual tokens, which should reveal more insights into what leads to the improvements in visual planning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1 Computation resources\n\nSorry for the ambiguity arising from distributed presentation of computational resource requirements across Appendix. The computational cost of PERIA across three primary stages: Perceive (8 V100 GPUs * 8 hours ), Reason & Imagine (8 V100 GPUs * 42 hours), and Act (single V100 GPUs * 12 hours).  For more details of model architecture and training hyperparameters, please refer to Appendix D and E in main paper.\n\n**More Analysis:**\n\nConsidering most PERIA's computational cost is to align between vision and language, and bridge the gap between pretrained LLMs and image editing models in general domains versus robotics manipulation domain, we conduct further investigation by substituting enhanced model backbones for both the visual encoder and LLM. We compared the training time and performance until convergence, follows as:\n|Model|Training Time(Perception)|Training Time(Reason + Imagine)|Success Rate|\n|--|--|--|--|\n|PERIA (ViT + Vicuna 7B, reported in paper)|8hrs|42hrs|68.2|\n|PERIA (ViT + llama2 7B)|8 hrs|41 hrs|69.0|\n|PERIA (ViT + llama3 8B)|7 hrs|39 hrs|73.1|\n|PERIA (LIV + Vicuna 7B)|5 hrs|38 hrs|69.2|\n|PERIA (LLaVA1.5 7B)|4.5 hrs|32 hrs|70.3|\n\n**Key Findings:**\n\na) **More Powerful LLM**: LLaMA-3-8B improved performance and facilitated easier training due to its superior common sense understanding.\n\nb) **Specialized Visual Encoder**: LIV, a robotics-specific representation with fewer parameters than ViT-B-32, leveraged its pre-training on robotics datasets to achieve vision-language alignment, alleviating further alignment efforts.\n\nc) **Pretrained MLLM:** Pretrained MLLMs such as LLaVA 1.5, having already undergone general domain vision-language alignment, obviate the need for fine-tuning from scratch, thus expediting adaptation to robotics tasks.\n\n**Summary：**\n\nWe will continue to explore **more powerful model backbones** and **more efficient fine-tuning techniques** to alleviate the computational cost. Additionally, we are interested in investigating **more lightweight subgoal modalities** (maybe obj mask, bbox or keypoint tracking) to balance computational cost and sufficient guidance in future work. We also hope PERIA can serve as a **potential base foundation model** for other MLLM-based embodied manipulation research to reduce computational costs, offering a more efficient starting point compared to learning from scratch using MLLMs not fine-tuned with robotics data.\n\n# Q2 Consistency loss& Accumulative Error\n\nGood questions! We structure our response into three key points.\n\n- The case that generated subgoal image is a one-step goal, pointed out by the reviewer, is **Granularity Error**, which also includings repetitions, skipping, or even backtracking to visited subgoals. We expect PERIA to learn appropriate task decomposition granularity with the **supervision loss** from groundtruth labels for both language planning and visual planning. When visual and language planning produce semantically consistent errors (e.g., images and language in one step)  at the same time, the supervision loss imposes significant penalties still discouraging such outputs even when consistency loss is low.\n\n- The MLLM's subgoal description ability mainly stems from action recognition pretraining in the perceive stage, not solely from consistency loss. Instead, we utilize this ability to implement consistency loss as a self-supervised mechanism during the reason and imagine stages, encouraging semantic alignment between generated instructions and subgoal images within MLLM's unified latent space.\n- The consistency loss, an auxiliary prediction task besides the supervision loss, serves as a bridge to enable language planning benefit from pixel-level guidance while allowing visual planning to be influenced by semantic-level supervision, thus avoiding isolated learning within each modality. This intermodal connection fosters a more holistic and coherent learning process of PERIA. \n\nFor more details of visual supervise guidance and consistency, please refer to Fig.5(a) in main paper and Tab.1 in the uploaded PDF.\n\n# Q3 More Related Work\n\nThanks for the suggestions!  We additionally included comparisons based on existing pretrained MLLM methods and video planning approaches in Tab.2 of the uploaded PDF. We will add theses in the revised version of main paper.\n\n# Q4 & Q5 Condition Input for Visual Generation\n\nAwesome question!  We conduct the detailed ablation studied on three generation mechanisms with language subgoal descriptions, generated visual tokens, or a combination of both as condition. The results can be found at Fig. 1 of uploaded PDF. \n\n**Key Findings:**\n\n- More tokens as condition input for visual generation is always better but improvement gets marginal when token reachs a threshold. \n- Visual tokens only lack sufficient semantic, potentially leading to overfitting or training collapse, especially with a limited number of tokens.  Increasing the number of visual tokens can alleviate this but it is essentially equivalent to incorporating language tokens. **Discarding the inherent high-level semantics of language tokens and add more visual tokens to learn these semantics from scratch is inefficient, requiring more training time and wastefully ignoring exsited language planning output.**\n- Language tokens contains rich semantics but can also get benefit from more visual tokens with visual details that are challenging to describe accurately in language. \n\nThe combinatorial fusion version can achieve comparable performance with fewer visual tokens, striking an efficient balance between semantic richness and visual precision.\n\n# Q6 Different LLMs\n\nSorry for the confusions. We use Vicuna-7B as our LLM backbone. The reviewer's insight is correct - the powerful LLM can bring performance increase and the comparisions can be found at Q1 and the Appendix F.3.\n\n---\n\n**Sincere thanks for your insightful review!  We hope our response can address the concerns.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVdP3LcARR", "reviewer": "Reviewer_C2qN", "summary": "The paper proposes a holistic vision-language planning method for long-horizon robot manipulation, by learning a multi-modal large language model (MLLM). The MLLM generates interleaved language actions and keyframe images based on language goal and the initial image. Each pair of generated language and keyframe image is used as conditioning of a learned motion policy for robot manipulation.\n\nBased on a pretrained MLLM model, the paper first learns a projector to align visual encoding to with language on image captioning tasks tailored to robot manipulation. Then it applies instruction tuning to fine-tune the MLLM, an output projector, and a diffusion model to generate interleaved language and images. Additional, the authors propose another training objective to align the generated language and images. All large models are fine-tuned with LoRA.\n\nOn simulated robot manipulatio benchmarks, the proposed method outperforms imitation learning, language planning, and vision planning methods. The paper also systematically evaluates capabilities of the MLLM along different axes, and justifies the benefits introduced by each loss design via ablation studies.", "questions": "- I wonder which of the three pretraining tasks is the most important for vision-language alignment in the context of robot manipulation. It will be interesting if the authors can show some ablation studies on this.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the important challenge of robot long-horizon planning. The proposed method plans jointly in the language and image space, providing rich information for the low-level policy to condition on.\n- The paper exploits the capabilities of MLLM to generate language and images for robot manipulation, used with a separate low-level policy. I think this is good practice as MLLM is not naturally suitable to generate robot motion.\n- The experiments are comprehensive and provide useful information on understanding the capability of the trained MLLM.\n- The paper is in general well-written and easy to follow.", "weaknesses": "- The explanation of low-level policy is missing from the main paper. This part is very important - the MLLM outputs language and images only, and it's not clear how these modalities are bridged with robot motion.\n- The contribution of the alignment loss between generated image and language is not sufficiently justified in the experiment. It will be helpful if the authors can provide the task success rate when the loss is absent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1 Details of Low-level Policy\n\nSorry for the confusion. Due to limited space, we placed the training details of  low-level policy in Appendix E. Thanks for bringing the attention to critical importance of this section for a comprehensive understanding of the PERIA architecture and we plan to incorporate this information into the updated version of main paper. Here we want to make more explanation about the low-level policy.\n\n- **Architecture**:  To enable multi-task learning, we adopt CLIPort, a wildly-used data-efficient end-to-end multi-task learning algorithm for Ravens that leverages an SE(2) action space. CLIPort introduces several variants and we utilize two version that conditioned on image and language respectively. Both utilize CLIP's visual and language encoders to extract embeddings, which are then fused with observation inputs via element-wise product. We directly train these variants with stepwise language sub-instructions and coherent keyframes as inputs, as **language conditioned version** and **image conditioned version** respectively. Moreover, to accommodate the simultaneous presence of stepwise language sub-instructions and coherent keyframes from vision planning and language planning, we design a variant that is conditioned on both image and sub-instruction simultaneously, noted as the **language-and-image conditioned version**. We maintain the output prediction architecture and just modifying the input processing for fair comparison. Specifically, we fuse the language sub-instruction and corresponding subgoal image through a cross-attention block comprising a 4-layer lightweight attention layer with 4 cross-attention heads. The fused embedding is combined with the observation via element-wise product, maintaining the subsequent modules of the original architecture. Detailed comparisons of the three policy architecture versions are presented in Fig. 4 of the uploaded PDF.\n\n- **Findings**: \n\n  - We investigated performance across tasks with varying horizon lengths of subgoals. Results in Fig. 5(b) in main paper demonstrate that the integration of both modalities provides more comprehensive guidance, significantly enhancing execution accuracy in long-horizon scenarios.\n  - Furthermore, we evaluated our model across diverse task types. As illustrated in Fig. 3 of the uploaded PDF, the additional guidance proves beneficial across all task categories. Notably, tasks from VIMA-BENCH with multi-modal instructions, which are challenging to be sufficient described by language only, exhibited particularly significant performance improvements with the incorporation of subgoal images. Enhancing subgoal guidance through richer modality consistently improves the semantic clarity and accuracy s across diverse task domains.\n\n  In future work, we aim to investigate more effective policy architectures as low-level policy backbones, such as diffusion policy, to further enhance the PERIA framework's performance and capabilities.\n\n# Q2 Contribution of Alignment loss\n\nGood suggestions! We conduct more analysis experiments to verify the effectiveness of alignment loss, including the lanugage accuracy, visual fidelity and success rate. The results can be found in Tab.2 in the uploaded PDF.\n\n- **The bridge between two modality**: The consistency loss, an auxiliary regularization term besides the supervision loss, serves as a bridge to enable language planning benefit from pixel-level guidance with subgoal images while allowing visual planning to be influenced by semantic-level supervision from language instructions, thus avoiding isolated learning within each modality. The ablation of the consistency loss degrades performance in both language accuracy and visual fidelity, demonstrating the effect of intermodal connection via consistency loss.\n- **Alleviate conflicts in holistic planning**: By incorporating the alignment loss, we strengthen the synergy between visual and language planning within the MLLM, mitigating the risk of the diffusion model and MLLM working in isolation.  The generalization evaluation across three levels demonstrate that the integration is particularly beneficial for unseen tasks, where each modality are more likely to output semantics in isolation, potentially resulting in semantic conflicts and task failure.\n\n# Q3 Importance of Three Pretraining Tasks\n\nThanks for this awesome suggestions!  Our pretraining strategy comprises three tasks, each targeting specific capabilities:\n\n1. Scene Description (SD): static understanding of single frame, including object understanding and spatial relationships.\n2. Action Recognition (AR): dynamic understanding of subsequent images and semantic correlation between language instructions and subgoal images.\n3. Video Understanding (VU): continuous comprehension across successive subgoal images and mitigating temporal hallucinations for long-horizon.\n\nWe conducted an ablation study with variants: PERIA (w/o pretrain), PERIA (w/ SD), PERIA (w/ SD + AR), and PERIA (w/ VU) followed as:\n\n||Language Accuracy ↑|Visual Fidelity ↓|Succees Rate(<= 8steps)↑|Success Rate(> 8steps)↑|\n|--|--|--|---|--|\n|PERIA (w/o pretrain) |80.2|16.8|55.5|42.4|\n|PERIA (w/ SD)|89.3|15.7|63.0|54.2|\n|PERIA (w/ SD + AR)|92.6|13.6|68.1|59.3|\n|PERIA (w/ VU)|84.2|15.4|61.0|57.2|\n|PERIA (w/ SD + AR + VU, Ours) |**97.6**|**12.3**|**71.2**|**66.1**|\n\n**Key Findings:**\n- SD is most important as the fundamental static comprehension task for other two pretrain tasks.\n- SD+AR achieve similar performance to the default version, indicating that the SD+AR pairing can develop certain VU capabilities. Conversely, VU as the sole pretraining task performs poorly may because it is too challenging to learn without sufficient foudnation capabilities from SD and AR.\n- Incorporating VU on top of SD+AR significantly improves the success rate for long-horizon tasks exceeding 8 subgoal steps.\n\n---\n\n**Sincere thanks for your insightful review!  We hope our response can address the concerns.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yF6zORO4R5", "reviewer": "Reviewer_tWdi", "summary": "The paper tackles the problem of long-horizon task planning on pick-and-place tasks in the Ravens domain. Given a dataset of trajectories, it first learns the projection to align the vision and language encoder for a multimodal LLM. Then it finetunes both the multimodal LLM and a diffusion model to generate a step action in language, where the diffusion model is used to generate a conditioning subgoal image, which is proposed as an intermediate step that helps with the step action generation in language.", "questions": "See \"weaknesses\" section above.", "limitations": "The limitations are described in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well-written and the figures are helpful for understanding the method.", "weaknesses": "- It is unclear, at least from the experiments in the paper, that the diffusion model is actually useful, especially when the output is still in language space. For example, it seems that the tasks studied in the paper can be easily tackled by a modern multimodal language model (likely even the open-sourced ones), by simply providing the the initial image and appropriate prompting. However, this is missing as an important baseline in the paper (and this does not require additional training data). Furthermore, to demonstrate the effectiveness of an image subgoal in addition to a language subgoal, the evaluation would have to be done on tasks that have subgoals that are difficult to describe in language but easy to describe in visual space, but all the evaluated tasks are the contrary.\n- A related work “Video Language Planning” also seems to be missing from the paper, despite it might involve closed-sourced models. However, the idea seems quite relevant and it’s unclear if the paper provides additional insights for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1 Effectiveness of Diffusion model &  MLLMs & Tasks better described in image\n\nThank you for insightful questions! We appreciate the opportunity to respond point by point:\n\n1. **Tasks better described in image**\n\n    We highly agree that tasks with subgoals better described in visual space are crucial. Our evaluation spans three task types. For Blocks & Bowls and Letters, the textures, colors, and backgrounds are relatively simple and straightforward to describe verbally. However, from VIMA-BENCH, which is specialized with multimodal instructions (interleaved image and language), we selected 8 long-horizon tasks categorized into three types:\n\n   - **Rearrange**: requires precise absolute positioning, which is challenging to specify accurately through qualitative language alone, image with subgoal location is more clear.\n   - **Constraint**: involves explicit intermediate or outcome constraints. Language instruction are often lengthy and unclear.\n   - **Follow:** infers related objects from given example and reason. Object shapes, colors, and textures are often difficult to describe verbally, especially with multiple distractors.\n\n   We believe these tasks are challenging to clearly described in language only and more suitable for image subgoals. For illustrating examples, please refer to Fig.2 in uploaded PDF.\n\n2. **Effectiveness of Diffusion model:**\n\n- **Providing holistic planning capabilities:** \n\nEvaluations across all task types demonstrate the performance gains of holistic planning over language planning methods, with improvements of +19.4% in Blocks & Bowls and +17.4% in Letters. The advantages of holistic planning are even more pronounced in VIMA-BENCH with +37.1%, highlighting the difficulty of adequately describing subgoals using language alone, as shown in Tab.1 of main paper. PERIA's own ablation studies on subgoal modality also support similar conclusions, shown in Fig.3 in uploaded PDF. More detailed subgoal guidance always works.\n\n- **Enhancing language planning accuracy:** \n\nDiffusion model, serving as a visualization module, introduces pixel-level supervision from groundtruth subgoal images to the MLLM. This integration, along with a consistency loss, forms an intermodal bridge that enables language planning to benefit from pixel-level guidance, thus avoiding isolated learning within language supervision.Our ablation studies, which excluded diffussion model and entirely removed visual supervision from PERIA, resulted in a decrease in language planning accuracy, shown in Tab.2 of uploaded PDF. This demonstrates that image generation as an auxiliary prediction task enhances the MLLM's understanding of task instructions and scenes, thereby improving reasoning capabilities, proving the phrase \"What I cannot create, I do not understand\".\n\n3. **Capability of pre-trained MLLMs:**\n\nGood idea!  We tested several popular MLLMs, including GPT4V, Claude 3.5 and InternVL-2 with web interfaces. And we conducted a comprehensive evaluation of GPT-4V key and LLaVa 1.5. The results are shown in Tab.2 of uploaded PDF.\n\nDue to domain gaps between robotics and general domain, these models often misidentified scene object properties, directly impacting reasoning, shown in Fig.5 in uploaded PDF. While GPT-4V demonstrated superior accuracy among MLLMs, it still underperformed compared to the fine-tuned EmbodiedGPT and PERIA. We do not expect to achieve superiority over pretrained MLLMs in general domain. Rather, we highlight that deploying MLLMs in robotics manipulation requires substantial domain-specific data fine-tuning and foundational capability enhancements (see LLaVa 1.5 after finetune). \n\n# Q2 Discussion with VLP\n\nGood suggestions! We now make a detailed comparison between PEIRA with VLP.\n\n**Similarities:**\n\n1. **Motivation:** Leverage a generative model to generate subgoal with modalities beyond language to offer more sufficient guide\n2. **Overall pipeline**:  VLM/MLLM + generative model + low-level policy\n3. **Necessity for fine-tuning:** VLP finetunes PALM-e 12B and PERIA finetunes Vicuna 7B both using addtional robotics domain dataset.\n\n**Key Differences:**\n\n1. **Training Paradigm:** VLP employs decoupled training for reasoning and imagination, with generated videos subject to VLM evaluation before adoption. In contrast, PERIA utilizes joint training of MLLM and diffusion model, aligning their latent spaces and employing a consistency loss to encourage semantically matching language and visual planning.\n2. **Subgoal Modality:** VLP generate short-horizon video as subgoal, while PERIA generate keyframe images, offering a more lightweight representation.\n3. **Low-Level Policy:** VLP offers three methods, including UniPi's inverse dynamics model and two variants of policy conditioned on either last frame or every frame of generated videos. PERIA's approach is most similar to VLP's last-frame conditioned policy, while avoiding the high fidelity and continuity requirements of the video generation, particularly the inverse dynamics approach.\n\n**Further Discussion:** \n\nThe core motivation of both video and image subgoals is to provide sufficient guidance for action prediction. These subgoals can be represented through various modalities, including language, images, or videos, each offering different trade-offs between expressiveness and computational efficiency. Broadening our perspective, can we explore more lightweight modalities to describe these subgoals, such as **object&place mask, object bbox** or **keypoint tracking**?  These approaches could potentially reduce computational cost by focusing prediction on relevant areas but preserve essential semantic information for sufficient guidance to low-level policy. We think this is an intriguing direction for future work, offering opportunities to balance off between computational efficiency and guidance richness of subgoal modality.\n\n---\n\n**Sincere thanks for your insightful review!  We hope our response can address the concerns.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hd3aedGTvC", "reviewer": "Reviewer_WmQe", "summary": "The paper proposes a framework that integrates large multimodal language models (MLLMs) and diffusion models to enable holistic language planning and vision planning for long-horizon robotic manipulation tasks with complex instructions. The authors jointly train the MLLM and diffusion model for language reasoning and visual imagination through latent image token generation. An explicit consistency loss aligns the reasoned instructions with the imagined subgoal images.", "questions": "1. While the authors acknowledge that training and inference costs are significant, the current draft lacks a more in-depth analysis of these problems. In particular, what are the various tradeoffs associated with different MLLMs that can be used, taking into consideration training time/FLOPs/MACs? How does varying these choices impact performance? Experiments answering these questions are equally as important as the ablations being run on training design choices (e.g. alignment loss).\n\n2. Lack of real-world evaluation. Many works ([1], [2]) in this problem setting leveraging foundation models for robotic manipulation demonstrate the advantages of these large MLLMs/generative models in real-world settings, where the distribution of objects is extremely long-tailed. Can the authors show that PERIA can operate with similar success in this regime?\n\n[1] [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://arxiv.org/abs/2311.17842)\n\n[2] [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://arxiv.org/abs/2310.10639)", "limitations": "Yes, the authors address the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel motivation for integrating of multiple modalities for providing better guidance.\n\n2. Principled design of the framework components like the encoding-side alignment and the latent image token generation approach.", "weaknesses": "1. Weak experimental evaluation (see below questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1 More analysis of Training Cost & Performance between LLM backbone\n\nThanks for the insightful suggestions! Considering most PERIA's computational cost is to align between vision and language, and bridge the gap between pretrained LLMs and image editing models in general domains versus robotics manipulation domain, we conduct further investigation by substituting enhanced model backbones for both the visual encoder and LLM. We compared the training time and performance until convergence and utilized CalFlops, an open-source tool designed for calculating FLOPs and MACs for neural networks, with results summarized as follows:\n\n|Model Variant|Training Time (Perception) ↓|Training Time (Reason+Imagine) ↓|FLOPs ↓|MACs ↓|Success Rate ↑|\n|--|--|--|--|--|--|\n|PERIA(ViT+Vicuna7B，reported in paper)|8 hrs|42 hrs|1815.38|907.3|68.2|\n|PERIA(ViT+Vicuna 13B)|13 hrs|66 hrs|3679.82|1810.7|71.9|\n|PERIA(ViT+llama2 7B)|8 hrs|41 hrs|**1740.89**|**874.8**|69.0|\n|PERIA(ViT+llama3 8B)|7 hrs|39 hrs|1926.17|962.2|**73.1**|\n|PERIA(LIV+Vicuna 7B)|5 hrs|38 hrs|1804.60|899.1|69.2|\n|PERIA(LLaVA1.5 7B)|**4.5 hrs**|**32 hrs**|1814.59|903.0|70.3|\n\n**Key Findings:**\n\n- **LLM Capabilities Impact Performance and Efficiency:**\n\nStronger LLMs generally improve PERIA's performance. Replacing Vicuna-7B with models like Vicuna-13B, LLaMA2-7B, and LLaMA3-8B led to varying improvements. Training time is influenced by parameter numbers and inherent LLM capability. Despite its larger size, Vicuna-13B underperformed LLaMA3-8B in both performance and training efficiency due to higher parameters and lower capability. Conversely, LLaMA3-8B, with more parameters than Vicuna-7B, showed reduced training time, likely due to its stronger general-domain knowledge facilitating easier fine-tuning.\n\n- **Pretrained MLLM and Visual Encoder Enhances Performance and Efficiency:**\n\nPre-trained MLLMs, such as LLaVA 1.5, which have undergone general domain vision-language alignment, obviate the need for alignment LLM and visual encoder from scratch. PERIA trained on LLaVA 1.5 can significantly reduce training time and improve the overall performance. Similarily, LIV, a robotics-specific representation with fewer parameters than ViT-B-32, leveraged its pre-training on robotics datasets to achieve vision-language alignment, alleviating further alignment efforts.\n\n**Conclusion:**\n\nOur findings suggest balancing stronger LLMs' performance benefits against the computational costs of larger parameters. We recommend prioritizing capable LLMs within similar size constraints and favoring MLLMs or models pre-aligned with the robotics domain. Future work will explore **more powerful model backbones** and **efficient fine-tuning techniques** to reduce computational costs. Additionally, we aim to investigate **lightweight subgoal modalities** (such as object masks, bounding boxes, or keypoint tracking) to balance cost and guidance. We also hope PERIA can serve as a **foundational model** for MLLM-based embodied manipulation research, offering a more efficient starting point than learning from scratch with non-robotics-tuned MLLMs.\n\n# Q2 Real Robotics Evaluation\n\nGood suggestions! We highly agree on the importance of demonstrating PERIA's capabilities in real-world settings, but we face limitations due to the absence of robotics in our lab. To address this, we use the BridgeData v2 dataset, which features real-world manipulation videos and corresponding action annotations, to evaluate PERIA's potential through:\n\n1. **Language Planning Accuracy**: Assessing action prediction accuracy against provided annotations.\n2. **Visual Planning Fidelity**: Evaluating instruction-following ability by generating goal images from given action annotations.\n\nThis approach allows us to isolate and assess PERIA's high-level cognitive and planning abilities in real-world scenarios, excluding low-level policy execution. The results are as follows (We want to clarify that SuSIE is already included as a key baseline of visual planning method in our main paper. ViLA is also mentioned in related work section but not as baseline before rebuttal due to its unavailability of code and its similar training-free paradigm with the PAR approach baseline, differing primarily in the LLM backbone used. For this, we implement the base version of ViLA with GPT4V based on PAR for the limited timewindow):\n\n||**Language Accuracy↑**|**Visual Fidelity↓**|\n|--|--|--|\n|ViLA|0.69|-|\n|SuSIE|-|21.2|\n|**PEIRA**|**0.76**|**17.5**|\n\nDue to the free-form nature of language annotations in BridgeData v2, we calculate language accuracy with semantic similarity rather than token accuracy, as both metrics show consistent trends in the simulator results presented in Tab.2 of main paper. The visualization example can be found at Fig.6 in uploaded PDF. More details of related work can also be found at Tab.2 in uploaded PDF.\n\n**Key Findings:**\n\n- Using the same backbone Instructpix2pix as SuSIE, PERIA generated more coherent and task-relevant images, highlighting the importance of visual tokens and the integration of MLLMs in our approach.\n- Language accuracy decreased for open-vocabulary tasks in real domains compared to simulator performance for the Sim2Real gap, but PERIA still outperforms ViLA for the enhanced perception pretraining, which improves its ability to correctly interpret and describe complex manipulation tasks in real-world scenarios.\n\n**Conclusion:**\n\nWe will conduct comprehensive real-world evaluations as soon as resources permit and we may try SimpleEnv as a temporal substitute of real-robot before ready. Additionally, we will focus on collecting and incorporating more real-robot datasets for pretraining, enhancing PERIA's adaptability to real-world scenarios.\n\n---\n\n**Sincere thanks for your insightful review!  We hope our response can address the concerns.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# **General Response**

---

 **Sincere thanks to all the Reviewers for the valuable suggestions and recognition of our work!**

We sincerely appreciate all reviewers' time and efforts in reviewing our paper. We are glad to find that reviewers generally recognized our key contributions and clear presentation of our paper:

**Method:**  **Novel motivation** for integrating of multiple modalities for providing better guidance [Reviewer WmQe]. The proposed approach follows **a natural and reasonable** pipeline to tackle manipulation tasks with complex language instructions and combining language planning and visual generation is **sound** approach [Reviewer HJPE]. The method plans jointly in the language and image space and is **a good practice** [Reviewer C2qN]. 

**Experiments:** The paper achieves **convincing** experimental results compared with previous works and conducts extensive ablative studies to **mine more insights** [Reviewer HJPE]. The experiments are **comprehensive** and provide useful information on understanding the capability of the trained MLLM [Reviewer C2qN]. 

**Clearness:** The paper is generally **well-written** and **easy to follow** [Reviewer C2qN, Reviewer tWdi]. The **figures are helpful** for understanding the method [Reviewer tWdi].

We also thank all reviewers for their insightful and constructive suggestions, which helped a lot in further improving our paper. In addition to the pointwise responses below, we summarize supporting experiments added in the rebuttal according to reviewers' suggestions.

**New Experiments:**

1. Additional ablation studies on image conditional generation mechanisms, LLM backbone, consistency loss, pretraining tasks.
2. Additional comparisons with pretrained MLLM in robotics manipulation scenario.
3. Additional evaluations in real-robot datasets BridgeData v2.

---

We hope these new additions help address reviewers' concerns around computational resources, effectiveness of different components, potential usage in real-world scenarios, and comparison with existing methods. **We thank the reviewers for their time and feedback in improving the quality of our work, and we hope the revisions further highlight the contributions made. Please let us know if any clarification or additional experiments would further strengthen the paper. We would be happy to incorporate all these suggestions in the revised version.**

Meta-review: The paper combines multi-modal LLMs and diffusion models to perform reasoning for table-top manipulation tasks based on the language instruction and the imagined sub-goals. The paper received unanimous acceptance ratings from the reviewers. The AC also believes the paper has valuable contributions to the community. However, it would be better to go beyond table-top manipulation in the future iterations of the work. Acceptance is recommended.
Decision: Accept (poster)