Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection
Authors: Yucheng Suo
Fan Ma
Kaixin Shen
Linchao Zhu
Yi Yang
Venue: ICLR 2025 Poster (2025)
Keywords: text to image generation
visual instruction generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a training-free framework for long-horizon visual instruction generation with logic and attribute self-reflection.

Abstract: Visual instructions for long-horizon tasks are crucial as they intuitively clarify complex concepts and enhance retention across extended steps. 
Directly generating a series of images using text-to-image models without considering the context of previous steps results in inconsistent images, increasing cognitive load.  Additionally, the generated images often miss objects or the attributes such as color, shape, and state of the objects are inaccurate.
To address these challenges, we propose LIGER, the first training-free framework for Long-horizon Instruction GEneration with logic and attribute self-Reflection. LIGER first generates a draft image for each step with the historical prompt and visual memory of previous steps. This step-by-step generation approach maintains consistency between images in long-horizon tasks. Moreover, LIGER utilizes various image editing tools to rectify errors including wrong attributes, logic errors, object redundancy, and identity inconsistency in the draft images. Through this self-reflection mechanism, LIGER improves the logic and object attribute correctness of the images.
To verify whether the generated images assist human understanding, we manually curated a new benchmark consisting of various long-horizon tasks. Human-annotated ground truth expressions reflect the human-defined criteria for how an image should appear to be illustrative. 
Experiments demonstrate the visual instructions generated by LIGER are more comprehensive compared with baseline methods. The code and dataset will be available once accepted.

Published as a conference paper at ICLR 2025
LONG-HORIZON VISUAL INSTRUCTION GENERATION WITH LOGIC AND ATTRIBUTE SELF-REFLECTION
Yucheng Suo
Fan Ma Kaixin Shen ReLER Lab, CCAI, Zhejiang University, China
Linchao Zhu
Yi Yang
Task: How to make matcha blondies?
Line 8x8 inch baking pan with parchment paper
cup purpuse four and matchu powder...
Fold 1/2 cup of white chocolate chips
Pour the batter prepured
Bake for 20-25 minutes or until toothpick inserted into the center...
Let the blondies cool in the pan before cutting into
baking
squares
LIGER SKCS
Missing object:
Missing history White chocolate should in the bowl
Wrong aftribute: batter green
Inconsistent identity: showing green matcha blondies
parchment
With object:
Consistent identity: green matcha blondies Figure I: Visual instruction generated by LIGER, key merits are highlighted in the figure.
With history context:
Correct attribute: green
parchment paper
White chocolate
ABSTRACT
Visual instructions for long-horizon tasks are crucial as they intuitively clarify complex concepts and enhance retention across extended steps. Directly generat- ing a series images using text-to-image models without considering the context of previous steps results in inconsistent images, increasing cognitive load. Addi- tionally, the generated images often miss objects or the attributes such as color, inaceurate. To address these challenges, we
shape, and state of the objects fuining-free free framework for Long-horizer Instruction propose LIGER, the first training- GEneration with logic and attribute self-Reflection. LIGER first generates draft image for each step with the historical prompt and visual memory of previous steps. This step-by-step generation approach maintains consistency between im- ages in long- long-horizon tasks. Moreover, LIGER utilizes various image editing tools to rectify errors including wrong attributes. logic errors, object redundancy, and identity inconsistency in the draft images. Through this self-reflection mechanism, LIGER improves the logic and object attribute correctness of the images. To verify whether the generated images assist human understanding, manually curated a new benchmark consisting of various long-horizon tasks. Human-annotated ground truth expressions reflect the human-defined criteria for how an image should appear to be illustrative. Experiments demonstrate the visual instructions generated by LIGER are more comprehensive compared with baseline methods. Code and dataset are provided in htttps://githbee com/ suoych LIGER
"Corresponding Author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Humans learn to accomplish real-world tasks quickly through step-by-step text instructions. How- ever, without visual aids, it is challenging to imagine the object attribute status and judge the comple- tion status of the steps. For instance, when frying potato chips, merely reading the text description makes it hard to judge whether the chips are done. In contrast, viewing a video or a series of images accelerates individual understanding of task procedures, enhancing the success rate of completing various tasks. Generating illustrative visual instructions eases the comprehension burden and there-
fore becomes crucial and trending task (Lu al. 2023 Bordalo et al.) 2024 Menon et al. 2024
Damen et al. 2024). Moreover, generating visual instructions unleashes the potential applications including multi- multi-modal embodied agent perception and new task adaptation (Fan al., 2024] Zhou et al. 2024a). In this paper, we aim to generate series of images given task step descriptions.
naive approach to generating visual instructions involves directly using text-to-image models, such as Latent Diffusion Models (LDMs) (Rombach et al., (022). As Figure [ illustrates, this method results in images lacking object consistency, thereby confusing users about the relationships between steps. To enhance image continuity, GenHowTo To (Damen al.] (024) trains controllable U-Net [Ronneberger et al., 2015) model to enhance identity consistency. StackDiffusion (Menon et al. 2024) uses a diffusion model that takes concatenated latents from different steps input. Sequential Latent Diffusion Model (SLDM) (Bordalo et al., 2024) trains a language model to re- generate consistent textual descriptions and use latents of the previous steps enhance consistency. However, these approaches tend to produce overly consistent images that fail to capture changes in object states, An illustrative visual instruction should balance continuity with sufficient variabil- ity. This leads to the first challenge: the need for logical coherence across steps while allowing for appropriate changes.
Moreover, we empirically observe that the attributes of objects, e.g. color, state, and shape, might be incorrect in the images as depicted in Figure []] These errors can accumulate, impacting the generation result of other steps and posing a significant challenge in long-horizon tasks. This leads to the second challenge, i.e. attribute error and cumulation.
Our intuition for addressing these issues is to first generate a draft image for each step with the visual and textual context of previous steps, ensuring continuity between images. Then, through a process of self-reflection, we refine the draft images by adjusting for excessive continuity and correcting object attribute errors. This iterative approach not only prevents the accumulation of attribute errors in long-horizen tasks but also maintains appropriate logic relations across steps, similar to drafting and refining sketches.
To this end, we propose LIGER, a vaining-ffree framework for long-horizon visual instruction gener- ation consisting of (1) historical prompt and visual memory, (2) self-reflection and memory calibra- tion. Specifically, we leverage the reasoning ability of LLM to explicitly output history context for each step, facilitating relation comprehension. Inspired by the recent training-free identity consistent generation works (Zhou et al. 2024b Tewel al., 2024 LIGER additionally injects the previous step visual latent embedding into the frozen text-to-imss diffusion model, generating coherent im- ages for different steps. To further refine the object attribute the images and avoid over-consistent, a MLLM receives multi-modal in-context prompting and tells the rectifying solutions. Various edit- ing tools deal with errors including attribute error, object redundancy, identity inconsistency, and logic misunderstanding. Then the visual memory is calibrated to the embedding of the edited image via a latent inversion procedure, avoiding the error affecting future step image generation. Having this step-by-step generation manner, LIGER is capable of tasks with arbitrary steps without training.
To evaluate whether the generated visual instructions align with human comprehension, we curate a benchmark containing 569 long- horizon tasks along with and logic relations. Moreover, we evaluate the method from semantic alignment, logic correctness, and illustrativeness. Results show that LIGER surpasses baseline methods by large margin. User studies and qualitative comparisons further verify that visual instructions generated by LIGER are more illustrative. In summary, the contribution of this paper includes:
ground truth expressions
(1) We propose LIGER, the first training-free free framework generating visual instructions for long-
horizon tasks.
2

---

Published as a conference paper at ICLR 2025
(2) History prompts, visual memory, and self-reflection introduced to promise logic coherent and object property accuracy. Inversion based memory calibration is devised to avoid exposure bias.
(3) A dataset for long-horizon tasks with human-annotated expressions is curated to evaluate the effectiveness of LIGER.
2
RELATED WORK
2.1 IMAGE GENERATION AND EDITING
Recent advances in modti-modal diffusion models (Rmeesh et al., 2022 Koh et al., 2024 Peebles
& Xie 2023 Saharia et al. 2022: Ho et al. 2020 Song et al. 2020) show remarkable ability to
generate images in high fidelity. Among these models, Latent diffusion models (LDMs) (Rombach et al. 2022) show strong robustness and semantic richness since the denoising process is conducted
on the latent space. Based on LDMs, researchers further exploit exciting application topics including
controllable image generation (Zhang al.,
2023 Mou et al.] 2024bb Liang et al.
2024 Ma et al.
2024b), personalized generation (Ruiz et al.
2023 Kumari et al. 2023] [Shi et al.] 2024a Gal et al.
2022). coherent generation Zhou al. (2024b); Tewel et al. (2024), image editing (Brooks et al,
2022 Nichol et al. 2021; Kim et al. [2022] Mou et al. 2023 Shi et al. 2024bb Mou al.) 2024 etc. Storydiffusion (Zoou et al. 2024b) and Consistory (Tewel et al.4 share
2023 Hertz et al.
a similar idea of KV sharing to generate content-consisten images in a training free manner.
Image editing, different from previous image generation tasks, involves manipulating the contents of the given image (Pan et al., (023). There are various settings for editing, including text-driven
(Tumanyan et al.) [023] Cao et al. 2023 Kawar et al., 2023), location- based (Chen et al. 2024b
Avrahami et 2023 Nichol et al. 2021). appearance modulation (Chen et al. 2024a Mou et al.,
2023), object moving Pan et al. 2023 Mou et al. 2024a). etc. Common techniques for text-guided
editing involve modifying the latent attention module e.g. MasaCtrl (Cao et al. 2023) or fine-tuning
a model e.g. Instructpix2pix and SmartEdit (Brooks et al. 2023: Huang et [l.44)22)))) Location- based editing leverages the region restriction prior like bounding box, mask, or even point (Ling
ct al. 2023). Our method utilizes different image editing methods to rectify the errors in the image.
2.2 TASK INSTRUCTION GENERATION
Generating procedures for a task is a popular research topic as has potential application scenarios like intelligent assistants (Shen et al. 2024 Surís et al. [023 Yang et al.] 2024b), embodied agents navigation (Liu et al. [023] 2024) and instruction comprehension (Xu et 2003), etc. This paper focuses visual instruction generation, i.e. generating series of images to explain task. Previous work like TIP (Lu et al., 2023) and MGSL (Wang et al., 2022) generates textual instructions for the tasks based on the visual information. StackDiffusion (Menon et al. 2024) is the first method for generating coherent visual instructions, which is trained on step-wise annotated VSGI dataset YYang al. 2021). However, the step number for task is restricted. GenHowTo (Damen al. 2024) infers states before and after actions by learning from instructional videos. Sequential Latent Diffusion Model (Bordalo al. 2024) trains a model output coherent text prompts for the text- to-image diffusion model, therefore generating coherent images. Phung et al. [Phung et al. (2024) propose a training-fres method yet the utmost step length is 5. Different from previous methods, LIGER is training-frre free method that can deal with long-horizon tasks having large step lengths.
2.3 TOOL-BASED METHODS
As the growing emergent capabilities of LLMs (Achiam et al. 2023). researchers deal with complex
vision and natural language tasks (Yao et al., 2022] Ma et al. al. 2024a) by using surrogate tools (Schick
et al. 2024) or programming languages, pioneer works inc VisProg (Gupta & Kembhavi] 2023),
ViperGPT (Surís et al. 2023), HuggingGPT (Shen et al. 2024), etc. In the image and video genera- tion area, LLMs are widely used for arranging layouts (Gani et al. 2023 Lin et al.) 2223 Lian et al.)
2023 Yang et al. 2024a). enriching textual prompts (Cheng et al. 2024 Long et al. [2224] Yuan al. 2024). verification (Wu et al. (024). Our method 18 also a based framework unleashing the strong reasoning ability of Multimmoal Large Language Models (MLLMs)) to call tools, enrich textual information, and do self-refiection.
et al. 2024 Zhuang al. 2024), tool calling (Wang et
3

---

Published as a conference paper at ICLR 2025
(1) Draft Image Generation
Step Start by preheating the grill to burn any
History Analyse (GPT4O)
Generation
Clean grates
Wipe exterior i of the grill with a damp cloth
Step 2: Use a wire brush to scrape off. the burnt food...
reinstalled
(SDXL Atin)
Inout:
Historcal Prompt
Step Description
Visual Memory
Image
Previous Steps
Vi
(2) Reff-Reflettio
(3) Post Pest-Processing
Error Detector
Regenerate
Referee
(GPT4O)
(SDXL)
(GPT4O)
Draft Image
Locator (LISA)
Rectifier
Revised
Memory Calibration
Output
V
Identity (DragonDiff)
Attribute
Removal (Lama)
(SD Inpaint)
Figure 2: Pipeline overview. LIGER generates visual instructions step-by-step, starting with (1) generating a draft image taking the visual memory, step description and historical prompt as input. (2) The error detector identifies the error and the corresponding tool fixes it, generating revised image. (3) The referee tool compares the two images and selects one as the final output. LIGER further uses inversion- guided visual memory calibration for future step generation.
METHOD
The overall pipeline of LIGER shown in Figure 2 Harnessing the visual memory and historical prompt, LIGER generates draft image for each step. Self-reflection mechanism corrects the errors the draft images. To prevent error accumulation the long horizon generation procedure, LIGER calibrates the visual memory according to the edited image through inversion.
3.1 HISTORY-AWARE DRAFT IMAGE GENERATION
Given a set step descriptions S" for a task Q of n steps, our goal is to generate a series of coherent images V for corresponding descriptions without training. To this end, a frozen text-to- image diffusion model generates a draft image V for step for each step in the task. The diffusion model generates single image through iterative denoising steps. Specifically, a U-Net network U predicts the noise
(1) where z, is the latent representation at timestep / and e the textual condition. Naively generating individual images using the step description ignores the continuity between steps. Therefore, first introduce the historical prompt and visual memory to enhance consistency.
4 U(1,,)) c),
Historical prompt. Each step description S, E often describes an incremental action relative to the previous scene settings. For instance, in a task cooking potato chips, two consecutive steps are: place the potato chips on a paper towel to drain excess oil and seasoning with salt and pepper. Without context, the eext-to-image diffusion model is unaware that salt and pepper should be added to the potato chips. Motivated by this, we use an LLM to generate a description H, for each step that specifies which objects from the previous steps should appear in the current step. The text condition c for the diffusion model formulated as
c= ES H.),
(2)
where Fy is the text encoder network.
Visual memory sharing. Merely using the historical prompt results in generating objects with varied appearances and backgrounds. To address this issue, inspired by StoryDiffusion (Zhou al. 2024b), we incorporate visual embeddings from the previous step as the visual context. When generating the draft image V of step i, we randomly sample several visual feature tokens Pi-1 RMXC of the previous image V,-1 V and inject them into the self-attention operation in the U- Net. Here M represents the number of sampled tokens and C' is the number of feature channels. The query input of the attention operation is the current image feature tokens p; R NxC, the key and value inputs are the concatenation of p;-1 and p;. The procedure can be formulated as:
Q: W'',, K; W* wy Pi p:- V; W [p., P:-
(3)
O Attention (Q,, Ki, Vi),
4

---

Published as a conference paper at ICLR 2025
Draft Image
Afler Sclfrefhetton
Draft Image
Afler Self-reflection
(b)
Previous Stop:
Current Stepe Heal olive oil skillet
Begining Step: Prehear the oven 525 degrees Fahrenheir
Stape
small mestballs
Over-consistent
Object Redundant
Draft Image
After After Selfeceflcttion
Draft image
After Self-reflection
(c)
(d)
Previous Step: Blend mixture until smooth
Current Step: Pour the milksiake into glass
Begining Step: Place the chicken breasts cutting board
Inconsistent identity
Wrong Attribute
Figure 3: Visualization of different eтrоr types and the effect of self-reflection. The motivation of self-reflection is to rectify errors including (a) over-consistent, (b) object redundant, (c) inconsistent identity, and (d) wrong attributes.
where W W are the linear projection layers for the query, key, and value respectively. The output feature O₂ is used as the input of the next layer in the UNet U. Note that neither the historical prompt nor the visual memory are provided the first step of any task.
3.2 TOOL-BASED SELF-REFLECTION
Empirically, observe errors in the draft images illustrated in Figure 3 Leveraging the ad- vanced multi-modal capabilities of MLLMs, LIGER employs the state-of-the-art GPT4O model as error detector to identify errors across four aspects, then output tool calling instructions to revise the draft images. For accuracy in eгтоr recognition, the error detector is prompted with multimodal in-context examples. The prompt template is attached in the appendix.
Over-consistent. long-horizon tasks, not all steps necessarily require visual continuity. For ex- ample, consider the task of cooking wanton noodles where the steps Drain the noodles and rinse with cold water and In separate pan, heat some oil are sequential yet independent. The former step con- cludes noodle preparation, while the latter step initiates cooking with different ingredients. These steps lack logistic connection, making consistency between the two images unnecessary. Breaking this consistency can help users recognize the transition issue, the егrоr detector assesses whether to maintain disrupt the continuity. If breaking con- sistency is required, the error detector outputs the error rectification instruction the format of Regenerate( NNw learl), then regenerates image according to the new description.
new step. To address the over-consistent
Identity inconsistent. Despite historical prompt and visual memory contributing to global visual consistency, local details occasionally remain misaligned, as depicted in Figure g To enhance lo- cal consistency, LIGER employs an intuitive method that aligns object appearances across images. Specifically, the emrrr detector compares objects in successive images, identifying whether two ob- jects should have similar appearance with the command Modi y' ('oject in Ví, object in V:-1). Subsequently, a locator tool, i.e. LISA (Lai al. 2024) outputs the masks of the objects according to the object descriptions generated by the error detector. Then the identity-keeping tool i.e. Drag- onDiffusion (Mou et al. 2023) receives the masks and modifies the object appearance the current image to match the previous image.
Wrong attribute. Correct object attributes such as color, shape, and state are crucial for instruc- tions. For instance, considering the tasks of baking chicken wings, the model may incorrectly gen- crate cooked chicken wings at the seasoning the prepared chicken wings step, where they should be raw. To address this problem, the error detector describes the desired attributes for object with the instruction Add(new description, object in V!). The same locator tool segments the object, then attribute reformulation tool i.e. SD inpainting Rombach et al. (2022) generates an image with modified object attributes according to the object mask. Redundant object. The last type of error is object hallucination, where frozen text-to-image
5

---

Published as a conference paper at ICLR 2025
diffusion models sometimes generate irrelevant objects for step description. For instance, in Figure 3 (b), the image illustrating preheating the oven mistakenly includes bread in the pan. The error detector flags the object to be removed in a format of Remove Remove(object in V.), and the locator tool pinpoints the specifie region. LIGER opts for the widely used LAMA (Suvorov et al. (2022) as an object removal tool. The tool removes the corre- sponding part of the image given the object mask. LIGER evaluates the image across these four as- pects iteratively and only modifies the draft im- age for once. In other words, once an error is detected, the verification procedure halts, and the corresponding editing operation is applied to the draft image. It is also worth noting that the over- consistent and identity inconsistent errors are veri- fied based two consecutive steps, while wrong attribute and redundant object are conducted as single-image verifications. The execution order of the pipeline is detailed in Algorithm | Conse- quently, for the draft image of the first step each task, LIGER only performs attribute modification or object removal. Having the various tools col- laboratively verify the images, LIGER generates il- lustrative visual instructions for long-horizon tasks with accurate logic in a self-reflection manner.
Algorithm 1 Single Step Self-reflection
Input: Draft Image V Previous Image V--1. Step Description S,-1, and Task Q. if = 0 then
A [Attribute, Object]
else
+ [Relation, Identity, Attribute, Object)
end
for A in A de if A in [Attribute, Object] then
error + Detect Vi St, Q)
clse
error +- Deteci(V,, St, Q, S;-1, Vi-1) end
error is detected then
Vi Rectify(VV St, Q)
V + C+mpprcc(V Vi
break
end
end
if V: V. then Refresh (V:) end Output: Final Image V..
3.3 JUDGEMENT AND MEMORY CALIBRATION
The aforementioned tool-based self-reflection generates a revised image V. Yet every rose has its thorn, self-reflection sometimes produces low-quality images or makes incorrect judgments during editing. To stabilize the pipeline predictions and improve robustness, we devise a referee tool to compare the draft image with the revised image. The referee evaluates both the quality and semantic alignment of the images and selects the better one as the final result V4. For more details, refer to the prompt template provided the appendix. Since LIGER generates images step by step, with visual memory providing visual continuity between steps, any error in the output image V; impacts the memory and can accumulate subsequent steps of image generation. To prevent this exposure bias, we propose inversion- guided visual memory calibration to update the memory.
Inversion -grided visual memory calibration. As discussed in Section 3.1 the visual memory is set of image feature tokens sampled from the previous generation step P:-1 E MxC These tokens are saved during the denoising process of the draft image, which exhibits a discrepancy with the features of the revised images. Since the revised image is generated in post-processing manner, storing the feature tokens alongside the generation process is inapplicable. However, the sampling process can be reversed using DDIM inversion which is formulated as:
++1 / 0/41/a, x Var+i (3.+1 - -.)).G. G.
(4)
where 0n is the variance schedule depend on timestep I, and the step-wise coefficient is set to 3. 11/a,-1, Ez is the noise predicted by the U-Net according to Eq I This allows us to obtain the attention output of the U-Net during the inversion procedure. Therefore, for the revised images, we apply this inversion operation over the same number of timesteps as in the generation procedure, effectively calibrating the visual memories to current image V, features. Correcting the visual memories prevents accumulated errors affecting subsequent image generation procedures.
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
For the historical textual prompt, the error detector and referee, we use GPT-40 (Achiam et al. 2023) introduced by OpenAl. The draft image generation uses the SDXL. (Podell al. (023) with

---

Published as a conference paper at ICLR 2025
100
Short: 30.93%
(Nedium: 44.64%;
Long: 24.45%
2.75
2.80
49.83%
+11.3%%
0.72
4.79%
SIDM Agentins
0.70
Tesk Numbeer
2.70
Dine Score
60
2.05
4.70%
2.60
0.66
$5.00 3.00%
40
2.50
2445
154
2
madium
long
short
medium
long
(a)
(b)
Figure 4: Dataset statistics and the influence of the step length of tasks.
guidance scale of 5 along with the Free-U plugin (Si al. 2024). The DDIM generation and inversion timesteps are set to 50. In terms of the visual memory, we set the number of the previous step image feature token M to half of the sequence length N, in other words. M N/2. For the location tool, we leverage the LISA-7B model (Lai et al. (024) to balance the performance and computing resources requirement. All experiments are conducted on a single RTX A6000 GPU.
4.2 DATASET
Effective visual instructions for long-horizon tasks should help users quickly understand complex procedures, but evaluating this capability remains challenging. Existing datasets lack appropriate evaluation methods for this aspect. To address this gap, we curate a new textual dataset consisting of 569 long-horizon tasks. These tasks are extracted from different resources including Howto
(Miech et al., 2019), Youcook2 (Zhou et al., 2018), and RecipeQA (Yagcioglu et al. 2018). The tasks focus the recipe domain, as cooking procedures typically involve strong logical relations between steps and require multiple stages. Specifically, we prompt the GPT40 model with in- context samples filter out tasks that are hard to illustrate and tasks that easy to accomplish, e.g. How to prepare a family meal for 20 people. The LLM then outputs step-by-step action de- scriptions for each task. Unlike existing planning datasets (Menon et al. 2024 Lu et al. 2023). our dataset offers following novel features:
nong-horizon tasks. The average number of steps per task is 9.8, with a minimum of steps and
a maximum of 17. The detailed distribution is shown in Figure 3 (a). We categorize the tasks into three types: short (6-8 steps), medium (9-11 steps), and long (12 more steps).
Manual annotations for step logics. For each task, we ask human annotators to select a pair of consecutive steps with continuous logic and another pair with logically independent steps. Our intu- ition is that the images corresponding to logically consistent steps should exhibit visual continuity, while the images of locally independent steps should be visually distinct.
Human-writter ground truth descriptions reflecting comprehension. We introduce novel an- notation for evaluating illustrative images. Since step descriptions often omit details about object attributes, we ask the annotators write a sentence describing what components should appear in the illustrative image for every step. These sentences reflect the appearance and state of the ob- jects with previous steps information. For example, the step Arrange the chicken wings on the wire rack from task How to bake chicken wings, one can infer the wings are raw and ready for baking. Therefore, a suitable illustrative expression could be The raw chicken wings are neatly arranged in single layer on the wire rack, with the spices and oil giving the skin a glossy, seasoned appearance. These expressions allow us to evaluate whether the generated images match human expectations of how an illustrative image should look. Annotation examples are provided in the appendix.
4.3 BASELINES
To thoroughly evaluate the effectiveness of LIGER and its components, we conduct both quantita- tive and qualitative comparisons with different baselines including: (1) Frozen SDXL (Podell et al. 2023). We simply generate visual instructions for the tasks using a frozen SDXL model prompted with the vanilla textual step descriptions. (2) Frozen SDXL + Visual memory (+V). The image generation model is provided with the visual memory while the text prompts remain vanilla step descriptions. (3) Frozen SDXL + Historical prompt (+H). The text prompt for the frozen SDXL model is modified by concatenating the step description and the historical prompt. No visual mem- ories are provided. (4) Frozen SDXL + Visual Memory + Historical prompt (+V+H). The image generation model is equipped with both visual memory and the historical prompt. This baseline can
7

---

Published as a conference paper at ICLR 2025
How to make california burrito?
Step1: Lay large
Step2. Specad gunea- mole the ollortilla
Add geneccus portins of curse tsada on top the mracomal
Stap::
Sept:
Spribli shrakted
Steps:
Add serving of Place handful Drizzle some
Shaps:
Supp:
Stepk:
Carefill bunito soll it femn tunile
Seppl: heated for cads
Bour toctilla clasn surtise
French fiics on tap lop the chese the pice de gallo French fiie
minttess
T21-Bndge
SLDM
LIGER
How make general tso chicken?
Suep::
bovil
Hal diop
bowt. skillet combining heean thip malum best
Camihh
fye or lagg dallo
chicken anal place tham toedd driin cropi
eloppsl choeen of
brtosinal pioze
are well coated
pieces uncil
with chicken
pices sett gilles
chickes
comrolanci. and...
wok
until thickened
skilkt and cock
and bubbly
sbilks tom to oast them with sorth Besasce
vide
SLDM Saraaaa
LIGER
make penne pasta arrabiata sauce?
Stepl:
Step:
Step3:
Stapk:
Stepd:
Stepé:
Skyp:
Skept: 3
Steptt Adda pinch ofsali and papper dee souce
Stepe: Simmer the mase for about 10 minues
Stepl:: Add the
Stpp2:
Step1.:
ha bege pol, Add 2 caps bring water
Cook parp
Doit the pada and set said:
in rigueate
2eimol Sauti unil
Toso the pasta Sevee depoot gamii Buikyy
of penne wa and
odisal:
panaa to the bellig water
aconting de pacaaa
pes
alvs ell our
gerlo exitee andie
shepped goldan ctil pereatt bown
ofuerato suuce and sar combuse
uninediesse pasa the sexe
coutel pinlo...
is the sance unil evenly depast
LIGER SLDM 722-Bridge
Figure Detailed qualitative comparisons on different long-horizon tasks. Zoom in see details.
also be considered LIGER without self-reflection. (5) T2I-Bridge (Lu et al. 2023 uses an LLM to imagine what the image for each step should depict based the step descriptions. T21-Bridge represents a type of re-captioning method. (6) Sequential Latent Diffusion Model (SLDM) (Bor- dalo et al. 2024) trains language model to produce coherent captions for the steps of task and uses a sequential context decoder to establish visual connections between images. Note that the text-to-image generation diffusion model is still frozen in SLDMM.
4.4 QUANTITATIVE EVALUATION
To assess the effectiveness LIGER, we conduct detailed quantitative comparison including: Automatic evaluation. We calculate several metrics using pre-trained models. First, evalu- ate the semantic aligmment between the images and human-annotated ground truth expressions by calculating the CLIP (Radford et al. 2021) similarity. These curated expressions reflect human un- derstanding of each step. Hence a higher CLIP-Score indicates that the images are more relevant to the expressions, implying that the images more illustrative for human comprehension.
8

---

Published as a conference paper at ICLR 2025
Automatic evaluation
GPT evaluation
Method
CLIP-Scoref
DINO-Score
BERT-Score f
Semantict
Logict
Illustrativeft
T2I-Bridge
2,4350
0.8576
0.8669
3.4717
2.5843
2.5150
SLDM
2,5054
0,6746
0.8694
3.3634
2.7286
2.5771
Ours
2.7555
0.6338
0.8743
4.1141
3.0595
3.0536
Table 1: Automatic quantitative evaluation and GPT evaluation results.
The second metric tests the logic correctness between consecutive steps. To evaluate image similar- ity, we use the DINO-v2 (Caron et al., 2021] Oquab et al., 2023) model and calculate the average Distance between the embeddings of the two images for the annotated step pairs. Inspired by the Signal-to-Noise Ratio formulation, we define the DINO-Score D, as the l2 distance between coherent steps divided by the l2 distance between independent steps which can be expressed as D2 = 12//2. This metric evaluates the ability to generate consistent images for logically coherent steps and distinct images for unrelated steps. A lower DINO-Score indicates higher logical accuracy.
The last metric evaluates the method performance in modality-transfer test. Our intuition is that illustrative visual instruction should help people summarize or describe the steps in text. There- fore, transfer the images back into text and measure the textual similarity with the annotated descriptions. Specifically, we adopt the widely-used BLIP-2 (Li et al. (023) model to gener- captions for images, then calculate the BERT-Score (Zhang al. 2019) between the cap- tions and descriptions. sults shown in Table [ demonstrate that LIGER significantly outperforms the baseline methods. GPT evaluation. We further harness the advanced logical reasoning and multi-modal per- ception ability of MLLMs evaluate the methods. Specif- ically, we prompt the GPT4O model rate how well each individual image aligns with its corresponding description. Then we input the entire image series to the MLLM and ask it to rate whether the image series illustrative with correct logics. The rating ranges from to 5, where represents low quality and 5 indicates perfect quality.
higher BERT-Score represents the image is more illustrative. The re-
pizza
Step5:
Stepó:
Step7:
Stepl::
Step10: Spread Sprakle shredded over dough
Step8:
Place the dough
Preheat the Punch down the Transfer the milled
bowl, rise...
475'F dougli drugh on pizza sbonp or...
(345"C)
over
IXQs
LIGER SDXL-V-HI SDXL-V SDXL-11
The results are shown in Table
D and the prompt templates are attached in the appendix.
User study. We invite 20 par- ticipants for the user study, with
each person asked to select the best generation results for 15 tasks. Participants rate aspects including semantic alignment, logical correctness, and task illustration. Results in Table 3 show that LIGER generates visual instructions that better match user preferences while maintaining mantic alignment and logic accuracy.
Figure 6: Qualitative ablation on different components.
4.5
QUALITATIVE COMPARISONS
The overall qualitative comparisons between LIGER and baseline methods are shown Figure We provide detailed comparison of LIGER with two prior works, namely T21-Bridge and SLDM. For the task How to make a California burrito, both Drl-Bridee and SLDM overlook that the seasoning and ingredients are added to the tortilla in Steps 3 to 7. In contrast, LIGER clearly illustrates the progressive process of adding different ingredients. Additionally, LIGER correctly visualizes the burrito being wrapped and heated in skillet. For the task How to make general
5

---

Published as a conference paper at ICLR 2025
iso chicken, LIGER presents smooth sequence, showing the process of frying the chicken pieces, making the sauce, combining sauce with chicken, and serving with rice. In comparison, SLDM omits the chicken pieces in Step 2 and incorrectly shows the finished dish in Step 5. T21-Bridge lacks visual continuity, making it hard to comprehend. To further demonstrate the effectiveness of LIGER in long-horizon tasks, visualize the results for the task How to make penne pasta with arrabbiata sauce consisting of 13 steps. SLDM shows an over-consistent process during cooking, while T2I-Bridge generates distinct images, In contrast, LIGER accurately illustrates the procedure.
4.6 ABLATION STUDY
Effectiveness of different components. We provide both qualitative and quantitative comparisons in Figure [6 and Table [ Results show that adding historical prompts and visual memory both improve the alignment between image and text semantics while also increasing logical accuracy. Additionally, these two components complement each other. When self-reflection is introduced, we observe a performance gain of +0.04 in CLIP-Score, reduction of -0. in DINO-Score, and an improvement of +0.002 in BERT-Score, demonstrating the importance of self-reflection. In Figure we observe that self-reflection correctly identifies which steps should be visually coherent and which steps should be distinet. Moreover, LIGER effectively shows the process of transforming pizza dough into a raw pizza. Essentially, the historical prompt and visual memory enhance visual continuity, while self-reflection aligns the images with human comprehension.
We further provide an example to highlight the importance of visual memory calibration in Figure 7 For Step of season the steak, the steak should be raw, yet the draft image incorrectly shows cooked appearance. After correcting the attribute, the subsequent step should also depict the steak as raw since the description does not indicate a state change. Without memory calibration, the steak in the next step still appears cooked, but with calibration, the steak correctly shown in a raw state.
CLIP-Score
DINO-Score
BERT-Score
Image
SDXL
2.5837
0.8516
0.8699
SDXL+V
2.6251
0.8239
0,8719
Season the stesk with walt and
SDXL++
2.6842
0.8224
0.8707
SDXL+V+H
2.7168
0.7459
0.8721
Ours
2.7555
0.6338
0.8743
W/O Memary Calibration After Memory Calibration
Table 2: Ablation different components of LIGER.
Method
Semantic
Logic
Illustrative
T21-Bridge
24%
18.3%
22.3%
skillee
SLDM
11.7%
21%
9.3%
Ours
64.3%
60.7%
68.3%
Table 3: User study on image-text semantic matching, Figure 7: Example of visual memory cali- logic continuity and illustrative.
bration.
Influence of task step length. In Figure 4 (b) and (c), we present the CLIP-Score and DINO-Score for tasks of varying lengths, comparing LIGER with SLDM. As the number task steps increases, the CLIP-Score of SLDM decreases significantly, while LIGER maintains stable performance. Ad- ditionally, the relative improvement in DINO-Score increases for medium and long tasks, indicating LIGER is robust to long-horizon tasks.
5 CONCLUSION
In this paper, we propose LIGER, the first training-free framework for long-horizon visual instruc- tion generation. LIGER first leverages historical prompts and visual memory to generate draft images step-by-step, enhancing continuity between images in long-horizon tasks. The tool-based self-reflection rectifies four types of errrrs in the draft images including over-consistent, identity inconsistent, wrong attributes, and object redundant. LIGER also deploys inversion -gideed visual memory calibration prevent error accumulation in the sequential image generation procedure. We also curate a new benchmark testing the alignment of generation results with human comprehension. We hope this work inspires future research instruction generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (U2336212). This work also supported in part by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No.2024CU) 142). We are grateful for the user study participants. This work was partially sup- ported by ZJU Kunpeng&Ascoe Ascend Center of Excellence. We also thank Dr Xiao Pan for discussing about the paper writing.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics 42(4) 2023.
João Bordalo, Vasco Ramos, Rodrigo Valério, Diogo Glória-Silva, Yonatan Bitton, Michal Yarom, Idan Szpektor, and Joao Magalhaes. Generating coherent sequences of visual illustrations for real-world manual tasks. arXiv preprint arXiv:2405.10122 2024.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: funing-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22560 225700 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervise vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Xi Chen, Yutong Feng, Mengting Chen, Yiyang Wang. Shilong Zhang, Yu Liu, Yujun Shen, and Hengshuang Zhao Zero-shot image editing with reference imitation. arXiv preprint arXiv:2406.07547, 2024a.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 6593-6602, 2024b.
Junhao Cheng. Xi Lu, Hanhui Li, Khun Loun Zai, Baiqiao Yin, Yuhao Cheng, Yiqiang Yan, and Xi- aodan Liang. Autostudio: Crafting consistent subjects in multi-turn interactive image generation. arXiv preprint ar arXiv:2406.01388 2024.
Dima Damen, Michael Wray, Ivan Laptev, Josef Sivic, et. Genhowto: Learning to generate actions and state transformations from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6561-6571, 2024.
Sheng Fan, Rui Liu, Wenguan Wang, and Yi Yang. Navigation instruction generation with bev perception and large language models. In ECCV, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint a2Xiv:2208. 01618, 2022.
Hanan Gani, Shariq Farooq Bhat. Muzammal Naseer, Salman Khan, and Peter Wonka. Lim blueprint: Enabling text-to-image generation with complex and detailed prompts. arXiv preprint arXiv:2310. 10640, 2023.

---

Published as a conference paper at ICLR 2025
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19953-1462,, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- -r. Prompt-to-pin poottoo--pppttoooo image editing with cross attention control. arXiv preprint arXiv:2208.0166 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8362-8371, 2024.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pp. 2426-2422442222 2206,2245,222 2022.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. LIm-grounded video diffusion models. arXiv preprint arXiv:2309. 17444, 2023.
Chao Liang, Fan Ma, Linchao Zhu, Yingying Deng, and Yi Yang. Caphuman: Capture your mo- ments in parallel universes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6400-6409, 2024.
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm-guided planning, arXiv preprint arXiv:2309.1509) 2023.
Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, and Yi Jin. Freedrag: Point tracking is not you need for interactive point-based image editing. arXiv preprint 2wxYi:2307700684,, 2023.
Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird's Sird's-eee-view scene graph for vision- language navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10988- 2023.
Rui Liu, Wenguan Wang. and Yi Yang. Volumetric environment representation for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 16317-16328. 2024.
Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Videodrafter: Content-consistent multi-scene video generation with Umm. arXiv preprint arXiv:2401.01256 2024.
12

---

Published as a conference paper at ICLR 2025
Yujie Lu, Pan Lu, Zhiyu Chen, Wanrong Zhu, Xin Eric Wang, and William Yang Wang. Multimodal procedural planning via dual text-image prompting. arXiv preprint arXiv:2305.0179, 2023.
Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance to visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13151-13160, June 2024a.
Wan-Duo Kurt Ma, Avisek Lahiri, John Lewis, Thomas Leung, and W Bastiaan Kleijn. Directed diffusion: Direct control of object placement through attention guidance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4098-4106, 2024b.
Sachit Menon, Ishan Misra, and Rohit Girdhar. Generating illustrated instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6274-6284, 2024.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
1630-2640, 2019.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:230..02421. 2023.
Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Diffeditor: Boosting accu- racy and flexibility on diffusion- image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8488-8497 2024a.
Chong Mou, Xintao Wang, Lianghin Xie, Yanze Wu, Jian Zhang. Zhongang Qi, and Ying Shan. Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296 4304, 2024b.
T2i-adapter:
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXi::230.007193, 2023.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your gan: Interactive point-based manipulation on the generative image mani- fold. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Quynh Phung, Songwei Ge, and Jia-Bin Huang. Coherent zero-shot visual instruction generation. arXiv preprint 2406::240060433377 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 9748-8763.. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 0661:225..0 1(2):3, 2022.
13

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings. part III 18, pp. 234-241. Springer, 2015.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning lext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusson models with deep language understanding. Advances in neural informa- tion processing systems, 1836479-36494 2022.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image gen- eration without test-time finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pp. 8543-8552, 2024a.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point- based image edit- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8839-8849, 2024b.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4743,, 2024.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456,, 2020.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 11888-11898, 2023.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 2159-2159, 2022.
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-fiee -free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 43(4): 1-18, 2024.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-iman tooeimaaeeeaaa translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Qingyun Wang. Manling Li, Hou Pong Chan, Lifu Huang, Julia Hockenmaier, Girish Chowd- hary, and Heng Ji. Multimedia generative script learning for task planning. arXiv preprint ar0i 12306, 2022.
14

---

Published as a conference paper at ICLR 2025
Zhenyu Wang. Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal Ilm agent for unified image generation and editing. arXiv preprint arXiv:2407.05600, 2024.
Tsung-Har Wu, Long Lian, Joseph E Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting I1m- controlled diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 6327-6336, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli kizler-Cinbil Recipeqa: challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint 10XX:::809.0000 2018.
Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and CUI Bin. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal llms. In Forty-first International Conference on Machine Learning. 2024a.
Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Bureh. Visual goal-step inference using wikihow. arXiv preprint (iti::104400845,, 2021.
Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. DoraemonGPT: Toward un- derstanding dynamic scenes with large language models (exemplified as video agent). In Forty- first International Conference on Machine Learning, 2024b. URL htttps://oppnreeeee
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.0362. 2022.
Zhengqing Yuan, Ruoxi Chen, Zhaoxu Li, Haolong Jia, Lifang He, Chi Wang, and Lichao Sun. Mora: Enabling generalist video generation via a multi-agent framework. arXiv preprint arXiv:2403.1 13248, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings the the IEEE/CVF International Conference on Computer Vision,
pp. 3836-3847, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 09675, 2019.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Tianfei Zhou, Fei Zhang, Boyu Chang, Wenguan Wang, Ye Yuan, Ender Konukoglu, and Daniel Cre- mers. Image segmentation in foundation model era: A survey, arXiv preprint 1iXi:24408119555 2024a.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffu- sion: Consistent self-attention for long-range image and video generation. arXiv preprint arXiv:2405.01434, 2024b.
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream vlog. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8806-8817, 2024.
A APPENDIX
A.1 ADDITIONAL RESULTS
Additional qualitative results shown in Figure 8 and Figure
15

---

Published as a conference paper at ICLR 2025
Task: How to Make General Tso Chicken
Step1:
Step2:
Step3:
Step4:
StepS: separate bood prepare the ouin by combeiing tbop
Step5:
Step7:
Step8:
Step8:
Step10:
so0g chicken to pleces
Hest of deep frvor lange skilktt and fry the chision plece....
Remove the chicken plezes and place them on poper scewess 1o excess of
Pear a thep of in bege sellet or wot ove modium heat
Ade the
And the fited chicken pieces the slilice and hose to cost them evently
Gor rich with chappes oniens and sosame soods
comstarch chicken until they well couted
mature
heated stilice and took antil trickenes and Butlcly
side
with the sauce
Task: How to Cook Parathas
Step1: white fivur
Stop2: Add salt and tabreazoonof
Step3:
Step4:
StepS:
Stop6:
Step7:
Step8:
Step9:
Step10:
Step1l: Gerve with your chokcs accomparimene
Sowty water and knead
Cover the dough Divid the the mature
Rull each bull isto that. round shape usinga rollng
Hext sawe folleillet
Plane the millec- Rip the dough out dough onto andiappls of or the tawo and gree on the cook until umall cookes iide bubbles appear
Cook bett until goiden brown
with a demp dath and letit rest for 20-30 minutes
dough into small equal- sited balls
oll ghlerrghee..
form smooth cough
pin
Task: How to cook calamari
Step1:
Step2:
Step3:
Step4:
Step5:
Step6:
Step7:
Step8: calamaii place them towal-inad plate to criin ovens oi
Step8:
CJean the freeh calomarizs by remowing the heat, beak. and innrrss
Sice
Marinote the calamani rings with lomon, juioo salt, and posper
ssperats bowr, mix fear. gellic
Dredge mari rased cabmert ringe the florr ribture, ensur well ocated
lieat deep dellet or fryento 350'F (L75%C)
Serve the oxlement hot leman wolges your cholce dipping suzce
cllmmni bodty ringinns
banches golcion brown..
Task: How to cook singapore curry laksa
Step1: Cotter rine noodless coconut milk curry paste vegesstles, chicken
2tep2:
Step3:
Step4: a heat
Step5:
Step5:
Step7:
Step8:
Step9: Serva hot, garnished line wedges
not water cook
Dosi she roodles them cs
cocenut milk, bringing
Add vegstables
Simmer proteirs
somz eurry
and proteino shamp chicken)
noodles according
pet
vegatablees
Task: How to Make California Burrito
Step1: Loy out large flur tartilss an deansuriaoe
Step2:
Step3:
Step4:
Step5:
Step6:
Step7:
Step8:
Step9:
Scroad a lover of goacamole in the center of
Add goreross portion of came saxda
Spr iide shrouded chene over the carme ssade
Addi servingef piza de gallo on top of the cheese
Flase a handiulof Franch free on top of the pico gello
Drizzle some sour cream over Franch
Caretully the
Gell the sumbo on hestel skilles couple misutes
then
giacamele
crupy
Task: How to Make Popcorn in a Pan
Step1:
Step2:
Step3:
Step4:
Step5:
Step6: keenels
Step7:
Step8: Shake ocasionally encure even
Step0:
Step10:
Mase large
A-52-3
2-
War unci the test kemnkk pop
Laten for the sounds diw down. then vemooe the pat from hest
Carefully
heeey- tahleggooms bemels into the vegezable and pan and rawer es heat alth
with
remova the id and pour the poperrs large
pooping
Figure 8: Additional qualitative results generated by LIGER. Zoom to see the detail.
A.2 ADDITIONAL QUANTITATIVE EXPERIMENTS
We present the following quantitative ablation studies in addition:
16

---

Published as a conference paper at ICLR 2025
Task: How to Make Stuffed Acorn Squash
Step1:
Step2:
Step3:
Stepd;
Step5:
Step6: in
Step7:
Step8: cookincooked quinraand sceson with herss and spian
Step9:
Step10: sullediqussh to the oown bain addiiioal
Step11: Remose from cann and serve hot
Prebwat
Cur the seom squaih scios
Place the squach falvess cut up
Brooke the scuashilahhes
Reaet the owen about
eopped
Rlll each roushed half with the quinos and suuaage manure e
owen (200%C)
the follet and
cook uniil tenter
minuees
Task: How to make Jelly Sandwich
Step1:
Step2: Openn jfly
Step3:
Step4:
Step5:
Step6:
two stoes reed them surface
eneenty
Ploce the other sike of tread on top the jully coverd dicce
Cutthe sandwich in half! desired and serve
from the jully
Task: How to Make Sweet Cream Cake
Step1:
Step2:
Step3:
Step4:
Step5:
Step6:
Stop7:
Step8:
Step8: thelstee
Step10: Allow coel Inserted red int minunes.. rester comees outclean
Step11: Once
your
together1 unseited
bow,, Add eggs one stantme, beating well after cach
another bowd, whise tngether the dry
Graduallyadd
Str 2
tesssoors
350 (2775))))))
inch mund
butter 2 cups ougugr unt light andition and fhally
sabkcspoon
ingredients
fowenite frosting
Task: How to Make Cabbage Rolls
Step1:
StepZ:
Step3:
Stept: Rellup the cabbage leaves foxding the addes andlose the diting
Step5:
Step6:
Step7:
Step8:
Step8:
Remose the com from hext of cablagge and blinch the
In boet,
Plane a portion of the meat mituure onto each sacbage leal
Rase the esbbage
Pour tomato
Remove oven
Covor the bakiiiigg with
comb ne ground chapped
rels scam side
sauce over the coblugge rolls
baise garriihed balee in preheated
down in baking dis
fresh desteed
cabblage covss...
Task: How to Make Pickles
Step1:
Step2:
Step3:
Stepd:
Step5:
Step6:
Step7:
Steg8:
Start gogtiiiriing cotumbers and preparing them by: wonhing thoruughy
Slice the cocumbers into sour desired ciness Spears chisa)
In exuraper, combine water, viregar, and pikking salt and bring
Add prike dil, and any other cesired sploes or herls to the bosking streger menure
Pack the cacember sRoes rightlyinoo stanklaee pars
Fhurthe hot imager moture cverthe oucumbers leaving small amount headspace
Seal thejars th sterliee lice and let them caal room temjeroture
Shore the pen the refrgeratoe or process them in hot water bont for longer
Task: How Make Gulab Jamun (Indian Sweet) from Package Mix
Step1:
Step2: Pour the package and so milk manng drocted package
Step3:
Step4:
Step5:
Step6: dough
Step7:
Step8:
Step9:
Step10:
Soak fried dough balls the ougarnyrup for fow hours
ball
room tamprrature
condamm and saftion
Figure 9: Additional qualitative results generated by LIGER. Zoom to see the detail.
(1) Effectiveness of different components. To thoroughly evaluate the importance, we conduct fur- ther ablation and report results in Table
17

---

Published as a conference paper at ICLR 2025
Method
CI.IP-Sooref
DINO-Score +
BERT-Score
SDXL
2.5837
0.8516
0.8699
SDXL+V
2.6251
0.8239
0.8719
SDXL+H
2.6842
0.8224
0.8707
SDXL+R
2.7270
0.7346
0.8732
SDXL+H+V
2.7168
0.7459
0.8721
SDXL+V+R
2.7428
0.7053
0.8734
SDXL+H+R
2.7440
0.6653
0.8740
LIGER
2.7555
0.6338
0.8743
Table 4: Different combination of Self-reflection and other components.
(2) Robustness towards MLLMs. LIGER integrates the strong GPT4o as the error detector and referee agent. To evaluate the influence of MLLM, we conduct an ablation by substituting the GPT4o model with two open-source models i.e. Pixtral-12B and VL.7B.. comparison is shown in Table 5 There is performance drop when using open-source models. We empirically find the output of the open-source model lacks reasoning ability and detail region comprehension ability, leading to misunderstanding the error type missing the obvious errors.
VL.-BB Automatic metric
Method
CL.IP-Score*
DINO-Score
BERT-Score
SDXL
2.5837
0.8516
0.8699
LiGER(QwenVL-77b)
2.7244
0.7305
0.8725
LIGER(Pixtral-122
2,7316
0.7061
0.8716
LIGER(GPP44)
2.7555
0.6338
0.8743
Table 5: Ablation MLLMs.
(3) Variance test. We run another trial on the whole 569 tasks and report the result in Table 6 Results indicate that there is relatively small variance in the three evaluation metrics. Never the less, LIGER still consistently outperforms baseline methods.
Method
CLIP-Score
DINO-Score
BERT-Score
T21-Bridge
2,4350
0.8576
0,8669
SLDM
2.5054
0,6746
0.8694
LIGER
2.7555
0.6338
0.8743
LIGER (new trial)
2.7738
0.6276
0.8745
Table 6: Variance test results on the whole dataset.
(4) Image quality evaluation. We evaluate the image quality using the GPT4O model to rate the quality of individual images of the whole 569 tasks from to 5 where a higher rating indicates higher quality. We further conduct user study, 5 participants view 50 images generated by each method and picked the best image from the three methods, and the win rate is reported in Table
Method
GPT-score
User Win Rate
T2I-Bridge 3,8525
41.2%
SLDM
3.7078
11.6%
LIGER 3.8976
47.2%
Table 7: Image quality evaluation.
A.3 ADDITIONAL QUALITATIVE EXPERIMENTS
We present the following qualitative comparison in addition:
(1) Comparison with relative works. Figure 10 shows a comparison between LIGER, Consistory and StoryDiffusion. LIGER shows clear object attribute change along the task procedure. Not that we also need to manually define a subject concept for Consistory and StoryDiffusion, which id not required by LIGER.
18

---

Published as a conference paper at ICLR 2025
Task: How cook pork chops?
Prof oat your oven to 400 degrees Fatrenhet
Beason the park chapa Heal litt over Sear the pork ahaps with sal and pepper on modium high heat with boin dides
Place the baking diet prehwated Oven
Reven for about 15-20 minutes
the skillet for about 2-3 minutes on each side
same oll
Consistory
StoryDifluskon
LIGER
Figure 10: Additional qualitative results generated by LIGER. Zoom in to see the detail.
(2) Error analysis. Figure 11 shows two types of errors in LIGER. For the reasoning error case, the precious step is putting an egg the batter and whisk. The current step is adding vanilla extraction. However, the error detector mistakenly believes the egg should be visible in the current step, which should not be after whisking. The referee agent finds the error and keeps the draft image as the final output. The right lane shows case of generation error due to the location tool failure. The balls are mistakenly removed and the referee agent finds the mistake considering image quality and picks the draft image the final output.
Error case: Reasoning
Error case: Generation
oggo
Stepe: Sär in Insepturs of vartils sotred
Curren Step: Form the mbture immo small balls or patties
Frevicus Step trage
Drafi Image [Tinal Output)
Edktd Image Lopk aggs should
Draft Image (Tinal Output)
Edited Image flo peranation qualtt)
Figure 11: Error case analysis.
(3) Qualitative results in different scenarios. In Figure 12 we show the generalization ability of LIGER on tasks in other scenarios.
(4) Comparison other short datasets. Figure 1 shows comparison the recipeplan dataset with short abstract textual instructions. LIGER generates images fluently indicating the task procedure.
A.4 LIMITATIONS
LIGER shows a strong ability generate visual instructions for various tasks, yet there are still limitations. First is that the action generation still uncontrollable. Future work may efficiently fine-tune the generation model add illustrative actions in the images. Second, the amount of ingredients is not controllable. identify how to visualize 1/2 cup of water and 1/4 teaspoon of salt. We believe future research generating videos for instructions might be an ideal way to show these details.
is challenging for the frozen text-to-image diffusion models to
Another limitation is that since LIGER is training free framework, the generation quality depends on the pre-trained diffusion model. We find the current models struggle to generate fine- fine-graine grained actions, part of complex structures. We believe LIGER can benefit from storng models.
19

---

Published as a conference paper at ICLR 2025
How to grow bell peppers?
Place the pot in a warm, seedlings sunny spot where it gets inches tail, thin them least 6 hours of sunlight daily
Monttor for pests and treat with appropriste methodd needed
Plant bell pepper seeds about 1/4 inch deep
Water the soil gently until it moiet
peppers when they reach the destred size and
one plant per pot
Task:
How to clean concrete steps? still-brislled brush dipped the dleaning solution
Sweep slaps to remove loose dirt and debris
cleaning solution water and mild detergent
Allow the stops dry
thoroughly with garden hose
completely betore use
Figure 12: Qualitative results other scenarios.
LIGER adopts many tools collaboratively generate illustrative instructions, therefore the inference time is longer. A speed test over 50 randomly selected tasks using locally deployed multi-modal large language model. LIGER takes around 120 seconds to generate instructions for 10-step task while the frozen stable diffusion model takes around 60 seconds on a single A100 GPU. In the future, using quantilized models or conducting accelerating strategies may increase the efficiency.
How to make fried egg with cheese?
Crack agga into bowl
Buttering pan
Pouring ogs
Adding Cheese
T2Bridg
SLDM
LIGER
Figure 13: Qualitative results on recipeplan dataset.
A.5 PROMPT TEMPLATES
We provide several detailed prompt templates here. The prompt for the four types of error
(I) Over-consistent:
Now I wish you to use logic reasoning ability to judge whether the current image should be regenerated. current step action does not have logic correlation, the previous scene description of current step description is wrong, then the current image needs to be regenerated. You need to change the step description and tell me in format of: *-Regeneaate (new step description)--. The new description should describe in detail about what objects should be in the new image. For example, (In-context. example with reason)
the previous and
20

---

Published as a conference paper at ICLR 2025
If the logic between the two step action descriptions ccherent, you just answer Correct, no error.
Please ignore the objects in the background and be tolerent to errors that are not obvious. You must tell why to make the choice and only correct the most obvious error with only one operation. Now, for a procedure of (task), the previous step is (pre_step), and the previous image is: (pre_image) the current step description is (cur_step) and the current image is: (cur_image)
(2) Identity:
Use logic reasoning ability to judge whether the subject object should look exactly the same between the two images based the previous and current step description. If there is an object should look totally the same but not, tell in a format of: M-Modify (object in current image, object in previous image)-x. -x. For example, (In-context example with reason). If the image is correct, you just answer Correct, no error. You only consider whether the foreground object appearance texture (not including shape and size) should be the exactly the same. Please ignore the objects in the background and be tolerent to errors that are not obvious. You must tell why to make the choice and only correct the most obvious error with only one operation.
Now, for procedure of (task), the previous step is (pre_step), and the previous image is (pre_image). The current step description is (cur_step) and the current image is (cur_image).
(3) Attribute:
We are generating illustrations for a procedure. Please evaluate the image quality according to the current step description. You need to identify whether the salient main object attribute matches the step description. If the attribute is not ideal, you need to tell me how to add it in a format of:
+-Add (object description, place to add the object)
Must start with A-Add and end with )-+,
For example, (In-context example with reason)
If the image is correct, you just answer Correct, no error. You only consider the forground salient object ignore the objects in the background and be tolerent to errors that are not obvious. You must tell why to make the choice and only correct the most obvious error with only one operation. The current procedure is (task) and the current step is (cur_step), and the image is (cur_image)
the image. Please
(4) Redundant:
are generating illustrations for a procedure. Please evaluate the image quality according to the current step description. need to identify whether there are redundant objects. If redundant object exists, you need format like:
to tell me *-Remove (object description)-+. Must start with +-Remove ( and end with ) For example, [In-context example with reason). The objects described in the Previous scene part of the step description should not be regarded as redundant object. If the image is correct, you just answer Correct, no error. You only consider whether there is obvious redundant object in foreground of the image.
21

---

Published as a conference paper at ICLR 2025
Please ignore the objects in the background and be tolerant to errors that are not obvious. You must tell why to make the choice and only correct the most obvious error with only one operation. The current procedure is (task) and the current step is (cur_step), and the image is (cur_image).
The prompt for comparing the draft image and the revised image is:
Please pick the better image between the two images considering the image quality and the alignment with the current step description. You only answer or В within one word. For example, (In-context example with reason). Now, consider the following step, linput_curl] and the image A is: (image_imgg initial), the image B is: {image_final); final).
The prompt for GPT evaluation is:
(1) Single image evaluation
Rate the image from 1 (worst) to 5 (perfect) considering: A. Does the image contains the objects should appear for the text description?
B. The image does not contain unrelated objects?
the text description, imagine the sub subject object attribute (adjective, state, color, texture), and does the image show correct attributes?
C. According
Give a rate from 1 to 5 on each aspect within 30 words in a format like A:rating*.
The text description is (input_overall) and the image is:
(2)lmage series evaluation
Please rate the series images from 1 (worst) to 5 (ok) considering:
A. In some consecutive steps, the images are coherent.
B. The image is diverse when the text descriptions deviate.
C. Overall, can the whole image series roughly describe the coarse idea of the task?
Give a rate from 1 to 5 on each aspect. Do not be too
strict since the task is hard. The response should be in format of A: (number of ratings) Reason: (reasons). Considering the task of (task). The text description for each step is (steps) and the image series are:
A.6 DATASET EXAMPLES DISCUSSIONS
We showcase an example of How to cook salmon fillet in the annotated dataset:
Most unrelated step: 1 and related step: and 5
Step 1:
Action: Preheat your oven to 40 (200
Ground Truth Description: modern metal oven is slightly open with the display showing 400*F. The interior oven light softly illuminates the empty metal racks inside, indicating the oven is warming up.
Step 2:
Ground Truth Description: baking sheet is lined with aluminum foil parchment paper.
22

---

Published as a conference paper at ICLR 2025
Action: Line baking sheet with aluminum foil or parchment paper.
Step 3:
Ground Truth Description: The salmon fillet. is placed the prepared baking sheet, skin side down, with the pink flesh exposed for seasoning. Action: Place the salmon fillet on the baking sheet, skin side down.
Step 4:
Ground Truth Description: Olive oil is being drizzled over the top of the salmon fillet, giving it a glossy sheen and helping to lock in moisture while baking.
Action: Drizzle olive oil over the salmon fillet. *
Step 5:
Ground Truth Description: The salmon is seasoned with salt and pepper, and herbs spices are sprinkled over the top for added flavor. Action: Season with salt and pepper, and add any desired herbs or spices..
Step 6:
Ground Truth Description: The baking sheet with the seasoned salmon fillet is placed in the preheated oven. Action: Place the baking sheet in the preheated oven.
Step 7:
Ground Truth Description: The salmon is baking in the oven for 12-15 minutes, turning opaque and flaking easily with fork when fully cooked.
Action: Bake for 12-15 minutes, or until the salmon is cooked through and flakes easily with fork.
Step 8:
Ground Truth Description: The baked salmon removed from the oven, resting for few minutes on the baking sheet to allow the juices to settle.
Action: Remove the salmon from the oven and let it rest for a few minutes before serving.
23


Reviews:
[{"review_id": "VCpyM5oldU", "reviewer": "Reviewer_fPFq", "summary": "This manuscript introduces an innovative task named\"Visual Instructions for Long-Horizon\" which aims to generate a series of continuously aligned visual images corresponding to textual instructions. The manuscript proposes four self-reflection methods that leverage both visual and historical prompts. To prevent cumulative deviation and help generate along the correct trajectory, an approach termed Inversion-Based Visual Memory Calibration is proposed. The proposed method is noteworthy for its approach to addressing attribute errors and inconsistencies by utilizing existing mLLM tools.", "questions": "Why is there no evaluation of the effect of “+self-reflection“ alone in the ablation experiments? It appears to be directly combined with memory calibration in the final method without independent verification.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed task and solution are indeed novel. Ablation experiments validate the contributions of the various proposed modules, including the visual prompt, historical prompt, self-reflection mechanism, and inversion-based memory calibration.\n2.\tNumerous qualitative experiments demonstrate that the proposed tool-based self-reflection method maintains alignment with textual instructions, ensuring that the generated images adhere to contextual logic, thereby validating the overall efficacy of the approach.", "weaknesses": "1.\tThis manuscript would benefit from a more comprehensive comparative analysis. I recommend including comparisons with additional train-free methods, such as [1] and [2], as well as approaches that share similar concepts, like [3], and non-tool-based methods, such as [4].\n2.\tFurthermore, not all metrics proposed by the authors are original, and the use of multimodal large language model (LLM) reasoning to evaluate images also not original. It is advisable to revise the relevant statements in the contributions section to reflect this accurately.\n3.\tWhile the research method is innovative, the reliance on several pre-designed strategies and external models for image refinement may not be particularly efficient in practical applications. This could be seen as a limitation of the manuscript and the authors are encouraged to provide an appropriate discussion of this issue.\n\n+ [1] Coherent Zero-Shot Visual Instruction Generation\n+ [2] Training-Free Consistent Text-to-Image Generation\n+ [3] Consistent self-attention for long-range image and video generation\n+ [4] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2AubClf4h", "reviewer": "Reviewer_nhsY", "summary": "The paper proposes a method for visual story generation that is completely training-free but consists of many steps. The main parts of the method include keeping a historical prompt and a visual memory for consistency and self-reflection for refinement. The method shows improved results over the previous methods.", "questions": "i have some serious concerns about the motivation of this task. what can be the differences of this particular task with semantically consistent video generation for a scene? \n\nplease also look at the weaknesses. overall i like the paper. but if these concerns are addressed i can update my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. the paper is written well. \n2. the method section reads clearly. \n3. the paper shows improved results over the baselines.", "weaknesses": "1. the method has a lot of moving parts. i would have liked to see some error analysis regarding how does the approach work if one of the components makes a mistake. for example, what happens if gpt-4o misses some details? \n2. the current work is overly reliant on the closed-source gpt-4o. there are also plenty of open source models available. some ablation on using open source vlms could be useful and beneficial for the community. \n3. error bars are not reported. \n4. how expensive is this approach. if the sequence length is too long of the tasks. would that mean that we will need to store a lot of 'visual memory'? some discussion on this could have been helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RfjnQVN4UH", "reviewer": "Reviewer_P3KH", "summary": "This paper focuses on long-horizon text-to-image generation tasks and proposes a train-free framework. Long-horizon text-to-image tasks face two main challenges: temporal consistency and the accumulation of attribute errors. To address this, the work injects historical text descriptions and visual tokens into a Diffusion Model, then leverages GPT-4 and the segmentation large model LISA to detect and locate errors in the images. Image editing tools are subsequently employed to correct these errors. Meanwhile, the framework uses DDIM inversion to obtain the features of the edited images. Additionally, the paper introduces an evaluation dataset of over 500 samples to assess the effectiveness of long-horizon image generation.", "questions": "- In Section 3.1, \"Visual Memory Sharing\" uses an attention module to inject historical visual information into the Diffusion model. However, the paper claims that the method is train-free, so where are the weights for this attention module loaded from? If this attention module is not trained on the specific task, can it really be used in a zero-shot manner? I have doubts about its performance.\n- The symbol $O_i$ in Eq.3 may lack an explanation, which might leave readers wondering how $O_i$ is used in the Diffusion model.\n- In Section 3.2, it is recommended to briefly explain how the image editing tools mentioned (such as DragonDiffusion, SD inpainting Rombach, LAMA, etc.) are used in this method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and smoothly written, making it easy for readers to understand.\n- The figures and tables in the paper are well-designed, making them easy to understand.\n- The motivation is well-defined. To address the temporal consistency issue in long-horizon visual instruction generation, it proposes injecting historical text and visual information into the diffusion model. To tackle the attribute error problem, it introduces a method of using MLLMs to detect errors and then calling an agent to correct them, which demonstrates a certain level of novelty.", "weaknesses": "- This paper only validates its effectiveness on a self-proposed evaluation dataset, which is somewhat unfair. It is recommended to find more suitable and fair evaluation datasets for verification. \n- The evaluation dataset used in this paper is limited to cooking scenarios, lacks generality, and is relatively small, with only about 500 samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xSXXZ7Eg4u", "reviewer": "Reviewer_UNJf", "summary": "This paper proposed a training-free framework for generating long-horizon visual instructions with logic and attribute self-reflection. It drafts consistent images for each step, using historical prompts and visual memory, and corrects errors with image editing tools. Experiments show it produces more comprehensive visual instructions than baseline methods.", "questions": "1. How does LIGER's self-reflection mechanism ensure that the identification and correction of errors in images are accurate and error-free? Is there a possibility of over-correction or failure to recognize certain types of errors? How you balance Over-consistent and Identity inconsistent?\n2. How much time consumption does LIGER introduce?\n3. Given the benchmark's focus on cooking tasks, how does LIGER address the generalization to other long-horizon tasks, and are there plans to broaden the benchmark's scope? Like engineering and sports?\n4. Is there a specific template or structure that the benchmark follows for the steps involved in the tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The training-free framework and self-reflection mechanism of LIGER provide a novel approach to visual instruction generation for long-horizon tasks.\n2.  The writing is clear and well-structured, making the concepts easy to understand.\n3. The manually curated benchmark tests effectively demonstrate the advantages of the images generated by LIGER in terms of comprehensibility.", "weaknesses": "1. In Automatic evaluation, the authors didn't evaluate the quality of the generated images, only assessing alignment and consistency.\n2. In this paper, the benchmark's narrow focus on cooking might not capture the full spectrum of complexities and variations present in long-horizon tasks across different industries or activities. It would be beneficial for the authors to provide more details on the types of tasks included in the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)