Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflection

Authors: Yucheng Suo, Fan Ma, Kaixin Shen, Linchao Zhu, Yi Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text to image generation, visual instruction generation

Abstract:
Visual instructions for long-horizon tasks are crucial as they intuitively clarify complex concepts and enhance retention across extended steps. 
Directly generating a series of images using text-to-image models without considering the context of previous steps results in inconsistent images, increasing cognitive load.  Additionally, the generated images often miss objects or the attributes such as color, shape, and state of the objects are inaccurate.
To address these challenges, we propose LIGER, the first training-free framework for Long-horizon Instruction GEneration with logic and attribute self-Reflection. LIGER first generates a draft image for each step with the historical prompt and visual memory of previous steps. This step-by-step generation approach maintains consistency between images in long-horizon tasks. Moreover, LIGER utilizes various image editing tools to rectify errors including wrong attributes, logic errors, object redundancy, and identity inconsistency in the draft images. Through this self-reflection mechanism, LIGER improves the logic and object attribute correctness of the images.
To verify whether the generated images assist human understanding, we manually curated a new benchmark consisting of various long-horizon tasks. Human-annotated ground truth expressions reflect the human-defined criteria for how an image should appear to be illustrative. 
Experiments demonstrate the visual instructions generated by LIGER are more comprehensive compared with baseline methods. The code and dataset will be available once accepted.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730380316286

thread_type:
official_review

<thread_object>
creator: reviewer_unjf

<content>
summary:
This paper proposed a training-free framework for generating long-horizon visual instructions with logic and attribute self-reflection. It drafts consistent images for each step, using historical prompts and visual memory, and corrects errors with image editing tools. Experiments show it produces more comprehensive visual instructions than baseline methods.

strengths:
1.  The training-free framework and self-reflection mechanism of LIGER provide a novel approach to visual instruction generation for long-horizon tasks.
2.  The writing is clear and well-structured, making the concepts easy to understand.
3. The manually curated benchmark tests effectively demonstrate the advantages of the images generated by LIGER in terms of comprehensibility.

weaknesses:
1. In Automatic evaluation, the authors didn't evaluate the quality of the generated images, only assessing alignment and consistency.
2. In this paper, the benchmark's narrow focus on cooking might not capture the full spectrum of complexities and variations present in long-horizon tasks across different industries or activities. It would be beneficial for the authors to provide more details on the types of tasks included in the benchmark.

questions:
1. How does LIGER's self-reflection mechanism ensure that the identification and correction of errors in images are accurate and error-free? Is there a possibility of over-correction or failure to recognize certain types of errors? How you balance Over-consistent and Identity inconsistent?
2. How much time consumption does LIGER introduce?
3. Given the benchmark's focus on cooking tasks, how does LIGER address the generalization to other long-horizon tasks, and are there plans to broaden the benchmark's scope? Like engineering and sports?
4. Is there a specific template or structure that the benchmark follows for the steps involved in the tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658454455

thread_type:
official_review

<thread_object>
creator: reviewer_p3kh

<content>
summary:
This paper focuses on long-horizon text-to-image generation tasks and proposes a train-free framework. Long-horizon text-to-image tasks face two main challenges: temporal consistency and the accumulation of attribute errors. To address this, the work injects historical text descriptions and visual tokens into a Diffusion Model, then leverages GPT-4 and the segmentation large model LISA to detect and locate errors in the images. Image editing tools are subsequently employed to correct these errors. Meanwhile, the framework uses DDIM inversion to obtain the features of the edited images. Additionally, the paper introduces an evaluation dataset of over 500 samples to assess the effectiveness of long-horizon image generation.

strengths:
- The paper is well-structured and smoothly written, making it easy for readers to understand.
- The figures and tables in the paper are well-designed, making them easy to understand.
- The motivation is well-defined. To address the temporal consistency issue in long-horizon visual instruction generation, it proposes injecting historical text and visual information into the diffusion model. To tackle the attribute error problem, it introduces a method of using MLLMs to detect errors and then calling an agent to correct them, which demonstrates a certain level of novelty.

weaknesses:
- This paper only validates its effectiveness on a self-proposed evaluation dataset, which is somewhat unfair. It is recommended to find more suitable and fair evaluation datasets for verification. 
- The evaluation dataset used in this paper is limited to cooking scenarios, lacks generality, and is relatively small, with only about 500 samples.

questions:
- In Section 3.1, "Visual Memory Sharing" uses an attention module to inject historical visual information into the Diffusion model. However, the paper claims that the method is train-free, so where are the weights for this attention module loaded from? If this attention module is not trained on the specific task, can it really be used in a zero-shot manner? I have doubts about its performance.
- The symbol $O_i$ in Eq.3 may lack an explanation, which might leave readers wondering how $O_i$ is used in the Diffusion model.
- In Section 3.2, it is recommended to briefly explain how the image editing tools mentioned (such as DragonDiffusion, SD inpainting Rombach, LAMA, etc.) are used in this method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660022973

thread_type:
official_review

<thread_object>
creator: reviewer_nhsy

<content>
summary:
The paper proposes a method for visual story generation that is completely training-free but consists of many steps. The main parts of the method include keeping a historical prompt and a visual memory for consistency and self-reflection for refinement. The method shows improved results over the previous methods.

strengths:
1. the paper is written well. 
2. the method section reads clearly. 
3. the paper shows improved results over the baselines.

weaknesses:
1. the method has a lot of moving parts. i would have liked to see some error analysis regarding how does the approach work if one of the components makes a mistake. for example, what happens if gpt-4o misses some details? 
2. the current work is overly reliant on the closed-source gpt-4o. there are also plenty of open source models available. some ablation on using open source vlms could be useful and beneficial for the community. 
3. error bars are not reported. 
4. how expensive is this approach. if the sequence length is too long of the tasks. would that mean that we will need to store a lot of 'visual memory'? some discussion on this could have been helpful.

questions:
i have some serious concerns about the motivation of this task. what can be the differences of this particular task with semantically consistent video generation for a scene? 

please also look at the weaknesses. overall i like the paper. but if these concerns are addressed i can update my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705029921

thread_type:
official_review

<thread_object>
creator: reviewer_fpfq

<content>
summary:
This manuscript introduces an innovative task named"Visual Instructions for Long-Horizon" which aims to generate a series of continuously aligned visual images corresponding to textual instructions. The manuscript proposes four self-reflection methods that leverage both visual and historical prompts. To prevent cumulative deviation and help generate along the correct trajectory, an approach termed Inversion-Based Visual Memory Calibration is proposed. The proposed method is noteworthy for its approach to addressing attribute errors and inconsistencies by utilizing existing mLLM tools.

strengths:
1.	The proposed task and solution are indeed novel. Ablation experiments validate the contributions of the various proposed modules, including the visual prompt, historical prompt, self-reflection mechanism, and inversion-based memory calibration.
2.	Numerous qualitative experiments demonstrate that the proposed tool-based self-reflection method maintains alignment with textual instructions, ensuring that the generated images adhere to contextual logic, thereby validating the overall efficacy of the approach.

weaknesses:
1.	This manuscript would benefit from a more comprehensive comparative analysis. I recommend including comparisons with additional train-free methods, such as [1] and [2], as well as approaches that share similar concepts, like [3], and non-tool-based methods, such as [4].
2.	Furthermore, not all metrics proposed by the authors are original, and the use of multimodal large language model (LLM) reasoning to evaluate images also not original. It is advisable to revise the relevant statements in the contributions section to reflect this accurately.
3.	While the research method is innovative, the reliance on several pre-designed strategies and external models for image refinement may not be particularly efficient in practical applications. This could be seen as a limitation of the manuscript and the authors are encouraged to provide an appropriate discussion of this issue.

+ [1] Coherent Zero-Shot Visual Instruction Generation
+ [2] Training-Free Consistent Text-to-Image Generation
+ [3] Consistent self-attention for long-range image and video generation
+ [4] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

questions:
Why is there no evaluation of the effect of “+self-reflection“ alone in the ablation experiments? It appears to be directly combined with memory calibration in the final method without independent verification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732263762174

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Update Information

comment:
We thank the reviewers for their insightful and professional suggestions. Based on the comments, we updated the manuscript in the following sections:

- **Quantitative experiments** (Appendix A.1.)

    1. Additional ablation on the self-reflection mechanism in Table 4 and Table 5.

    2. Ablation on MLLM model in Table 6.

    3. Variance test in Table 7. and Table 8.

    4. Image quality evaluation in Table 9.

- **Qualitative experiments** (Appendix A.2.)

  1. Comparison with Consistory and StoryDiffusion in Figure 8.

  2. Error case analysis in Figure 9.

  3. Qualitative results on other scenarios in Figure 10.

  4. Recipeplan dataset example in Figure 11.


- **Text Revision**

  1. We modified the statement of evaluation metrics.

  2. Adding an Explanation of how the output in Eq 3. is used. The inputs of the editing tools are clarified.

  3. Limitation discussions are included in the appendix A.3.

All the modifications are marked blue.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732534966913

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder Regarding our Rebuttal

comment:
We sincerely thank reviewers for their professional and detailed reviews. 

As the discussion period deadline approaches, we are looking forward to receiving feedback on our rebuttal, so we can engage in further discussions and refine our work.

We understand the reviewers may be under a large workload, and we are grateful for their time and patience.

Best regards, authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents LIGER, a novel training-free framework for generating long-horizon visual instructions which utilizes historical prompts and visual memory. The framework is designed to improve the consistency and accuracy of instructional outputs. According to the reviewers (UNJf, nhsY), LIGER shows promising results on a new benchmark dataset, with particular emphasis on cooking tasks.

The strengths of the paper include the innovative training-free approach and the self-reflection mechanism of LIGER, which are recognized as significant contributions to the field (UNJf, P3KH). Additionally, the paper is well-structured and clearly written, and the introduction of a new benchmark dataset is commended for providing a solid foundation for evaluation (P3KH, UNJf, nhsY, fPFq).

However, the weaknesses lie in the limited scope of the dataset, focusing primarily on cooking scenarios which may affect the generalizability of the findings (P3KH, UNJf). The reliance on proprietary models like GPT-4 also raises concerns about accessibility and replicability (nhsY, fPFq). A notable oversight is the absence of a detailed error analysis and robustness testing (nhsY, UNJf).

The decision to accept rests on the novelty and effectiveness of LIGER, despite the raised issues. The authors' thorough responses during the rebuttal period have been considered to balance out the concerns about generality and reliance on closed-source models.