Chain-of-Focus Prompting: Leveraging Sequential Visual Cues to Prompt Large Autoregressive Vision Models

Authors: Jiyang Zheng, Jialiang Shen, Yu Yao, Min Wang, Yang Yang, Dadong Wang, Tongliang Liu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Visual In-context Learning

Abstract:
In-context learning (ICL) has revolutionized natural language processing by enabling models to adapt to diverse tasks with only a few illustrative examples. However, the exploration of ICL within the field of computer vision remains limited. Inspired by Chain-of-Thought (CoT) prompting in the language domain, we propose Chain-of-Focus (CoF) Prompting, which enhances vision models by enabling step-by-step visual comprehension. CoF Prompting addresses the challenges of absent logical structure in visual data by generating intermediate reasoning steps through visual saliency. Moreover, it provides a solution for creating tailored prompts from visual inputs by selecting contextually informative prompts based on query similarity and target richness. The significance of CoF prompting is demonstrated by the recent introduction of Large Autoregressive Vision Models (LAVMs), which predict downstream targets via in-context learning with pure visual inputs. By integrating intermediate reasoning steps into visual prompts and effectively selecting the informative ones, the LAVMs are capable of generating significantly better inferences. Extensive experiments on downstream visual understanding tasks validate the effectiveness of our proposed method for visual in-context learning.

Decision: accept (poster)
Reviewer scores: 5, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730121599714

thread_type:
official_review

<thread_object>
creator: reviewer_p93y

<content>
summary:
This paper aims to address an inherent challenge in the computer vision community, where, unlike language data, images lack clear logical structures. CoF prompting introduces intermediate steps in the visual learning process to enhance the ability of VLMs to understand and predict complex visual images. The authors propose using saliency maps to determine which parts of an image should be focused on sequentially, mimicking human cognitive processes. This method allows VLMs to perform better on fundamental vision tasks such as image segmentation, pose estimation, and object detection by progressively building a context around the test input. The authors comprehensively evaluated the proposed method across several models and datasets, with extensive experiments showing the efficacy of CoF prompting. It outperforms several baseline methods and demonstrates some improvements, such as Intersection over Union (IoU) and Pixel Accuracy (P-ACC). This paper also delves into ablation studies to assess the contributions of different components, providing insights into the role of cognitive reasoning, query relevance, and annotation diversity.

strengths:
1. Innovative Adaptation of CoT for Vision: The introduction of Chain-of-Focus (CoF) prompting is a creative adaptation of chain-based reasoning, typically used in language models, for vision tasks. The authors contribute a novel and interesting insight into the inherent logical structures of images. Previously, we thought images contained visual redundancy, which was proposed and widely adopted in Masked Autoencoders pretraining. Such insights propose a new idea for visual modeling, which improves vision models to handle complex, multi-step tasks.

2. Comprehensive Evaluation: The authors conducted thorough experiments across various models (e.g., LLaMA-300M, LLaMA-1B, and LLaMA-7B) and tasks (e.g., segmentation, pose estimation, object detection). This robust evaluation not only demonstrates the generalizability of the approach but also its effectiveness across different settings.

3. Interesting Saliency-Based Reasoning: The use of visual saliency to create intermediate reasoning steps adds a cognitive layer to how models process visual data, mirroring human visual attention mechanisms. This results in more informed predictions, especially in tasks where spatial and object importance matters.

4. Detailed Ablation Studies: The authors provide detailed ablation studies, which highlight the relative importance of different components of the proposed method (cognitive reasoning, query relevance, and annotation diversity). This adds clarity to which parts of the model drive the most significant improvements.

weaknesses:
1. Limited Exploration of Computational Complexity: While the method is novel, there is limited discussion on the computational overhead introduced by the saliency-based intermediate reasoning steps. Given that real-world applications demand efficient processing, an evaluation of the method’s scalability in terms of computational cost would have been valuable.

2. Generalization Across Different Vision Tasks: While CoF prompting is tested on several vision tasks, the experiments are focused on a few select tasks, mainly segmentation and pose estimation. Broader evaluation on a wider range of tasks, such as video understanding or 3D object recognition, would further validate the method’s versatility. Therefore, I strongly suggest the authors try some experiments on video understanding to further elaborate on the effectiveness and commonality of this method.

3. Handling Failure Cases: The authors acknowledge that VLMss can produce failures (such as pure black predictions), but the analysis of these failures remains surface-level. A deeper dive into understanding why these failures occur and how to mitigate them would strengthen the paper’s conclusions. This part could be integrated into the Conclusion and Limitation section, and it will also inspire follow-up research.

4. Dependence on Saliency Detection: The success of CoF prompting is tied closely to the quality of the saliency detection model used. The authors briefly mention the model (U2-Net), but the sensitivity of the approach to variations in saliency detection quality is not explored. If the saliency model performs poorly, it could compromise the entire prompting process.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730217982019

thread_type:
official_review

<thread_object>
creator: reviewer_gmfc

<content>
summary:
This paper extends Large Autoregressive Vision Models (LAVMs) with the capability of in-context learning for visual inputs by proposing a chain-of-focus (CoF) prompting approach. The CoF decomposes a visual prompt to intermediate reasoning steps by ranking salient regions of the prompt image, as well as taking into account the visual similarity and annotation richness to the test image. The saliency scores are obtained by an off-the-shelf saliency detector. The relevance of queries is modeled by the Jaccard similarity index of the two sets of visual codebooks in VQ-GAN. The annotation richness is modeled by the number of unique indices in their codebooks. The proposed method has been evaluated on 4 vision tasks, e.g., image segmentation, object detection, image inpainting, and pose estimation, which show the effectiveness of the CoF prompting, upon two backbone LAVMs.

strengths:
1) The proposed CoF prompting method is well motivated, which shall interest many ICLR attendants.
2) The proposed approaches, using the saliency score rank, the similarity of two sets of codebooks in VQ-GAN, are reasonable and easy to re-produce. 
3) The evaluation is fairly convincing to show the effectiveness of the proposed method, though not as significant as I expected.

Overall, this is a descent work which may deserve to share with the community in time.

weaknesses:
1) The reasoning steps of 0, 1, 2 are evaluated in Fig.4. More steps (just 2) do not show clear advantage, which is a bit disappointing to validate the proposed CoF prompting. I wonder if just using more diverse visual prompts that are also relevant with enough annotations, the performance may match or even out-perform using intermediate reasoning steps of the visual prompts?  

2) The comparison baseline is a random selection scheme, which may be too simple. The performance gain over this simple baseline is not that significant as I expected.

3) The specific approaches, like the off-the-shelf saliency score and the number of unique indices in their codebooks, are somewhat simple and intuitive.

questions:
Please discuss the pros and cons of using more diverse visual prompt images or more intermediate reasoning steps of one prompt image.

Any alternatives to the saliency scores to measure the reasoning steps? Some objects like faces, animals or intensive actions tend to capture visual attentions easily. 

Some relevant missing references, please discuss the differences:

Chain-of-Spot: Interactive Reasoning Improves Large Vision-language Models, 2024.
Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight, 2024.

What is the subtle difference between image relevancy and relevance? ll.140 => an image retrieval framework; ll.216, in Fig.2, => query relevance or relevance of the queries; => target informativeness or informativeness of task objectives? Btw: informative and informativeness are very vague terms, please define them first.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730523242605

thread_type:
official_review

<thread_object>
creator: reviewer_eyxr

<content>
summary:
This paper propose Chain-of-Focus(COF) Prompting, which enhances vision models by enabling step-by-step visual comprehension. CoF automates prompt design by selecting the most relevant and informative prompts from existing candidates and creates intermediate reasoning steps for prompt targets. Experiments on various downstream vision tasks demonstrate the effectiveness of the proposed method.

strengths:
This paper is well written. The methods are innovative, the experimental results are significant, and the visualizations are intuitive.

weaknesses:
1. The candidate prompt pool comprises 50,000 training images and annotations, which raises concerns in specific scenarios due to its inability to ensure data privacy. Additionally, it necessitates significant resources for the storage and retrieval of prompts.
2. Figure 3 does not include visualized results of the ground truth.
3. There is no comparison of the computational complexity and resource overhead of the different methods.

questions:
Can CoF address the issue of distribution shift, that is, when there is no data related to the test data in the prompt pool?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683007937

thread_type:
official_review

<thread_object>
creator: reviewer_ckd8

<content>
summary:
The paper introduces Chain-of-Focus (CoF) Prompting as a method to enhance large autoregressive vision models (LAVMs) for in-context learning (ICL) in computer vision. Drawing inspiration from Chain-of-Thought prompting in natural language processing, CoF Prompting enables vision models to perform step-by-step visual comprehension. It tackles the lack of logical structure in visual data by generating intermediate reasoning steps based on visual saliency. CoF Prompting also facilitates the creation of customized prompts by selecting the most contextually relevant ones, based on query similarity and target richness. The effectiveness of this method is supported by extensive experiments, demonstrating that it significantly improves the models' ability to make inferences on visual understanding tasks, leveraging the recent advancements in LAVMs that utilize purely visual inputs for ICL.

strengths:
- This paper introduces the interesting method of Chain-of-Focus Prompting, with strong and reasonable motivation. 

- The experiments are comprehensive.

weaknesses:
- Compared to previous work, such as "A Generalist Painter for In-Context Visual Learning," this paper only addresses two tasks: image segmentation and pose estimation. If this method does not work for other tasks, it could decrease the contribution of the paper.

- I am somewhat skeptical about why this method works. For instance, would the model perform equally well if it viewed the same number of images within reasoning steps (not randomly selected, such as viewing the same number of final step reasoning images)?

[1] Wang, Xinlong, et al. "Images speak in images: A generalist painter for in-context visual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

questions:
- Would the model perform equally well if it viewed the same number of images within reasoning steps (not randomly selected, such as viewing the same number of final step reasoning images)?

- If you simply use this logic for Chain-of-Focus: each step of reasoning progresses from images of a single object to two, and then to multiple objects, where do you think the disadvantages lie compared to your method?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces chain-of-focus prompting, a new approach that enhances vision models through step-by-step visual comprehension and contextually informative prompts, enabling better inferences in visual in-context learning tasks. The proposed method has strong and reasonable motivation. The experiments are comprehensive.

The reviewers raised concerns about data privacy, resource overhead, limited exploration of computational complexity, and the narrow focus on segmentation and pose estimation, suggesting broader task evaluation and deeper analysis of failure cases. The authors provided detailed responses. Three reviewers of 4 have expressed the satisfication of the responses. Reviewer CkD8 did not further response. I think the response has well addressed the concern of CkD8.

Given the overall score of 6.75 and concerns beening addressed, AC suggest to accept this paper.