Chain-of-Focus Prompting: Leveraging Sequential Visual Cues to Prompt Large Autoregressive Vision Models
Authors: Jiyang Zheng
Jialiang Shen
Yu Yao
Min Wang
Yang Yang
Dadong Wang
Tongliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Visual In-context Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In-context learning (ICL) has revolutionized natural language processing by enabling models to adapt to diverse tasks with only a few illustrative examples. However, the exploration of ICL within the field of computer vision remains limited. Inspired by Chain-of-Thought (CoT) prompting in the language domain, we propose Chain-of-Focus (CoF) Prompting, which enhances vision models by enabling step-by-step visual comprehension. CoF Prompting addresses the challenges of absent logical structure in visual data by generating intermediate reasoning steps through visual saliency. Moreover, it provides a solution for creating tailored prompts from visual inputs by selecting contextually informative prompts based on query similarity and target richness. The significance of CoF prompting is demonstrated by the recent introduction of Large Autoregressive Vision Models (LAVMs), which predict downstream targets via in-context learning with pure visual inputs. By integrating intermediate reasoning steps into visual prompts and effectively selecting the informative ones, the LAVMs are capable of generating significantly better inferences. Extensive experiments on downstream visual understanding tasks validate the effectiveness of our proposed method for visual in-context learning.

Published as a conference paper at ICLR 2025
CHAIN-OF-FOCUS PROMPTING: LEVERAGING SEQUEN- TIAL VISUAL CUES TO PROMPT LARGE AUTOREGRES- SIVE VISION MODELS
Jialiang Shen', Yu Yao', Min Wang,, Yang Yang', Dadong Wang', Tongliang Liu" 'Sydney AI Center, The University of Sydney PCSIRO, Data61 'Shanghai Jiao Tong University Sjzhe574C, jshe [dadong.
Jiyang Zheng Jialiang
wydney.dddyyyyy... au csiro......................or... au tongliang 1 waw.ang...eesuleeeee seleeeeddyy.d.... edu.au
ABSTRACT
In-context learning (ICL) has revolutionized natural language processing by en- abling models adapt diverse tasks with only a few illustrative examples. However, the exploration of ICL within the field of computer vision remains lim- ited. Inspired by Shain-offThhoueee (CoT) prompting the language domain, we propose Chain-of-Focus (CoF) Prompting, which enhances vision models by en- abling step-by-step visual comprehension. CoF Prompting addresses the challenges of absent logical structure in visual data by generating intermediate reasoning steps through visual saliency. Moreover, it provides a solution for creating tailored prompts from visual inputs by selecting contextually informative prompts based on query similarity and target richness. The significance of CoF prompting is demon- strated by the recent introduction of Large Autoregressive Vision Models (LAVMs), which predict downstream targets via in-context learning with pure visual input. By integrating intermediate reasoning steps into visual prompts and effectively selecting the informative ones, the LAVMs are capable of generating significantly better inferences. Extensive experiments on downstream visual understanding tasks validate the effectiveness of our proposed method for visual in-context learning.'
INTRODUCTION
Utilizing pre-trained, general-purpose purpose vision model to perform multiple downstream visual tasks with only a few illustrative examples represents a significant advancement toward artificial general intelligence. Recently, the emergence of Large Autoregressive Vision Models (LAVMs) (Bai et al., 2024; Guo et al., 2024) has presented a promising approach for achieving this unification of tasks. The principle behind this integration involves building an autoregressive model (Touvron et al., 2023a) that enables visual in-context learning (Bar et al., 2022; Zhang et al., 2023b; Wang et al., 2023a; Li et al., 2024), where given a test input and a pair of prompts containing an input image and its visualized target annotation, the vision models endeavor to recognize the visual patterns between the prompt image and its target, thereby making analogous predictions on the test image.
In the realm of large language models (LLMs), in-context learning (ICL) has been extensively studied (Dong et al., 2022). Among these approaches, Chain-of-Thoupg (Co')) prompting (Wei et al., 2022; Wang al., 2022; Zhang et al., 2022b) is of the most influential methods, significantly enhancing the predictive abilities of LLMs by introducing intermediate reasoning steps within the contextual language prompts. Given that LLMS and LAVMs share similar autoregressive architectures. we are inspired to explore whether injecting intermediate steps into visual contextual prompts can similarly unlock the capabilities of LAVMS. Building upon the principles of CoT prompting, we propose Chain-of-Focus (CoF) prompting, novel prompting method tailored for LAVMs.
"Corresponding Author
hhttps://g thub. com/ tm1 1 1 1ab/ 20/5025_ 2022__CII_____00 CCLR_COF

---

Published as a conference paper at ICLR 2025
Standard Visual Prometing
Model Iiput
Large Autonogresshre
Model
Querg
Tirger
Predlctiie
of (CoF) Prompting
Large Autoregraatie
Modal Input
Pronge Owery
Intemederrs Reannting Stipe
Proege Torpet
trpuzt
LAVM
Illustration of Chain-of-Foouu (CoF) prompting. The top section illustrates the current strategy for prompting LAVMs, where the prompt query (image) is randomly selected for the test input, and the task-specific prompt targets are visualized to form a prompt pair, enabling LAVMs to make in-context, analogy-based predictions. CoF prompting (bottom section) generates intermediate steps leading to the prompt target while selecting informative prompt pairs based on prompt query similarities to the test input and the richness of usable information contained in the prompt target.
Figure I:
Nevertheless, implementing contextual and sequential prompts in the vision domain presents two significant challenges. First, unlike text, which follows syntactic and semantic rules, visual data inherently lacks the clear logical structure, making it difficult to decompose and sequence for step-by- step interpretation. Second, in the language domain, hand-crafted prompts can be tailored specifically to the test input by providing analogous examples that closely relate to the problem at hand. For instance, if the test input for LLMs is geometry problem, the language prompt can include similar geometry problem with its solution, making the answer more informative to the model for analogy-based predictions. This level of customization is challenging in the visual domain, images cannot be easily modified or restructured to fit new test inputs.
CoF prompting addresses the first challenge by adapting a cognitive strategy that is fundamental to human visual understanding: visual salience, which enables individuals to sequentially process visual information and draw intermediate conclusions based on the prominence of salient objects in a scene (Wertheimer & Riezler, 1944). For example, when viewing an image of a kitchen containing numerous objects, an observer's attention will initially focus on larger and closer items, such as the benchtop and chairs placed in front of it, before shifting to smaller appliances. As illustrated in Figure 1, CoF prompting replicates this cognitive process through generating intermediate reasoning steps within the prompt targets by ranking the salient regions of the prompt image in descending order. Specifically, we generate a saliency probability map using a pre-trained saliency detection model (Qin al., 2020) obtain the order of salient regions in the prompt image. Incrementally annotating different parts of the image based on saliency scores to create intermediate steps, allowing the models build context progressively and enhance their predictive capabilities.
On the other hand, in the language domain, it has been shown in CoT prompting that finding in- formative prompt queries is crucial for enhancing LLM's predictive accuracy. Inspired by this, in CoF prompting for visual inputs, we utilize two selection criteria to search for the most informa- tive prompts relative to the test input. First, we consider image relevance, which measures how semantically related the prompt image is to the test input image. Prior research (Zhang et al., 2023b) has demonstrated that images sharing similar semantic meanings with the test input serve as better illustrations, enabling the model to draw more accurate analogies. However, we find that for certain downstream tasks, these semantically similar images may have sparse annotations, meaning they cannot provide sufficient knowledge to the model. Therefore, we introduce the second criterion, anno- tation richness, to ensure that the selected prompt images contain comprehensive annotations useful for the test case. By integrating both image relevance and annotation richness, our approach addresses the challenge of creating tailored visual prompts, enhancing the model's 's ability to generalize from a few examples to unseen inputs.
2

---

Published as a conference paper at ICLR 2025
We build our method upon the framework of Large Autoregressive Vision Models (LAVMs) (Bai et al., 2024; Hao et al., 2024), leveraging their ability to perform simultancous predictions across multiple downstream tasks within one single pre-trained model. To quantify the similarity between the prompt image and the test image, we employ the encoder from the pre-trained LAVMs and evaluate the distance between their encoded representations. This encoder transforms raw images into discrete indices within a codebook via vector quantization(Esser et al., 2021; Van Den Oord al., 2017). By treating these codebooks as sets and calculating the intersection over union between them, we effectively capture semantic equivalence while disregarding the specific order of indices. After identifying prompts similar to the test input, we assess the richness of prompt annotations by examining the diversity of entries in the prompt targets' codebooks. This approach ensures that the selected visual prompts are not only highly relevant but also possess rich annotations, thereby enhancing the in-context performance of the LAVMs.
To summarize our contributions, we propose a new visual prompting paradigm called Chain-of- ooFuu (CoF) prompting. Our approach mimics progressive thinking by incorporating intermediate steps into visual prompts and addresses the challenge of prompt customization by directly selecting the most informative prompts relative to test inputs. Our method can be seamlessly integrated with the recently proposed Large Autoregressive Vision Models (LAVMs) (Bai et al., 2024; Hao et al., 2024) through visual in-context learning, significantly improving their performance on downstream visual tasks.
2 RELATED WORKS
In-Context Learning and CoT Prompting In-context learning (ICL) (Huang et al., 2024a; Wang al., 2024) is a paradigm where models learn to perform tasks by conditioning on examples provided in the input context during inference. Rather than relying on traditional training processes with gradient updates, the models leverage the contextual information from query-target pairs presented at inference time to make predictions on new test inputs. In the language domain, recent advancements have highlighted the effectiveness of hierarchical reasoning techniques, known as Chain-off -of oh-Thtthh prompting, in enhancing the performance of large language models (LLMs) (Kojima et al., 2022: Wang et al., 2022; Wei et al., 2022; Zhang et al., 2023a; Luo et al., 2024; Zhang et al., 2024a; Lin al., 2025; Tu et al., 2024). These methods leverage sequential reasoning steps to improve inference. Inspired by these developments, researchers have extended hicrarchical reasoning frameworks to the vision-language domain (Lu al., 2022; Zhang et al., 2023c). Among these, the most related stream of works to ours has attempted to explore the rationale within or across images and express them in textual descriptions (Ge et al., 2023; Mitra et al., 2023; Rose et al., 2023; Zheng al., 2023). This integration of visual information into its language counterpart has yielded significant improvements for large -aiguael models (LVLMs) (Liu et al., 2023; Zhang et al., 2022a; 2024b; Zhou et al., 2024), yet it also reveals the challenges of applying CoT-based methods directly to the pure vision domain (i.e., expressing reasoning without the use of language). Unlike language, images lack explicit symbolic structures, making it challenging to express reasoning steps as in LLMS or LVL.Ms. In purely visual contexts, Zhang et al. (Zhang et al., 2023b) develop a prompt retrieval framework for selecting in-context examples that maximize models' performance. Chain-of-Spot (Liu et al., 2024) develops a multimodal promtping method for LVLMs. It leverages language prompts to use only regions of interest (ROIs) for visual understanding. Chain-of-Sight (Huang et al., 2024b) introduces purely visual framework that employs a sequence of visual resamplers to capture visual details at different spatial levels, generating tokens across multiple scales.
Large Autoregressive Vision Models The inspiration behind autoregressive vision models stems from the advancements of large language models (LLMs) (Brown et al.. 2020; Touvron et al., 2023a;b). Using contextual information, LLMs are able to capture long-range dependencies and make coherent predictions with sequential modelling techniques. Building on this concept, Bai et al. (Bai et al., 2024) propose Large Autoregressive Vision Models (LAVMss, which adapt this modelling strategy to the visual domain by constructing "visual sentences" that enable sequential prediction. This approach involves representing visual inputs as sequences of tokens, analogous to the text tokens used in LLMs. By processing visual data sequentially, the model employs self-attention mechanisms to understand dependencies and relationships within the visual context, thereby enabling effective in- context learning from purely visual inputs. By including query-target pairs from different downstream tasks in these visual sentences, the model can accomplish various visual downstream tasks within
3

---

Published as a conference paper at ICLR 2025
single framework. Hao et al. (Hao et al., 2024) extend the work and introduce data-efficient LAVM, which is designed to operate effectively on limited datasets by making use of data augmentation and knowledge distillation. The primary purpose of LAVMs is to unify all vision tasks within a single model, making the adaptation to downstream tasks highly efficient.
METHODS
3.1 PRELIMINARIES
The Large Autoregressive Vision Model (LAVM) (Bai al., 2024) is foundational vision model that synthesizes visual predictions through sequential modeling, inspired by the successes of Large Language Models (LLMs). In LLMs, autoregressive model predicts the next word in sentence based on previous words. Similarly, LAVM aims to predict the next visual token in a visual sequence given the previous tokens. This is achieved using tokenization network E: hxxxxx + Ru that transforms raw images X = {x1. 42,,.. Ix E R Rhxwxe into visual tokens Z {21, ..,..., 2n E Raxd, followed by a sequential model f Rº + that predicts outputs in an autoregressive manner 2 f 21-2,... ................................ 2t=p) + Et, where p is the total number of previous time steps in the sequence, t the current step, and is the noise. The predictions are then detokenized back to pixel space by a decoder network D Raxd - Rxxwxc
In implementations of LAVMs (Bai et al., 2024; Hao al., 2024), a pre-trained VQ-GAN (Esser et al., 2021) model is employed as the tokenizer. The VQ-GAN model encodes the image into discrete codebook, with the indices in the codebook serving as the tokens for the autoregressive model. The pre-trained VQ-GAN decoder then decodes the codebook/tokens back into pixel space for generating images. At its core, the autoregressive model in LAVM utilizes a causal transformer (Touvron et al., 2023a) which employs causal masking to compute each token's 's representation based solely itself and the preceding tokens, thereby preserving the sequence's ' temporal order. This allows the model to capture dependencies and patterns within the data effectively, enhancing its ability to generate coherent sequences during inference.
The visual sentences used to train LAVM are either derived from natural visual sequences, such as videos or multi-views of a 3D object (Zhan et al., 2022), or handcrafted by connecting raw images with their target annotation pairs from various visual downstream tasks. This allows the model to adapt to any downstream task given images (a.k.a. prompt queries xpg)) and annotations (a.k.a. prompt targets p. the inference stage, LAVM employs prompted inference. Given several examples of image and target annotation pairs, the tokenizer first transforms each input into tokens and constructs a visual sentence using the paired image and annotation data. The test input is appended at the end of the visual sentence the last token. This sentence is then passed into the autoregressive network for the prediction the next token in the sequence. The predicted tokens are subsequently constructed into a codebook and decoded into pixel space.
3.2 SALIENCY-BASED INTERMEDIATE REASONING STEPS
Sequential Prompt Construction In our approach, we construct prompts that not only present visual queries and targets but also sequentially introduce reasoning steps. Each prompt consists of a visual query Tpg. and a series of m intermediate reasoning steps {arpe 2 #pt..... ... x leading up to the final target Tpt- This setup mimics human reasoning processes, where intermediate conclusions are drawn before reaching a final decision. In practice, when constructing the model input with the intermediate steps, we find that the best order is denoted as: [Zpq- Z Zxg: 2 ... Pgg zpt Ptgl- where the prompt query and intermediate targets are ordered alternately, with the test query Beg appended at the end of the sequence. We suggest that the optimal construction order depends on the pre-trained model itself. The pre-trained LAVMS (Hao et al., 2024; Bai et al., 2024) are primarily trained on visual sentences in the format of query and target pairs, thus its sequential prediction ability is restricted to paired representations. Similarly, finetuning the pre-trained LAVMs with natural reasoning steps allows for different construction of prompts: [xpg 2/e0 z Pptt.... 2nth N'gl, which follows the natural sequential order. These intermediate reasoning steps decompose the complex answers into sub-pieces for understanding.
4

---

Published as a conference paper at ICLR 2025
Queries Relevance
Targets Informativeness
Prompt
Selccted Prompe Query
VOGAN Encoder
VOGAN Encoder
E
Tost Input
Sallency Model
Emcoder of
Jacoard Sinfurity
Sborilivted
Cornoeandding Torgets
Encedar of
Indier
S
Input
Romoning Stepe
Saftenc Mop
P.Quury
Reasenting Steps
Prompting Large Autoregressive Vision Models (LAVMS)
Sequential Visual Steps
Figure 2: Illustration of Generating CoF Prompts. The framework can be viewed in two steps. First, CoF identifies a of the most relevant queries the test input and assesses the informativeness of their targets to filter out less informative prompt pairs. This step ensures that the prompts are highly relevant and informative to the test input. In the second step, CoF uses saliency-based strategy to create intermediate steps for the answers to the query, which implicitly injects sequential visual cues into the prompt targets. CoF follows the general structure of Chain-of- prompting, with improvements in automating the process of both prompt selection and intermediate steps generation.
Visual Reasoning via Exploring Salient Regions To simulate cognitive reasoning process, we generate sequence of intermediate answers using visual saliency information. Given a visual query Zpy and its corresponding answer Tpt. where both Tpg and zpt are images. We utilize the salient regions within these images for constructing informative prompts. To quantitatively assess the salience of different regions within the images, we compute saliency score (()) for each region where the regions are defined by the masks objects of interest in the image. In tasks such as image segmentation and pose estimation, the auxiliary information on masks are often provided with the ground truth as their segmentation masks and bounding boxes. We use pre-trained saliency detection model (Qin et al., 2020) to obtain a saliency probability map for the image. For each region, compute the saliency score as:
(r) = Σ - (((p))))(pp)))))))))
M,(i,)) the mask for the region r. where M, (i.j) = if the pixel (i,j) ) within the region r and M,)i,j 0 for pixels that do not belong to the region. The function S hxx + g extracts the pixel-wise saliency probability scores from the prompt query Tpg and forms the probability map. The function (r) computes the summed probability for the masked area the region saliency score. We label the regions an incremental manner using the saliency scores. For each intermediate step, the target is given by:
= = =x; U (r (r)) > 741).
where 7; saliency threshold for step defining the minimum saliency required for a region to be included in the intermediate target z², In the last step, all regions in the image will be labelled, providing a complete prompt target. This ordered introduction of information helps the LAVM to focus on relevant features at each step, allowing it to build context progressively. By leveraging saliency-based cognitive pathway, we aim to mimic the hierarchy focusing observed in human visual attention, enhancing the understanding of visual content through structured, human-like reasoning.
3.3 INFORMATIVE VISUAL PROMPTS
In chain-of- -f-thughtt (CoT) prompting, selecting relevant queries is crucial as it directly impacts the quality of the generated responses. Traditionally, CoT involves manually choosing prompt queries
5

---

Published as a conference paper at ICLR 2025
for each test imput, process that ensures alignment with desired outcomes but is prone to human bias. In our method, we aim to automate this process by selecting the most relevant and informative visual query and target pairs to the test input, thereby enhancing the in-context learning performance of LAVMs. The following details our strategy for selecting visual query and target pairs to serve the prompts for inference.
and
Selection of Relevant Queries Given a test query Ttg and candidate pool of prompt pairs consisting of prompt queries Tpg E X and prompt targets dipt E Xpt, our goal to first shortlist subset of prompts contain queries that is similar to the test query. To this end, employ the same VQGAN encoder from the LAVM framework to serve as the feature extractor for the prompt queries and the test query. The encoder transforms the queries into discrete codebooks (zgg 2pg , Spgz...... Zpg..) }. Each entry in the codebook is a discrete generative factor that corresponds to the pixel space, therefore, more aligned entries in the two codebooks of queries indicate that the two queries contain similar objects or scenes in the pixel space. Through manual testing, we find that the relative position of the objects and the number of the objects in the prompt query do not affect the performance of inference as long as the two queries are semantically aligned. Hence, we convert the codebooks into sets and measure the similarity of each encoded prompt query Zpy and 21g using the Jaccard similarity index, which is defined as:
12gg O 2pg
J [Zig Zpq)
(3)
Ezgg U Zpg
This measure counts the number of unique indices shared between Σty and Zoq without considering the position of the indices in the codebook. The set operation also helps avoid overrrepresentation of redundant and repeating background features that are not pertinent to the task. Through this process, we shortlist a subset of N prompts that have queries similar to the given test query.
Selection of Rich Targets Once have selected the N most similar queries, need to further refine our selection for target informativeness, that is to ensure the chosen answers are providing rich information for inference. As observed in tasks such as image segmentation and keypoint detection, the presence of diverse and richly annotated segmentation masks is crucial for effective in-context learning. We quantify the informativeness of prompt target Zye by assessing the diversity of its encoded discrete representation Ept. The intuition behind this involves ranking the prompt targets based on feature richness, where prompt targets with less information tend to have fewer variations in their features. For given prompt from the shortlisted subset, we calculate the number of unique indices in its encoded targets zpt- Formally, we maximize the function:
& Dz(2p)) = arg max /zpt|: Ape
(4)
where 12pel denotes the number of unique indices in Ipe codebook, and the Zpt are from the shortlisted subset. We select the top prompts with the highest number of unique indices in their target codebooks, which ensures that the selected examples contain diverse annotations with varying meanings and structures. The final selection comprises the prompts with the most relevant queries and informative targets, which serve as our baseline prompts for the following visual reasoning step.
4 EXPERIMENTS
In this section, conduct evaluation on CoF prompting for LAVMs. In Section 4.1, we introduce experiment settings, including dataset, pre-trained models, metrics, and other details. In Section 4.2, we report our main results on downstream visual tasks and present extensive quantitative and qualitative analyses. In Section 4.3, we conduct ablation experiments on the three major components in the CoF framework to study the contributions of each module and provide discussions. Due to page limitations, we have included additional results and analyses in the Appendix.
4.1 EXPERIMENTAL SETUP
Tasks and Dataset For our experiments, we select four downstream visual tasks: image segmenta- tion (Hong al., 2024), object detection (Zheng al., 2022), image inpainting and pose estimation.

---

Published as a conference paper at ICLR 2025
Image Segmenation
Method/ / Model
LLaMA-300M (Hao et al,, 2024)
LLaMA-IB (Hao et al., 2024)
LLaMA-7B (Bai et al., 2024)
ToU (8/)
P-ACC ($1)
JoU (%1)
P-A)C(
(9/1
P-ACC($91)
Random Selection
26.31 # 0.8
42.96 = L.
27.21 + 0.4
41.88 = 1.0
45.69 + 1.4
59.06 # 2.2
SegGPT (Wang et al., 2023b)
26.52 + 1.4
42.54 # 2.7
26.39 +: 1.2
42.71 = 1.6
45.38 0.8
60.72 :: 1.9
SupPR (Zhang al,, 202.bb
27.05 +: 1.1
43.52 + 1.4
27.94 + 0.9
42.16 1.2
49.41
+ 1.7
65.04 + 1.1
CoF Prompting (Ours)
28,35+ 0.6
46,36 # 0.8
28.79 + 0.3
44.75 1.0
52.53 + 0.3
67,05 + 0.7
Table 1:
Segmentation results of CoF prompting on LLaMA-300M, LLaMA-IB IB and LLaMA-7B.
Pose Estimation
Method Model
LLaMA-300M (Hao al., 2024)
LLaMA-IB (Hao et al,, 2024)
LLaMA-7B (Bai et al., 2024)
IoU (%1)
P-ACC (8+)
IoU (%t)
P-ACC (81)
(%6)
P-ACC (561)
Random Selection
0.60 4: 0.07
1.44 0.09
1.00 1.00::0.05
2.96 4: 0.10
2.40 + 0.07
10.23 4: 0.16
SupPR (Zhang et al,, 2023h)
0.67 + 0.04
1.65 0.13
1.04 + 0.02
2,93 + 0.18
2.87 + 0.22
11.29 + 0.21
CoF Prompting (Ours)
0.68 + 0,04
1.75 + 0.05
1.09 0.02
3,29 0.07
2,80 = 0.04
13.34 = 0.13
Table 2: Pose Estimation Results of CoF Prompting on LLaMA-300M, LLaMA-IB and LLaMA-7B.
Image segmentation involves partitioning an image into multiple segments or regions. The primary objective of this task is label each pixel in the image with class label, identifying the object to which it belongs. Pose estimation refers to the task of determining the configuration of the body in given image by predicting the locations of keypoints or joints. The goal here to detect and classify the keypoints representing the positions of body parts. To facilitate these tasks. we employ the MS-COCO dataset (Lin et al., 2014), adhering to the settings outlined in Bai et al. (2024); Guo et al. (2024). Our experimental protocol involves extracting 50,000 training images and their corresponding target annotations to form the candidate prompt pool, and we rigorously test our methods on the entire validation dataset. Note that, the pre-trained LLaMA-3000 and LLaMA-ID IB only support the image segmentation and pose estimation tasks, while LLaMA-7B supports all four downstream tasks.
Pre-trained Models We utilize pre-trained LAVMs from (Bai et al., 2024) and (Hao al., 2024) for in- in-context learning. Specifically, we employ the VQ-GAN model as proposed by (Chang et al., 2023) to generate discrete visual representations of 2048 dimensions. For the autoregressive network, we leverage pre-trained LLaMA models (Touvron et al., 2023a;b) at different scales, including LLaMA-300M, LLaMA-1B, and LLLMA-7BB for sequence modeling. Additionally, we incorporate an off-the-shelf saliency detection model from U2-Net (Qin et al., 2020), which takes RGB images as input and outputs a saliency probability map of the same height and width as the input image.
Visual ICL Baselines We compare our method with existing visual in-context learning approaches, specifically SupPR (Zhang et al.. 2023b) and SegGPT (Wang et al., 2023b). SupPR is a general prompt retrieval framework that extracts prompt pairs that contain images similar to the test input. SegGPT is a prompting method designed for segmentation tasks. We only adopt its central idea of using the same color mask for the same object class when prompting for segmentation tasks.
Post- post-processing and Evaluation Metrics Following (Guo al., 2024; Zhang al., 2023b), We utilize Intersection over Union (IoU) and Pixel accuracy (P-ACC) as our evaluation metrics for segmentation and pose estimation. We convert the predicted outputs into binary pixel masks and compare them against the binary ground truth masks. loU measures the overlap between the predicted and ground truth regions by dividing the area of intersection by the area of union. The P-ACC calculates the proportion of correctly classified foreground pixels in the binary prediction mask compared to the ground truth. For detection, since the model only outputs the visualised bounding box as in image, we cannot directly obtain the coordinates for evaluation. To address this, we employ a post-process network that intakes images with visualised bounding box and outputs the box coordinates. We then calculate the IoU of the bounding boxes to the ground truth. We name the metric as Learned IoU (L-IoU). For image inpainting, we report the MSE loss and LPIPS score. We also measure the failure cases of LAVMs in making predictions, that is when the LAVMs fail to output any meaningful prediction, where the output appears in pure black. Due to page limit, we put the analysis of object detection and image inpainting in the Appendix section A.
7

---

Published as a conference paper at ICLR 2025
Figure 3: Results LLaMA-7B Model. The first and fourth the original test inputs for image segmentation, detection, inpainting and pose estimation, respectively. Orange boxes show the predictions given random prompts. Maroon boxes show the predictions using SupPR method (Zhang et al., 2023b). Blue boxes show the predictions using Miain-of-Focuu prompting.
4.2 RESULTS
Image Segmentation Table reports the quantitative performance of CoF compared to random prompting, same colour masking (Wang al. 2023b) and SupPR (Zhang et al., 2023b). The CoF method demonstrates notable percent- age increases compared the second best per- forming methods across various metrics. For image segmentation with LLaMA-300M, the increases are approximately 4.81% in IoU and 4.77% in P-ACC, while for LLaMA-ID 1B and 7B, the increment in proportion is 3.04% and 6.31% % in IoU, and 6. 1.14% and 3. 10% P-ACC, respectively. The results are reported with predictions that have black rate > 0.2. We report the failure cases for segmentation in Table 3. Compared to the baseline, using CoF eliminates the failure cases caused by the incapability of two LAVMs by 1.64%, 13.9% and 7.7%, respectively. Figure 3 demonstrate the predictions made by Random, SupPR and CoF
Model
Random
CoF
1LaMA-300M w/ VQ-GAN
58.58 +: 1.8
57.62 +: 0.6
LLaMA-IB w/ VQ-GAN
50.08 + 2.5
43.12 + 1.9
LLaMA-7B w/ VQ-GAN
45.28 +
42.03
Table 3: Failure Rates (L) Image segmentation
8

---

Published as a conference paper at ICLR 2025
Image Segmentation
Pose Estimation
Model
CR
QR
AD
loU (%f)
P-ACC (%+1)
IoU (%+)
P-ACC (%+)
27.92
46.17
0,65
1.63
26.32
42.99
0.59
1.41
26.
41.92
0.61
1.44
LLAMA-300M w/ VQ-GAN
I
28.13
45.32
0,68
1.77
26.95
41.72
0,63
1.55
28.21
46.10
0.65
1.71
28.63
45.07
1.04
2.87
26.14
43.05
0.99
2.73
27.39
43.02
1.01
2.84
LLaMA-IB VQ-GAN
27.90
44.16
1.09
3.30
27.33
42.19
1.07
3.01
28.50
44.35
1.12
3.22
51.74
67.01
2.91
13.46
50.98
65.07
2,66
11.62
47.13
61.26
2,41
10.26
LL w/ VQ-GAN
52.01
65.83
2.88
12.89
50.34
64.00
2.67
11.62
51.97
66.41
2.64
12.55
Table 5: Ablation Study on the three major components involved in CoF pipeline. CR represents Cognitive Reasoning, which creates intermediate reasoning steps for the prompt target. QR represents Query relevance, which measures the similarity between the prompt queries and the test input. AD is Annotation Diversity, which involves accessing the diversity of indices within the targets' codebooks.
prompting with LLaMA-7B model. We observe improvement in the objects that models successfully identified and the accuracy of masking. The models using CoF prompting also demonstrate better scene understanding ability, which outputs complete masks for the same objects. These suggest that the in-context object discovery and segmentation ability of LAVMs can be enhanced by prompting them with our method.
Pose Estimation A similar trend is also found the pose estimation task, where, as illustrated in Table 2, CoF prompting outperforms the other methods by a noticeable margin. For pose esti- mation using LLaMA-300M, compared to the second highest scores, the increases are approx- imately 1,49% in IoU and 6.06% in P-ACC. Moreover, I.LMMA-IB shows larger improvement, with an increase of 4.81% % in loU and 11.15% in P-ACC. For LLaMA-7B, the P-ACC is increased by 18%. but the IoU is 2.5% lower then the second highest method. The failure rates are reported in Table 4. Despite pose estimation being challenging task for LAVMs, CoF prompting reduces the failure rate on both LLaMA-3000
Model
Random
CoF
LLaMA-300M w/ VQ-GAN
$3.49 + 3.7
$2.38
LLaMA-IB w/ VQ-GAN
43.67 1 2.0
42.54
1 0.9
LLaMA-7B w/ VQ-GAN
41.74 + 1.5
35.62 + 1.9
Table 4: Failure Rates (%.. Pose Estimation
LLaMA-1 1B and LLaMA-7B by 2.08% and 2.59%, 14.7% respectively. We qualitatively compare the results of the LLaMA-7B model in the top three rows Figure 3. CoF prompting demonstrates better performance compared to the baselines, with improvements in the completeness of the skeletons, the accuracy of pose detection, and number of human targets that the models successfully identify the given test input, demonstrating the effectiveness of method. More results provided in the appendix.
4.3 ABLATION STUDIES
Intermediate Step and Prompt Selections understand the impact of various components in our CoF prompting method, we conduct series of ablation studies. Specifically, the designed experi- ments isolate and evaluate the contribution individual components by systematically removing or modifying specific parts of the model and observing the resulting performance changes. Through this analysis, we seek to identify the critical factors that drive the success of our method and provide insights into potential areas for further improvement. We divide our entire framework into three parts: cognitive reasoning (CR), query relevance (QR), and annotation diversity (AD). Cognitive reasoning involves generating intermediate reasoning steps using object saliency. When removing

---

Published as a conference paper at ICLR 2025
CR, we directly prompt the LAVMs using the query and its complete target. Query relevance involves selecting prompts by measuring their relevance to the test input. When removing QR, we randomly sample the candidate set of prompts. Annotation diversity involves evaluating the prompt target. When it is removed, the CoF does not access the prompt target for prompt selection.
We present the results of our ablation experiments in Table 5. In the image segmentation task, for all models, we observe that the primary performance improvement originates from cognitive reasoning, which incorporates intermediate steps for the prompt targets. The standalone performance of the prompt retrieval component does not significantly benefit the LAVMs, as evidenced by the predictive performance, which is comparable to the random selection baselines. However, the integration of prompt selection with cognitive reasoning shows a marked improvement, with both CR QR and CR + AD combinations achieving better results than cognitive reasoning alone. A similar trend is observed in pose estimation, where cognitive reasoning remains the most crucial component, demonstrating a significant enhancement when applied. Notably, in pose estimation, prompt selection can also achieve good performance independently, without the aid of cognitive reasoning. This provides insight into the contribution of each component within the framework, highlighting cognitive reasoning the most critical strategy, with the two steps involved in prompt selection seamlessly enhancing the efficacy of the reasoning strategy.
Number of Reasoning Steps Here exam the influence of different number of rea- soning steps for CoF prompting. Our setting includes using [0, 1,2] intermediate steps in between the prompt queries and the prompt target. Due to the maximum input length to the autoregressive model employed in (Hao et al., 2024), injects two intermedi- ate steps before the final targets is the maximum for in-context learning using the model. We use the same prompt queries and origi- nal target for all three experiments to avoid influence from the prompt selection. Fig- ure 4 demonstrates our results, where both models show improved performance with an increasing number of reasoning steps, in- dicating that more reasoning steps enhance their capabilities. However, in the case of the 300M model, the scores decrease when increasing from one intermediate step to two intermediate steps in image segmenta- tion. Conversely, the LLaMA-1B IB model exhibits a more stable linear increment com- pared to LI MM-3300M, demonstrating that the larger model benefits more significantly from reasoning steps. These results high- light the importance of CoF prompting in achieving better performance.
LLaMe w/ VQ- GAN
UaMM-1B w/ VO-GAN
025
078
120
180
115
12
26
Figure 4: Comparison of using different reasoning steps. The first row of figures captures the perfor- mance measures of the image segmentation task, and the second row captures the performance measures of the pose estimation task.
CONCLUSION
The paper introduces Chain-of-Focus (CoF) prompting, a novel method designed to replicate the sequential steps of Chain-of-Thoughh prompting in the visual domain by bridging the gap between symbolic reasoning in language models and perceptual reasoning in vision models. CoFl automates prompt design by selecting the most relevant and informative prompts from existing candidates and addresses the inherent challenge of the lack of explicit symbolic structure in images by utilizing visual saliency to create intermediate reasoning steps for prompt targets, capturing the intrinsic logic of the human perceptual system. By leveraging this hierarchical information, COF allows Large Autoregressive Vision Models (LAVMs) to process and understand visual information progressively, thus enhancing their sequential predictive performance experiments on image segmentation and pose estimation using LLaMA- 1B and 7B w/ VQ- GAN models demonstrate that embedding visual reasoning into prompts significantly improves the model's inference capabilities. CoF prompting represents a significant adyancement in visual in-context learning, with potential for broader applications in machine learning and computer vision.
various downstream vision tasks. Our
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
The authors would like to thank Muyang Li for his valuable feedback throughout the project. Jiyang Zheng supported by the CSIRO Next Generation Graduates and AI for Missions PhD program. Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, IC190100031.
REFERENCES
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances Neural Information Processing Systems, 35:25005 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners Advances in neural information processing systems, 33:1877-19011 2020. 2020.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey in-context learning. arXiv preprint arXiv:2301.00234 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Jiaxin Ge, Hongyin Luo. Siyuan Qian, Yulu Gan, Jie Fu, and Shanghang Zhang, Chain of thought prompt tuning in vision language models. arXiv preprint arXiv:2304.07919, 2023.
Jianyuan Guo, Zhiwei Hao, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, and Chang Xu. Data-efficient large vision models through sequential autoregression. arXiv preprint arXiv:2402.04881, 2024.
Zhiwei Hao, Jianyuan Guo, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, and Chang Xu. Data-efficient large vision models through sequential autoregression. In Forty-first International Conference on Machine Learning, 2024. URL https: /ttp://///eeeeeeeeeeeeee net/
Jie Hong, Weihao Li, Junlin Han, Jiyang Zheng, Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Goss: Towards generalized open-set semantic segmentation. The Visual Computer, 40(4):2391- 2404, 2024.
Zhuo Huang. Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Hamessing outtof-distribution examples via augmenting content and style. In The Eleventh International Conference on Learning Representations, 2023, URL https: /ttp:///opeeeeeeeeeeeee net Eorum? (arammmmdddaa 1dwboooggo-00DD00
Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. In Forty-first International Conference on Machine Learning, 2024a. URL https:/ wpwnreveeeeeeeeeeeeeeeeeeeeeeeee net forum? f
Ziyuan Huang, Kaixiang Ji, Biao Gong, Zhiwu Qing, Qing-Long Zhang, Kecheng Zheng, Jian Wang, Jingdong Chen, and Ming Yang. Accelerating pre- training of multimodal llms via ofain-of-sight. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
11

---

Published as a conference paper at ICLR 2025
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Feng Li, Qing Jiang, Hao Zhang, Tianhe Ren, Shilong Liu, Xueyan Zou, Huaizhe Xu, Hongyang Li, Jianwei Yang, Chunyuan Li, et al. Visual in-context prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12861-12871, 2024.
Runqi Lin, Bo Han, Fengwang Li, and Tongling Liu. Understanding and enhancing the transferability of jailbreaking attacks. arXiv preprint arXiv:2502.03002 2025.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403. 12966, 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Long-Chun&u Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-25211 2022.
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, et al. Deem: Diffusion models serve the eyes of large language models for image perception. arXiv preprint arXiv:2405.15222 2024.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- tiiin-of-thouggt prompting for large multimodal models. arXiv preprint N.Nv:2311..101722 2023.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern recognition, 107404 2020.
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang, Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain of thought: Bridging logical gaps with multimodal infillings. arXiv preprint arXiv:2305.022 02300,002377222 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 61 218-226, 2017.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.cm 092888, 2023b.
Weijie Tu, Weijian Deng, Dylan Campbell, Yu Yao, Jiyang Zheng, Tom Gedeon, and Tongliang Liu. Ranked from within: Ranking large multimodal models for visual question answering without labels. arXiv preprint arXiv:2412..0--61, 2024.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Haoyu Wang. Zhuo Huang, Zhiwei Lin, and Tongliang Liu. Noisegpt: Label noise detection and rectification through probability curvature. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2025
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything in context. arXiv preprint N.X.v:23304032284. 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint www.x::223..117700 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Max Wertheimer and Kurt Riezler. Gestalt theory. Social Research, pp. 78-99, 1944.
Huangying Zhan, Jiyang Zheng, Xu, Ian Reid, and Hamid Rezatofighi. Activermap: Radiance field for active mapping and planning. arXiv preprint arXiv:2211.12653 2022.
Lei Zhang, Yunshui Li, Jiaming Li, Xiaobo Xia, Jiaxi Yang. Run Luo, Minzheng Wang, Longze Chen, Junhao Liu, and Min Yang. Hierarchical context pruning: Optimizing real-world code completion with repository-level pretrained code Ilms. arXiv preprini arXiv:2406. /8294, 2024a.
Shaokun Zhang, Xiaobo Xia, Zhaoqing Wang, Ling- Hao Chen, Jiale Liu, Qingyun Wu, and Tongliang Liu. Ideal: Influence-drivee selective annotations empower in-context learners in large language models. arXiv preprint urXiv:2310. 10873, 2023a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search, 2022a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36, 2023b.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL https:///lavae .l..g...... gigggittuuu thub. o bl 20000/2222202222 10443-001aasa
vivvviie wiiexttvviee......
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thain-of-thought reasoning in language models. arXiv preprini arXiv:2302.009233 2023c.
Ge Zheng. Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of- thought prompting for multimodal reasoning in language models. Advances in Neural Information Processing Systems, 36:5 168-5191, 2023.
Jiyang Zheng. Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes. Towards open-set object detection and discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3961-3970, 2022.
Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vasion-language models. arXiv preprint arXiv:2403. 14774, 2024.
A ANALYSIS OF OBJECT DETECTION AND IMAGE INPAINTING
Object Detection Table 6 presents the quantita- tive performance of our CoF method compared to baselines. CoF achieve increment over the random on the L-IoU by 14.8%. While the quantitative results of SupPR and CoF are very similar in this tasks, with CoF slightly higher in the metric. However, by observing the qualitative results in Figure
Model
Random
CoF
L.aMA-7B w/ VQ-GAN $7.49 3.7
$1.63
0.8
Table 7: Failure Rates (L) - -bject Detection
13

---

Published as a conference paper at ICLR 2025
Object Detection
Image Inpainting
Method
($7)
MSE (%.)
(%L
Random
17.19 0.6
0.91 4 0.04
0.64 +: 0.01
SupPR (Zhang et al,, 2023b)
19.65 2.9
0.87 0.06
0.55 0.07
CoF Prompting (Ours)
19.74 0.8
0.61 0.01
0.47 0.02
Table 6: Object Detection and Image Inpainting Results of CoF Prompting LLaMA-7B
we can still observe the difference in between the two methods, where CoF are more accurate in locating the boxs and reconstruct the original input. We additional calcuate the failure rate, where the predicted bounding boxs are completely disjoint to the ground truth boxs. Failure cases for detection are detailed Table where CoF reduces failures by 11.9% on LLaMA-7B for object detection.
Image Inpainting As shown in Figure 3, the overall qualitative performance of LAVM on inpainting task exceptional. However, they still benefit from proper prompting. By applying CoF prompting. the generated patches are more natural and of higher quality compared to the baselines. Table 6 (Right) shows that our CoF method quantitatively outperforms the baselines, achieving a 4.3% and 1.7% improvement in MSE and improvement in LPIPS over the second -ighhsst prompting method.
B THRESHOLDING PERFORMANCE ANALYSIS
LLAMA 200M
CCP
Hardan
100.9 DO 0O0 7. EESOROSS 20000 2.9903
1,000 EPORSOR SWSTCC 500 440 3.50-3 Trowehold
w/
1008 DHOOSOR ANDT M00 WWW. 2.
1020 DROWSER SOODO 938 61640300
Figure 5: Image Segmentation and Pose Estimation Results for various black rate thresholding. Our method consistently outperforms the baselines on different pre-trained models across various threshold rates, demonstrating the stable performance of CoF prompting.
In this section, we analyze segmentation performance by thresholding the black rate of the prediction. The black rate represents the proportion of the black area in the predicted results. We assess the performance of COF prompting at different sizes of the predictable object to ensure its contribution stable to the LAVMs. Figure demonstrates the performance comparison between COF prompting and the Random baseline across two metrics. Across varying thresholds, COF consistently outper- forms the Random baseline. This showcases that COF prompting maintains robust performance in enhancing the predictive capabilities of LAVMs regardless of the size of the predictable object.
C VISUALISATION OF RESULTS OF LAVM w/ LLAMA-1B
Here we present qualitative results of image segmentation and pose estimation using LAVM with LLaMA- IB. As demonstrated in Figure 6, using CoF prompting significantly improves the accuracy of object mask identification for image segmentation. Similarly, the quality of the estimated skeletons also better when applying CoF prompting.
14

---

Published as a conference paper at ICLR 2025
$
L
Roj
Figure 6: Results on LLaMA-IB IB Model. The first and fourth rows are the original inputs for image segmentation and pose estimation, respectively. Orange boxes show the predictions given random prompts. Blue boxes show the predictions using Chain--o-Foceh prompting.
D REVERSING ORDER OF INTERMEDIATE REASONING STEPS
HaMM--B
Prompting Method
Image Segmentarion
Proce Estimation
(8/"
P-ACC (%K))
Toll (9h1)
P-ACC (9h1)
COF
52.53
67.05
2,80
13.34
65.37
10.52
Table 8: Reversed Intermediate Reasoning Steps with the LAVM w/ LLaMA-7B
The core of CoF prompting is generate series of intermediate reasoning steps for sequentially prompting the LAVMs. The reasoning path is created based on the saliency paths we identify within individual images. Here, we explore whether the order of reasoning steps will affect the in-context learning of LAVMs. To this end, we present qualitative comparison in Figure 7 and quantitative comparison in Table 8. is observed that reversing the order the intermediate steps can impact the in-context predictions of LAVMs; however, compared the predictions in Figure 3, we conclude that reverse sequential prompting is still better than directly showing the LAVMs the full target.
E DEPENDENCY ON SALIENCY DETECTORS
Measuring saliency (Huang et al., 2023) is important step in method. Here, assess the sensitivity of our prompting method to variations in saliency detectors, we further employed a different approach: GradCAM (Selvaraju et al., 2017) compute saliency scores. Figure 8 demonstrate the
15

---

Published as a conference paper at ICLR 2025
Figure 7: Qualitative Results of reversing intermediate reasoning steps with the LAVM w/ LLaMA- 7B. The second row shows the CoF prompting output. The third row show the results of using the same prompt, but reversing the order in intermediate steps.
Figure 8: Visual Attention of different saliency detectors. Green boxes show the results given by GradCAM, Yellow boxes show the results given by U2-net.
different attention maps visualized from GradCAM and U2-Net. respectively. Table shows the results for the LLaMA-7B LAVM on the four tasks, comparing U2-Net and GradCAM. Notably, switching the method for measuring saliency scores does not result in significant differences in performance, Based on the observation, we conclude that both approaches effectively detect salient regions, and the consistent in-context learning performance further highlights the robustness of our approach.
Prompting Method Task
Segmentation
Pose Estimation
Object Detection
Image Inpainting
loU (1V P-ACC (1)
ToU (1V ACC
L-oo (*)
MSE ()) / /LPPPS (L) 0.63
CoF Prompting GradCAM
52.687 67.61767.14 67.14
2.77 13.16
19.77
Col Prompting C2-Net
52,537 67.05
2.80 13.34
0.61/
Table 9: Comparison of CoF Prompting with different saliency detectors.
F ADDITIONAL QUALITATIVE RESULTS
In this section, we present additional visualizations of the image segmentation and pose estimation tasks to illustrate the in-context learning performance of COF prompting. For image segmentation, the results for the LLaMA-300M model are depicted Figure 10, while Figure 11 showcases the outcomes for the L.LMM-1B model. For pose estimation, provide visual evidence of the
16

---

Published as a conference paper at ICLR 2025
improved performance facilitated by COF prompting using LLaMA-300M (Figure 12) and LLaMA- IB (Figure 13). These results demonstrate the efficacy of our method enhancing LAVMs' predictive ability both tasks through structured, reasoning- based prompting. Based these additional visualizations, as well as the results shown in Figure 3, we can observe that larger models have strong predictive power both tasks. This implies that, in order achieve predictive capabilities similar to expert models, we will need to scale the parameter size of the models as well as the size of the training data.
G LIMITATIONS AND FUTURE DIRECTIONS
While visual prompting methods can potentially enhance the predictive performance of Large Vision Models, their limitations are constrained by the capacity of these models. Practical usage of LAVMs requires stronger and more robust pretrained models, along with the advancement of in-context learning methods. Instances where current LAVMs produce pure black predictions highlight their fundamental instability, raising concerns about their trustworthiness in real-world deployment.
We observed that the failure cases are primarily associated with two factors: model scale and prompt selection. Model scale the major factor, as LAVMs with larger parameter sizes tend to exhibit fewer failure cases, Failure cases can also arise from the choice of visual prompts, and our experiments demonstrate that the proposed prompt selection module effectively reduces the number of failures.
To further investigate, we identified prompts that previously caused failures in in-context predictions and were not encountered by the model during pre-training. We then switched the test input while using the same failure-inducing prompts, and the failure persisted across differrnnt test inputs. However, when we replaced the prompts with training samples from datasets used the pre-training process, the success rate significantly increased. Based these observations, hypothesize that the root cause of failure cases is related to the model's out-of-distribution generalization ability with respect the prompts. The model may fail to perform in-context learning if the prompts unseen during pre-training or exhibit a domain gap.
The community as whole desires unified solution for all vision tasks. Therefore, the authors advocate for continued research into building robust large vision models.
17

---

Published as a conference paper at ICLR 2025
Figure 9: Ground Truth Visualization for the test input.
18

---

Published as a conference paper at ICLR 2025
Figure 10: Image Segmentation Results from LLaMA-300M w/ VQ-GAN using COF prompting.
Figure 11: Image Segmentation Results from LLaMA-IB w/ VQ-GAN using CoF prompting.
19

---

Published as a conference paper at ICLR 2025
Figure 12: Pose Estimation Results from LLaMA-300M VQ-GAN using COF prompting.
Figure 13: Pose Estimation Results from LLaMA- IB w/ VQ-GAN using CoF prompting.
20


Reviews:
[{"review_id": "pO5DyIMlHV", "reviewer": "Reviewer_CkD8", "summary": "The paper introduces Chain-of-Focus (CoF) Prompting as a method to enhance large autoregressive vision models (LAVMs) for in-context learning (ICL) in computer vision. Drawing inspiration from Chain-of-Thought prompting in natural language processing, CoF Prompting enables vision models to perform step-by-step visual comprehension. It tackles the lack of logical structure in visual data by generating intermediate reasoning steps based on visual saliency. CoF Prompting also facilitates the creation of customized prompts by selecting the most contextually relevant ones, based on query similarity and target richness. The effectiveness of this method is supported by extensive experiments, demonstrating that it significantly improves the models' ability to make inferences on visual understanding tasks, leveraging the recent advancements in LAVMs that utilize purely visual inputs for ICL.", "questions": "- Would the model perform equally well if it viewed the same number of images within reasoning steps (not randomly selected, such as viewing the same number of final step reasoning images)?\n\n- If you simply use this logic for Chain-of-Focus: each step of reasoning progresses from images of a single object to two, and then to multiple objects, where do you think the disadvantages lie compared to your method?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper introduces the interesting method of Chain-of-Focus Prompting, with strong and reasonable motivation. \n\n- The experiments are comprehensive.", "weaknesses": "- Compared to previous work, such as \"A Generalist Painter for In-Context Visual Learning,\" this paper only addresses two tasks: image segmentation and pose estimation. If this method does not work for other tasks, it could decrease the contribution of the paper.\n\n- I am somewhat skeptical about why this method works. For instance, would the model perform equally well if it viewed the same number of images within reasoning steps (not randomly selected, such as viewing the same number of final step reasoning images)?\n\n[1] Wang, Xinlong, et al. \"Images speak in images: A generalist painter for in-context visual learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJ64LqFnVO", "reviewer": "Reviewer_eyXR", "summary": "This paper propose Chain-of-Focus(COF) Prompting, which enhances vision models by enabling step-by-step visual comprehension. CoF automates prompt design by selecting the most relevant and informative prompts from existing candidates and creates intermediate reasoning steps for prompt targets. Experiments on various downstream vision tasks demonstrate the effectiveness of the proposed method.", "questions": "Can CoF address the issue of distribution shift, that is, when there is no data related to the test data in the prompt pool?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well written. The methods are innovative, the experimental results are significant, and the visualizations are intuitive.", "weaknesses": "1. The candidate prompt pool comprises 50,000 training images and annotations, which raises concerns in specific scenarios due to its inability to ensure data privacy. Additionally, it necessitates significant resources for the storage and retrieval of prompts.\n2. Figure 3 does not include visualized results of the ground truth.\n3. There is no comparison of the computational complexity and resource overhead of the different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRhvPPUSuj", "reviewer": "Reviewer_GMfc", "summary": "This paper extends Large Autoregressive Vision Models (LAVMs) with the capability of in-context learning for visual inputs by proposing a chain-of-focus (CoF) prompting approach. The CoF decomposes a visual prompt to intermediate reasoning steps by ranking salient regions of the prompt image, as well as taking into account the visual similarity and annotation richness to the test image. The saliency scores are obtained by an off-the-shelf saliency detector. The relevance of queries is modeled by the Jaccard similarity index of the two sets of visual codebooks in VQ-GAN. The annotation richness is modeled by the number of unique indices in their codebooks. The proposed method has been evaluated on 4 vision tasks, e.g., image segmentation, object detection, image inpainting, and pose estimation, which show the effectiveness of the CoF prompting, upon two backbone LAVMs.", "questions": "Please discuss the pros and cons of using more diverse visual prompt images or more intermediate reasoning steps of one prompt image.\n\nAny alternatives to the saliency scores to measure the reasoning steps? Some objects like faces, animals or intensive actions tend to capture visual attentions easily. \n\nSome relevant missing references, please discuss the differences:\n\nChain-of-Spot: Interactive Reasoning Improves Large Vision-language Models, 2024.\nAccelerating Pre-training of Multimodal LLMs via Chain-of-Sight, 2024.\n\nWhat is the subtle difference between image relevancy and relevance? ll.140 => an image retrieval framework; ll.216, in Fig.2, => query relevance or relevance of the queries; => target informativeness or informativeness of task objectives? Btw: informative and informativeness are very vague terms, please define them first.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed CoF prompting method is well motivated, which shall interest many ICLR attendants.\n2) The proposed approaches, using the saliency score rank, the similarity of two sets of codebooks in VQ-GAN, are reasonable and easy to re-produce. \n3) The evaluation is fairly convincing to show the effectiveness of the proposed method, though not as significant as I expected.\n\nOverall, this is a descent work which may deserve to share with the community in time.", "weaknesses": "1) The reasoning steps of 0, 1, 2 are evaluated in Fig.4. More steps (just 2) do not show clear advantage, which is a bit disappointing to validate the proposed CoF prompting. I wonder if just using more diverse visual prompts that are also relevant with enough annotations, the performance may match or even out-perform using intermediate reasoning steps of the visual prompts?  \n\n2) The comparison baseline is a random selection scheme, which may be too simple. The performance gain over this simple baseline is not that significant as I expected.\n\n3) The specific approaches, like the off-the-shelf saliency score and the number of unique indices in their codebooks, are somewhat simple and intuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGNvDQ59Fu", "reviewer": "Reviewer_p93y", "summary": "This paper aims to address an inherent challenge in the computer vision community, where, unlike language data, images lack clear logical structures. CoF prompting introduces intermediate steps in the visual learning process to enhance the ability of VLMs to understand and predict complex visual images. The authors propose using saliency maps to determine which parts of an image should be focused on sequentially, mimicking human cognitive processes. This method allows VLMs to perform better on fundamental vision tasks such as image segmentation, pose estimation, and object detection by progressively building a context around the test input. The authors comprehensively evaluated the proposed method across several models and datasets, with extensive experiments showing the efficacy of CoF prompting. It outperforms several baseline methods and demonstrates some improvements, such as Intersection over Union (IoU) and Pixel Accuracy (P-ACC). This paper also delves into ablation studies to assess the contributions of different components, providing insights into the role of cognitive reasoning, query relevance, and annotation diversity.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative Adaptation of CoT for Vision: The introduction of Chain-of-Focus (CoF) prompting is a creative adaptation of chain-based reasoning, typically used in language models, for vision tasks. The authors contribute a novel and interesting insight into the inherent logical structures of images. Previously, we thought images contained visual redundancy, which was proposed and widely adopted in Masked Autoencoders pretraining. Such insights propose a new idea for visual modeling, which improves vision models to handle complex, multi-step tasks.\n\n2. Comprehensive Evaluation: The authors conducted thorough experiments across various models (e.g., LLaMA-300M, LLaMA-1B, and LLaMA-7B) and tasks (e.g., segmentation, pose estimation, object detection). This robust evaluation not only demonstrates the generalizability of the approach but also its effectiveness across different settings.\n\n3. Interesting Saliency-Based Reasoning: The use of visual saliency to create intermediate reasoning steps adds a cognitive layer to how models process visual data, mirroring human visual attention mechanisms. This results in more informed predictions, especially in tasks where spatial and object importance matters.\n\n4. Detailed Ablation Studies: The authors provide detailed ablation studies, which highlight the relative importance of different components of the proposed method (cognitive reasoning, query relevance, and annotation diversity). This adds clarity to which parts of the model drive the most significant improvements.", "weaknesses": "1. Limited Exploration of Computational Complexity: While the method is novel, there is limited discussion on the computational overhead introduced by the saliency-based intermediate reasoning steps. Given that real-world applications demand efficient processing, an evaluation of the method’s scalability in terms of computational cost would have been valuable.\n\n2. Generalization Across Different Vision Tasks: While CoF prompting is tested on several vision tasks, the experiments are focused on a few select tasks, mainly segmentation and pose estimation. Broader evaluation on a wider range of tasks, such as video understanding or 3D object recognition, would further validate the method’s versatility. Therefore, I strongly suggest the authors try some experiments on video understanding to further elaborate on the effectiveness and commonality of this method.\n\n3. Handling Failure Cases: The authors acknowledge that VLMss can produce failures (such as pure black predictions), but the analysis of these failures remains surface-level. A deeper dive into understanding why these failures occur and how to mitigate them would strengthen the paper’s conclusions. This part could be integrated into the Conclusion and Limitation section, and it will also inspire follow-up research.\n\n4. Dependence on Saliency Detection: The success of CoF prompting is tied closely to the quality of the saliency detection model used. The authors briefly mention the model (U2-Net), but the sensitivity of the approach to variations in saliency detection quality is not explored. If the saliency model performs poorly, it could compromise the entire prompting process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)