CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning
Authors: Ji Qi
Ming Ding
Weihan Wang
Yushi Bai
Qingsong Lv
Wenyi Hong
Bin Xu
Lei Hou
Juanzi Li
Yuxiao Dong
Jie Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Reasoning
Vision-Language Models
Datasets Synthesis
Graphical Math Annotation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Vision-Language Models (VLMs) have shown broad effectiveness due to extensive training that aligns visual inputs with corresponding language responses. However, this conclusive alignment training causes models to overlook essential visual reasoning, leading to failures in handling detailed visual tasks and producing unfaithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs to tackle problems step-by-step with evidence. After training, models can solve various visual problems by eliciting intrinsic manipulations (e.g., grounding, zoom in) with results (e.g., boxes, image) actively without relying external tools, while also allowing users to trace error causes. In this paper, we study the comprehensive methodology that includes: (1) a flexible design of manipulations based on extensive analysis, (2) an efficient automated data generation pipeline, (3) a compatible VLM architecture capable of multi-turn, multi-image, and (4) a model training process for versatile capabilities. With the design, we also manually annotate **6K** high-quality samples for challenging graphical mathematical problems. Our trained model, CogCoM, equipped with this mechanism and 17B parameters, achieves SOTA performance across **9** benchmarks in **4** categories, demonstrating its effectiveness while maintaining interpretability. Code, model, and data are available at https://github.com/THUDM/CogCoM.

Published as a conference paper at ICLR 2025
COGCOM: A VISUAL LANGUAGE MODEL WITH CHAIN- OF- OF-MANIPULATIONS REASONING
Ji Qi ,Mig Ding', Weihan Wang Yushi Bai Qingsong Lv Wenyi Hong Bin Xu Lei Hou", Juanzi Li", Yuxiao Dong", Jie Tang Tsinghua University TZhipu Al
Find regions using GROUNDING/rood sigri) to get 1144661222 15, Zoom into: (3.4,666))
No, there sign showing one way.
with Chain of Manipulations
Can turn left on this road?
plane!
GogCoM: Applying GRUUNDINGdhe clock) Q.
Using GRUUNDINGode plano) to rind planes, and there are two regions (1255, 217, 312, 245), (195, 113, 481, tooj thet con found.
CogCoM:
Using GROU DENGUU the inviiduaa wheels the mick) to ooate all the wheels in the picher, rostting in the boues of (1S6, 256, 96, 515 1. [105, 200, 430)....
CogCoM:
resaiting in the region of L1844,51 (MR), 1ORa))
Use Comp/oominn $14,53 531 (480 1962
With the most nootlle one. using GROUNDIE UNNIING
[1-2) to crop and then moom in two iincs re-laput this now image.
the loao in 1933 113, 481, 205TD to find the lopo. resuting in (1246, 142.307,
Based the identified beues of the wheclls, the zumber of visiile wheel the tuck is ten.
Liing RoaTTinails cleck now
Using CropZoenlal11266 142, 07: 16]]]. Sy get image
AARRANCE
imsgel to countse the time on the cleck. roullt
The brand rome Aittirnnc
Detailed Recagnition
Counting Objects
Read Time
Q what year did the famiitiies pesic?
Whar color unbrella the men are standing)
Q: cirole with radius 5.0.0 the length chard) then divance
CogCoM: Findly using GIROUNDENG faulitios values) to find the dush will, and values posiiion (1116, 114, 7551) are 221, 213, 212, 193, 15% 195, 183, 151,
CogCoM: Devemine if umbeella
CogCoM: Dan stniggtt line with LINE/]]:54
x113,701 obtin new image after duvisg.
124,
vieble.
With the now image
odicuiar AB point Since
is the mieporr of All, miking
71,, using LI74 imag: duswing.
jo be in frro ing stance.
AC 4.n long in the right tringle AOC: OX 5.0, AC-4 Accondi ing the Pathogoc theorem. unsver
Sincs fiorc umbeella colte cennot be dccemized..
Read Figures
De-halluciation
Math Geometry
Manipulation creation
Manipulation result (same color for referential relation)
Figure 1: CogCoM solves various visual problems with Chain of Manipulations (CoM) reasoning, which generates evidential and explainable steps without relying on external tools.
Work done when JQ, WW. YB, and WH interned Zhipu AL "Corresponding authors: BX and JT (xubin jietang taanng@@tsingu..d..ccm

---

Published as a conference paper at ICLR 2025
ABSTRACT
Vision-Language Models (VLMs) have shown broad effectiveness due to extensive training that aligns visual inputs with corresponding language responses. How- ever, this conclusive alignment training causes models overlook essential visual reasoning, leading to failures in handling detailed visual tasks and producing un- faithful responses. Drawing inspiration from human cognition in solving visual problems (e.g., marking, zoom in), this paper introduces Chain of Manipulations, a mechanism that enables VLMs tackle problems step-by by-step with evidence. After training, models can solve various Visual problems by eliciting intrinsic manipulations (e.g, grounding, zoom in) with results (e.g., boxes, image) actively without relying external tools, while also allowing users to trace error causes. In this paper, we study the comprehensive methodology that includes: (1) a flexible design of manipulations based extensive analysis, (2) an efficient automated data generation pipeline, (3) compatible VLM architecture capable of multi-turn, multi-image, and (4) model training process for versatile capabilities. With the de- sign, we also manually annotate 6K high-quality samples for challenging graphical mathematical problems. Our trained model, CogCoM, equipped with this mecha- nism and 17B parameters, achieves SOTA performance across 9 benchmarks in 4 categories, demonstrating its effectiveness while maintaining interpretability. Code, model, and data are available at https: / gi seggttuse thub. com/ TH/DM/ com/THUDM/CCoCoM
INTRODUCTION
we apdi Ca
Zanriee beeogge re 11-
tel
Large Language Model
Large Language Model
Imag toham
wywtkaar Bhy plaarit thond moit?
Image Shans
What pilae in frsett theirann?
QUICK DEPOSIT
Figure 2:
In comparison with existing VLMMs, CogCoM performs the multiple steps of evidential reasoning with chain of manipulations (CoM) to achieve the faithful answer to visual scene.
Benefiting from the advantage of Large Language Models (LLMs) in broad world knowledge, large Vision-Language Models (VLMs) (Alayrae et al., 2022 Wang et al. 2023a) that further trained to understand visual inputs have shown strong capabilities across a wide range of multimodal scenarios, such as visual question answering (Liu et al. (023b), visual grounding (Peng et al., 2023],, and optical character recognition ZZhag et | 2023bb. Research employing VLMs as foundation models (Bai al. 2023 Sun et al. 2023b] Wang al. (023a) typically involves two main training stages, where the first stage develops intrinsic visual understanding through exposure to massive image-caption pairs, while the second stage endows the models with problem-solvin abilities via the instruction
tuning.
However, existing tuning methods train models to provide conclusive responses to instructions based on visual inputs, causing them to overlook essential intermediate visual reasoning. This often leads to failures in handling visual tasks requiring subtle details, producing unfaithful responses, and even generating hallucinations. For example in the left subpiot of Figure 2 we test the top-performing model, CogVLM (Wang et al.) 2023a) on the details in the image (i.e., text written on a pillar), and it directly responds an incorrect answer (i.e., NO SMOKING), most likely due to bias to visual or linguistic priors (i.e., rypical office scenes with a pillar). The lack of essential visual reasoning about the visual scene can result in hasty response (Hwang al. 2023
2

---

Published as a conference paper at ICLR 2025
Humans solve problems regarding visual details by marking processing the given images for convenience and precision, which we refer to as manipulations. For example, we identify targets by sequentially locating references, and focus subtle details by zooming into the corresponding region. Most VLMs develop numerous intrinsic capabilities (e.g. grounding boxes, recognizing text) during the first stage of training. By further imitating the fundamental human behaviours (e.g.. cropping, zooming in), models have the potential to perform this cognitive reasoning process. Three major obstacles enabling VLMs perform such reasoning are (1) the flexible definitions of manipulations that cover most visual problems, (2) an efficient data collection pipeline capable of producing abundant training data, and (3) a multi-turn multi multi-image structure that is compatible with existing models.
Inspired by the human cognition in solving visual problems, we introduce Chain of Manipulations (CoM), a mechanism that enables VLMs to solve problems step-hy-step with evidence, with each step potentially involving a manipulation on the visual input and its corresponding result, both generated by the model to facilitate the success and fidelity. This paper studies comprehensive methodology, including manipulations design, data collection, model architecture, and training process for developing general VLMs with this mechanism. Based pilot experiments, we first formally design 6 atomic manipulations, that are capable of addressing diverse visual problems. Next, we propose cascading data generation pipeline that leverages reliable large language models (LLMs, serving as linguistic annotators) and visual foundational models (VFMs, serving as visual annotators), to automatically generate abundant training data. We collect 70K training samples involving visual reasoning chains using this pipeline. We then devise a multi-turn multi-image model architecture that is compatible with typical VLMs. Based on data recipe incorporating the curated corpus, we finally train general VLM equipped with CoM reasoning mechanism, named CogCoM, which possesses capabilities of Chat, Captioning, Grounding and Reasoning. Additionally, benefiting from the expressive capability of the proposed mechanism, we manually annotated 7K high-quality graphical mathematical samples, each accompanied by a CoM reasoning process, to further advance VLM research in solving challenging mathematical tasks.
We conduct extensive experiments on 9 benchmarks from 4 categories, including TextVQA (Singh
iet alla| 2019). ST-VQA (Biten et al. 2019), Tally VQA (Acharya al. 2019), and GQA Hudson & Manning (2019) for detailed visual question answering, RefCOCo YYu et al. 2016). RefCOCO+ Yu
et 2016)) and RefCOCOg (Mao et al., 2016) for visual grounding, POPE (Li et al. 2023d) for hallucination validation, and MM-Vet ( (Yu et al, 2023b) for general multimodal ability. Our model
the detailed VQA and grounding benchmarks, respectively, and the superior performance on the general multimodal benchmark. The results demonstrate the effectiveness the mechanism while preserving the interpretability of outputs.
achieves up 9.0 and 1.09 accuracy improvement
2 TERMINOLOGY
We first conduct pilot experiments to explore potential atomic manipulations that can address diverse visual problems.
Specifically, given a question about an image, we prompt the advanced large language model, GPT--4, to generate solving steps by optionally utilizing possible actions on the image to fa- cilitate problem-solving
12300
We conduct this ex- periment on I 170K questions from TextVQA, dataset requiring detailed reasoning and recogni- tion on images. To ensure stability, we manually write 4 demonstrations as priors. The detailed statistics are available in Appendix E.3
1550, extract
locats, Kot for
zoom In. focas
2200
4Tr-22401:05
We utilize the StanfordCoreNL.P toolkit to ex- tract verb phrases referring to the actions, and the distribution of frequencies is shown in Fig- ure 3. Through result analysis, we find that most of the actions can be mapped to 6 fundamen- tal manipulations on images: OCR, Grounding, CropZoomln, Counting, Calculate, and Line.
Count number....
Calcurate 355e2
draw, adl....
Figure 3: Distribution of the generated 465 actions base on GPT-4, mapped into 6 manipulations.
3

---

Published as a conference paper at ICLR 2025
Based on the observation, we formally predefine set of 6 atomic manipulations, which can either be developed during pre-training or learned through fine-tuning by imitating human behaviors: M C(OCR(tgt) + tæt, Grounding (ttt) + bbx, Counting(tgt) prop/zoomnn(brr x) - img, Line(pls) + img), where the corresponding parameters or results tgt, fxt, bbx, num, I. img.pte refer to the target description, texts, bounding boxes, numbers, zoom ratio, image, and points, respectively. In addition to the predefined manipulations. we also allow models to create new manipulations during inference to facilitate problem-solving. We empirically find that more complex goals can be derived from these fundamental manipulations.
num, Calculate(tgt) - num,
We then define the standard CoM data structure to streamline the subsequent data construction and validation process. Given a question Q about an initial input image Io. a VLM equipped with chain of manipulations mechanism solves the problem to achieve final answer as VI.M. (A, To Q), where ç refers to the reasoning chain with evidence,
(stepi, step2,... step, = (fr.a), f M
(1)
where C= (ca, .2.,..9..) refers to the free-form textual descriptions incorporating manipulation names Si and corresponding results by utilizing St. This definition explicitly declares the symbolic execution process, while remaining compatible with linguistic reasoning steps. Based on this defini- tion, we can clearly construct standard CoM samples that incorporate the manipulation executions and linguistic steps with evidence. After the data construction, we can utilize a simple method to convert the standard CoM samples to the format of compatible VQA structure.
3 DATA COLLECTION
VQA Sample
Linguistic Annotation
Visual Annotation
Compatibbe VQA
wittien gillar
top?
A QUICK DEPOSIT
Ganerate sclving stops aiven quastion, optionally in manipuaation JM neady
Govredingtot [ blir Greendingligtl Mayy
Grunddingooa. shry (C()nng)) lafs
Lingulstic Annotators (LLMMs)
Visual Annotators (VFM)))
Ao Using Brourding to mas black an to
(2, In Ad
Liing tnd pilee at 2 to find .. the read sog on liter.
Using Chourdingtte sot witten the pilarr 1o
region
Using Cre.zoennn 4i toop on soem by ttrme
DFB positive chains
BUCCA
REPESIT
QUCKESPOOTT
Laing OCRL (:) recogrizo the tuxt withinn Image resullii "OUICK DEPOSITT
A
(a.
Figure 4: A cascading data generation pipeline that automatically produces standard CoM samples. Given an original VQA sample, the linguistic annotator (LLM) taught with usage of manipulations (prompt) is first asked to provide solving steps for the question Q, and the visual foundational models (VFMs) are then engaged to replace the manipulations results, followed by a final traversal on the tree branched by the possible manipulation results to find positive paths terminating to the answer A.
In this section, we first introduce the automated data generation pipeline (illustrated in Figure 4. that employs reliable LLMs as linguistic annotators and VFMs as the visual annotators to produce CoM samples upon prevalent VQA corpus, and then present the manual annotation of high-quality CoM samples for the challenging graphical mathematical problems.
3.1 AUTOMATED DATA GENERATION
Given a general corpus D {11, Q, A]} consisting of triplet samples of images with corresponding visual question-answer pairs, our automated data generation pipeline consists of linguistic annotator and several visual annotators according to the manipulations. For a question Q in each sample, we first engage the linguistic annotator generate manipulations-assiiee format (fr. c.), where the corresponding results of the instantiated manipulation executions are set with variables as placeholders. In this paper, we adopt GPT-4 (OpenAl 2023ab, large language
solving steps with the CoM
4

---

Published as a conference paper at ICLR 2025
model with reliable language understanding and generation abilities the linguistic annotator. We design a comprehensive prompt including the task requirements, usage of manipulations, and output data format, and further manually annotate 5 demonstrations for stable generation. The detailed implementations are available in Appendix E.4
We then employ essential visual annotators to supply the results of manipulations requested in the solving steps by exactly performing the corresponding manipulations. By empirically analyzing the manipulations from both predefined set and newly created ones (refers to Appendix [E.3 for a detailed statistics), we reveal the Grounding and OCR are two fundamental manipulations, and most of the others can be consequently derived (e.g., CropZoomin along a region of box, Counting upon recognized boxes, and Calculate for the recognized formula). Therefore, we employ two visual fundamental models, GroundingDINO (Liu et al. 2023c) and PaddleOCR (Du et al. 2020), and develop the implementations of these manipulation The execution of the manipulations will transform the sequential reasoning steps into tree T, as the input of current manipulation fi (ara) may rely on one of the multiple results of previous manipulations f2 (Zb, Le), L.C., La rely on Is (e.g,, step 2 for finding pillars in Figure [5) We then perform a Depth First Search (DFS) traversal on each produced tree to find all positive paths {PP,), E T, = 1, 1.2..... that terminates with the golden answer as the result of the last manipulation. Based on this recursively searching method, most of the generated positive paths are guaranteed to be error-free. We implement this pipeline on 3 existing datasets that require detailed recognition or counting, TextVQA (Singh et al. 2019). ST-VQA (Biten ct al. 2019). and TDIUC (Shrestha et al. 2019), build 70K CoM samples The prompt design, an example with linguistic and visual results, and algorithm are available in Appendix E.
3.2 HUMAN ANNOTATION
The analysis from Fig. I of AlphaGeometty (Trinh et al. 2024) shows that outputting auxiliary lines in linguistic reasoning process helps LLMs to solve complex geometry problems. Benefiting from the expressive capability of CoM structure, we have also manually annotated high-quality CoM samples for the graphical mathematical problems to facilitate VLMs in solving this challenging scenario. Similar the automated pipeline, we engage 10 human experts as the linguistic annotators and visual annotators, where each expert is asked to annotate the linguistic solving steps and the use of manipulations, as well as the results of manipulations on images. We perform this annotation on the Math Vista (Lu et al. 2023) and ChartQA (Masry et al., 2022). which include geometric and chart math problems, resulting the collection of 7K high-que quality CoM math samples.
Finally, we adapt the CoM samples to be compatible with VQA-style training samples. For each CoM sample including n images with manipulations outputs (Iu, Q, Co. I1, C1. Ci.....yy, A), we convert into multi-turn VQA sample segmented by the images [(oo Q, Co), (,,, Q,C,), (Tn, Q, A)), where C, represents the intermediate steps between 7, and / and Q a simple prompt asking model to answer question based on history. This transformation converts CoM samples into multi-ture VQA samples that are compatible with existing VLMs training corpus. The detailed statistics of the data generation are available at Appendix E3
4 MODEL TRAINING
4.1 ARCHITECTURE
We use the same model architecture as CLM (Wang et al. (003a), a general VLM approach that involves four fundamental components: (1) a Visual Encoder, (2) an MLP Adapter, (3) an LLM Backbone, and (4) a Visual Expert Module, for a reliable multimodal understanding. Concretely, the pre-trained EVA2-CLIP-E (Sun et al. 2023a) with 4B parameters and Vicuna-7B-vl.5 (Chiang et af. 2023) are adopted as the visual encoder and LLM backbone, respectively. A two-layer MLP (SwiGLU Shazeer 2020 is further engaged map the output of the visual encoder into the linguistic space of the LLM backbone. The visual expert module adds the vision-specific weights into the attention layer and feed-forward layer of each block in the LLM backbone, resulting in total of 6.5B additional parameters for the deep fusion of modalities.
We simply implement the CropZoomfn referring to human hehaviors with local code interpreter.
'The success rate of GPT-4 to achieve the positive paths is 0.3555.
5

---

Published as a conference paper at ICLR 2025
Prop 0000.
Reeporee_
1-0
Stage 1: Pre-tralning
Data structure:
Foundational understanding Grounded understanding
-lmagg, caption)) -fragee responsej)
LLM
Klemwarry
Image
Stage 2: Fine- tuning
Data structure
Projector
Muttimodal problams- addking
(image prompt, responsel)
VIT
Chat, Captioning, Grounding, Ressoring
Figure 5: Left: A compatible VLM architecture capable of multi-tury turn imate understanding. Right: An effective training process to develop a general VLM with versatile capabilities.
Based this general architecture, develop memory-based multi-turn multi-image VLM architecture. Specifically, for a multi-turn VQA sample [[,, Qe. Ac) = 1, 2, ...], where A, refers to C. in CoM, we keep the accumulated KV memories of each layer in the L.LM backbone throughout these turns. And at each turn t in training and inference, we calculate the attention function att as:
att X softmaa( Q,Ki V
Vd
(2)
K trunc(concat) Ko: Ki,.., K())
V trunc(concat( Vo, Vi,.., V:))
where Q E IR <x is query representation of current layer, and the Ky V E ((xxt)xd refer to the concatenation of accumulated representations and will be further truncated if the sequence length s x is greater than a predefined threshold. At > 0, the new image 4, will be cropped from 4,-1 and amplified with the Bicubic Interpolation (Keys 1981.
4.2 TRAINING
The proposed CogCoM-17B relies on two main stages training, to develop the capabilities of general multimodal task-solving well as the visual reasoning.
First Stage Pre- Tre-Training This stage consists of two ordinal sub- sub-phases of training for foundational visual understanding and grounded generation. Following the pre-training of Cog VLM (Wang et al. 2023a). we first train model on 1.5B image-text pairs cleaned from the LAION-2B (Schuhmann et al. 2022) and COYO-700M (Byeon et al., 2022) with 120,000 iterations and batch size of 1,2. We then train model on 40M grounded image-question- saree--uuustttnnnsssweeee -anweerr--- triples cleaned from LAION-I 15M (Li al., 2023b) with 60,000 iterations and batch size of 1,024, where each noun phrase in the answer followed by list of coordinates [[ Yo, I1, I1,
y.).....i referring the phrase to the grounded objects in the image. Both phases adopt the next token prediction objective, and train the 6.5B parameters of visual experts.
Second Stage Alignment This stage further trains the model to align with human preferences on solving practical visual problems. We fuse the produced CoM data with 3 types of corpus, including MultiInstruct (Xu al., 2022), LLaVAR (Zhang et al. (023b)), and ShareGPT4V (Chen et al. al.||202. 2023b) referring the abilities of Ietruction-following texts-recognizing, and detailed-captioningg This fusion results in a total 570K (I, Q, A) samples, where the answer in CoM data consists of multiple turns. For the training data of CoM, randomly prepend a lunching promp Pd to questions Q= P- + Q asking models to optionally use manipulations for the adaption of explicitly eliciting. We empirically show that the model can effectively learn the evidential visual reasoning by ingesting this portion of CoM data. We train model with 14,000 iterations and batch size of 160, where the learning rate reaches 10-5 after 280 steps of warm-up and then decays lincarly. The parameters of 6.5B visual experts are trained with the objective of next token prediction. These two stages of training result in our standard version of CogCoM involving both chat and reasoning capabilities. More training details are available at Appendix F2
2i, f/1 E [000, 999 refer to the normalized pixel coordinates.
*See Appendix F.1 for examples.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENT
To quantitatively validate the suitability and efficiency of the proposed method, we conduct exper- iments 9 benchmarks corresponding 4 categories of multimodal capabilities, well newly constructed testbed that includes the CoM reasoning paths with keypoints-aware metric. Following previous works, train two generalist versions of CogCoM to adapt to different scenarios: Visual Question Answering and Visual Grounding, and we evaluate the standard version through qualitative analysis (Hwang al. 2023). We also evaluate the time complexity of our model.
Detailed Visual Question Answering. This task involves models to perform detailed reasoning recognition on images. We use 4 prominent benchmarks including, GQA (Hud- son & Manning 2019), TextVQA (Singh et al. 2019). ST-VQA (Biten et al. 2019). and VAA (Acharya et al. 2019).
Visual Grounding. Visual grounding evaluates the crucial abilities of VLMMs in meticulous position understanding. We evaluate our model 3 standard benchmarks, RefCOCO Yuu et al. 2016), RefCOCO+ (Yu et al. 2016), and RefCOCOg (Mao et al. 2016).
General Multimodal Capabilities & Hallucination. We also evaluate on a general mul- timodal benchmark, MM-Vet (Yu et al. (202b)), and a hallucination detection benchmark POPE (Li al. 2023d), to investigate the helpfulness visual reasoning.
5.1 EXPERIMENTS ON DETAILED VQA
VLMs have demonstrated well-known superiority in visual scenes with salient content understanding. We evaluate the effectiveness of CogCoM on VQAs for detailed understanding, which typically require models perform multiple actions (find, read) multiple reasoning steps (recognizing and then calculating). Following previous studies (Wang et al., (023a). we train our model obtained from the first-phase of stage-l on mixture of data, including an instruction corpus of Multilnstruct, 13 publicly available VQA datasets (only using training set), a newly created VQA dataset built through promoting GPT-4V (OpenAl 2023b) for image-oriented question- ansser generation, and the automatically generated 70K CoM corpus. This training results in generalist VQA model incorporating CoM reasoning. For all existing VQA tasks, we directly prompt CogCoM with given questions and examine the correctness of outputted answers.
GQA test-balanced
Tally VQA
TextVQa
ST-VQA
Type
Model
simple
complex
Flamingo (Alayrac et al.
2022
GIT
[Wang et al. 2022a
Generalist
G12
Wang
67.3
BL.IP-2
([a
20235
44.7
21.7
InstructB Dai ct al.
2023
49.5
50.71
Monkey
[ 2024
60.7
67.7
Qwen- VL
al. 2023
59.3
63.8
IJaVA-1
Tau al. 20123a
64.7
62.5
CogVLM
Wang ct al.
20258
65.2
79.8
68.0
69.7
61.0
CogCoM
71.7
84.0
70.1
71.1
70.0
Specialist SOTAs
72.1
86.0
75.6
71.4
86.0
(CFR)
(PaLI-X)
(PaLI-X)
(PaLI-X)
(SMoLA)
Table I: Performance on VQA benchmarks, where the results with 1 refer to the few-shot setting.
5.1.1 GQA, TEXTVQA, ST-VQA, TALLYVQA
Settings GQA a compositional VQA benchmark with diverse reasoning questions coming from semantic functional programs. VQA is an object counting benchmark with human-annotatd complex counting questions involving challenging non-zero counterparts. TextVQA and ST-VQA are two texts understanding benchmarks requiring models to answer questions through textual cues on images. We use the official evaluation scripts for GQA and TallyVQA, which calculate the accuracy score by the Exact Matching (EM) between model predictions and answers. For TextVQA and ST- VQA, we submit our model predictions to the official online websites for calculating the accuracy with VQA Score metric (Antol et al. 2015).
7

---

Published as a conference paper at ICLR 2025
Results As the results shown in Table 2 all generalist models, and achieves significant improvements over the baseline model. Specifically, compared to the baseline model, our model achieves up to 5.97 and 9.0 percentage points improvement on the benchmarks that require complex reasoning and detailed recognition, respectively. On GQA and TextVQA, CogCoM also obtains comparable results with the large-scale specialist SOTAs. This result demonstrates the effectiveness of the proposed approach in solving details recognition problem.
CogCoM achieves SOTA performance compared to
5.1.2 EXPERIMENTS FOR REASONING ACCURACY AND TIME COMPLEXITY
Due to the lack of resources, we build CoM-test, a benchmark with CoM reasoning chains on the TextVQA test set based on the proposed data generation pipeline, and also introduce keypoints- aware metric to validate the correctness of reasoning paths (see Appendix EE.3 for detailed statistics). We also evaluate the time complexity for model generation held-out benchmark, MM-Vet.
60.00
CagCaM/ aapearaaar
55.59
332
20.0
332-
17.5
Accuracy Soure
40.00
252
15.0
12.5
30.00
1:24
10.0
130
20.00
130
727
2.5
2.35
22
Training Steps
Figure 6: Left: Results on reasoning testbed CoM-test show CogCoM achieves satisfactory performance with only 70K training data and 2K steps. Right: Results on MM-Vet show that CogCoM produces comprehensive reasoning content without incurring excessive time overhead.
Reasoning Accuracy To validate the correctness of execution and results of manipulations in reasoning paths, we introduce keypoints-aware evaluation metric that concentrates on these contents and their order. Concretely, given predicted chain- pair (C",A') A') and the ground truth pair (C, A). we first extract the keypoints (i.e., the name, parameters, and results of manipulations) in A', A to form two lists, and then discretize these two lists into K' and K based on a bag- of-words composed of keypoints. Then, we calculate the normalized Levenshtein Distance SK Levenshtein(KK K', KK/N as the manipulation score. We also compute the BLEU (Papineni ct al. 2002) score sc = BLEU(C", C') as the paragraph score. Finally, a weighted average of these two scores serves as the ultimate reasoning score acc (0.6 SK 0.4 x sc)
We train our first-stage model only using the 70K automated CoM data without other supervision for qualitatively evaluate the effectiveness of chains, and the results are shown in the left subplot of Figure 6 We find that by training with the CoM chains, our model can swiftly achieve the satisfactory performance of 48.41 accuracy score with 2k training steps, and obtain the optimal result of 55.59 with 8K steps. Additionally, the explanation scores gradually improve along with the model performance, indicating that successful reasoning steps contribute to the achieving of final answer.
Time Complexity We also evaluate the time complexity and average length of tokens during model reasoning on held-out test set. MM-Vet. Specifically, we run CogCoM and the baseline model on all 218 questions, and record the time overhead as well as the average number of outputted tokens (using the Vicuna-7B-vl. tokenizer). We divide the 218 samples into intervals based on the time expenditure for each sample and calculate the average values of the time complexity and the number of tokens for each interval, with the results presented in the right subplot of Figure 6
From the results we find that compared to baseline model, CogCoM produces information-intensie reasoning content (e.g., detection boxes, auxiliary lines) without incurring infeasible time overhead. For example, without quantitive optimization, CogCoM outputs 262.9 informative tokens in approxi- mately seconds. With the advantages in long-context optimization techniques (Hooper et al. 2224). believe that it is crucial for models produce informative content and accurate responses.
8

---

Published as a conference paper at ICLR 2025
5.2 EXPERIMENTS ON VISUAL GROUNDING
The task of visual grounding requires models precisely provide the corresponding coordinates of regions in an image based on the given target description. Following the existing work (Wang al., 2023a), we train our model obtained by the first stage on a mixture of datasets, including an instruction corpus Multilnstruct, high-quality grounded VQA corpus introduced in CogVLM, and the 70K CoM data. This training results in generalist grounding model that is excelling visual grounding while capable of reasoning. For all benchmarks, ask the model to provide grounded coordinates, such as "Where is (expr) answer in 1x0,y0,x1,yl format." ", where the (expr) refers to the target expression. We use the standard metric, that considers a prediction as correct when the intersection-oover-uioo (IoU) between boxes is greater than 0.5.
prompt CogOM in chat manner to
RefCOCO
RefCOCO+
Re(COCOg
Type
Model
val
test-A
test-B
vul
test-A
test-B
vul
test
OFA-L* Wang al
2022b
79.96
83.67
76.39
68.29
76.00
61.75
67.57
67.58
Shikra-7hh Chenn el all 21125a
87.01
90.61
80.24
81.60
87.36
72.12
82.27
82.19
Shikra-
Chen
2023a
87.83
91.11
81.81
82.89
87.79
74.41
82.64
83.16
Generalist
Qwen-VL
[Bai ot 2025
89.36
92.26
85,34
83.12
88,25
77.21
85,58
85.48
CogVI.M
Wang 2t
A123a
92.51
93.95
88.73
87.52
91.81
81.43
89.46
90.09
CogCoM
94.57
89.15
88.19
92.80
82.08
89.32
90.45
Specialist
92.64
94.33
91.46
88.77
92.21
83.23
89.22
89.37
SOTAs
(UNINERT
(UNINEXII
(UNINEXII
(ONE-EECC)
JONE-PEACE
JONE-EACC
(ONE/EACS)
(UNINEXTH
Table 2: Results VG benchmarks, where the specialist SOTAs are quoted from
(Bai al.
2023
Results As shown in Figure CogCoM achieves the best performance in 6 of all 8 sub-sets. Based on the training with mixture of broad capabilities, this result indicates that our model exhibits superior grounding ability while offers potential to solve a variety of tasks. In addition, CogCoM achieves performance par with the specialist SOTAs and surpasses the ONE-PEACE with a leading performance on the test-A from RefCOCO+ This result demonstrates that under a generalized training integrating multiple capabilities, model engages grounding as foundational skill and cultivate the capability to accomplish complex problems.
5.3 EXPERIMENTS ON GENERAL MULTIMODAL EVALUATION AND HALLUCINAATION EXAMINATION
We further examine the general multimodal capabilities, and the hallucination issue. We use the generalist VQA model and obtain model predictions by directly asking the original questions in benchmarks. We use the challenging adversarial version and official evaluation scripts for POPE.
Method
LLM
MM-Vet
POPE..
InstructBLIP (Dai et al.
2023
Vicuna- 13B
25.6
77.3
LLaVA (Lu al. 20236)
LLaMA2-7B
28.1
66.3
DreamL M (Dong
2023
Vicuna-7B
35.9
76.5
Monkey
LLi ct al. 22024
Qwen-7B
36.2
LLaVA-1.5
Liu ct al.
[2023a
Vicuna- 13B
39.4
85.0
CogVLM Wang et
2023a
Vicuna-7B
45.5
87.2
CogCoM
Vicuna-7B
46.1
87.8
Table 3: Evaluation results on the general and hallucination assessment henchmarks.
As shown in Table 3 compared to the baseline model on MM-Vet and achieves the superior performance on POPE which in consistent with the baseline model. This result suggests that out model maintains superior reasoning capabilities while preserving effectiveness in general multimodal tasks, and simultaneously exhibits lower hallucination.
Results
we can that CogCoM improves the performance by 0.6 points

---

Published as a conference paper at ICLR 2025
5.4 ABLATION EXPERIMENT FOR TRAINING w/wo COM DATA
We conduct experiments on our generalist VQA version model CogCoM-chat, where we removed the 70K CoM training data from the training procedure and keep all other settings unchanged (i.e., using the corpus of public VQAs and 500K training data from Multilnstruct, LLaVAR, and ShareGPT4V). The results on three typical benchmarks are shown in Tabld4 We can see that our model benefits significantly from the produced CoM training data on these benchmarks that require detailed recognition, multimodal reasoning and mathematical capabilities to achieve substantial improvements.
Model
Tex
MMVet
Vista
CogCoM (wo)
64.5
45.9
34.8
CogCoM (w)
71.1 (1 6.6)
46.1 (1 0.2)
35.7 (1 0.9)
Table 4: Ablation experimental results for training CogCoM with/without 70K CoM training data.
RELATED WORKS
Most of LVLMss rely on the training publicly available image-caption pairs, including ALIGN (Jia)
ct al., 202I), MSCOCO (Lin et al., 2014), VG Krishna et al. (2017), CC3M Sharma et al. 2018).
CC12M (Changpinyo et al. (2021) 1. SBU (Ordonez et al. 2011 LAION2B (Schuhmann et al. 2022
LAION Schuhmmmmnnann et al. (02T). Starting from Flamingo (Alayrac et al. 2022). a series of
LVLMs have focused on training the adaptation layers align the visual representation to the frozen
LLMs on a mixture of image-text pairs with the above corpus, including BLIP2 Li et al. (2023b), KOSMOS Huang et al. (202bb), and OpenFlamingo (Awadalla et al., (023). Inspired by success of instruction tuning in LLMs (Wang et al. 2022c), a line of works have devoted efforts to build vision- oriented instruction-ansse answer pairs through GPT4 and train models for imitation, such as LLAVA (Liu) ct al. 2023b), Otter (Li et al. 2023a). L. (Wang et al. (023b). Multilnstruct (Xu et al. 2022). Lynx ZZnng et al. 2023). InstructBLIIP (Dai et al.), and StableLLLaV (Li et al. 2023c). Recently, researchers have proven the efficiency of developing LVLMs with two stages of training, the first stage of abundant pretraining on image-caption pairs and the second stage of alignment on image-
question-answer triples, such as PALI (Chen et al.),, PaLI-X (Chen et al. (023c), Qwen-VL (Bai)
ct al. 2023). and CLgVLM Wang et al. 2023a).
To further enhance the ability of LVLMs in solving high-level visual problems, research focusing on various aspects of reasoning is attracting broad attention. We simply divide existing studies into three broad categories. The first line of research focuses enhance train models with a mastery of cross-modal grounded reasoning, where grounded iestructioo-follovvii public visual grounding dataset GPT4-V for training, including KOSMOS-2 (Peng al. 2023), Shikra (Chen et al. 2023a). and GPT4ROI (Zhang et al. 2023a). The second aspect of efforts have been devoted promoting models to understand artificial visual scenes, such as figures, charts, and receipts. These studies includes CogAgent (Hong et al. 2023) and CHARTVE (Huang et al.) (023a)). Some other studies address the crucial problem of hallucination in LVLMs with counterfactual or interpretable reasoning (Yu et al. 2023a; Yin et al. 2023). V* (Wu & Xie 2023) also contributes efforts to enhance the details recognition of VLMS based on the --M-gddded searching process.
supervision is build through
CONCLUSION
This paper studies the problems presented by the conclusive alignment training of VLMs, and proposes mechanism, Chain of Manipulations (CoM), that enables VLMS to solve problems step- by-step by actively manipulating visual inputs as evidence. We realize this methodology by proposing (I) flexible data structure, (2) an efficient data generation framework capable of producing abundant samples, (3) memory-based architecture compatible with existing VLMs, and (4) training process for versatile capabilities. We also annotate 7K graphical math samples with reasoning chains to facilitate the advancement of VLMs in solving mathematical problems. Experiments on 9 public benchmarks show that our trained 17B general VLM can produce informative reasoning content while achieving superior performance diverse multimodal problems.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by National Natural Science Foundation of China (No. 62277033), and National Natural Science Foundation of China (No. 62476150). Thanks the support from National Engineering Laboratory for Cyberlearning and Intelligent Technology, Beijing Key Lab Networked Multimedia, and Zhipu AI. This work is also supported by the Natural Science Foundation of China (NSFC) for Distinguished Young Scholar 62425601, and the Natural Science Foundation of China (NSFC) 62495063.
REFERENCES
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: questions. In AAAI, 2019.
Answering complex counting
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilitics. arXiv preprint, 2023.
Apratim Bhattacharyya, Sunny Panchal, Reza Pourreza, Mingu Lee, Pulkit Madan, and Roland Memisevic. Look, remember and reason: Grounded reasoning in videos with language models. In ICLR.
Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawa- har, and Dimosthenis Karatzas. Scene text visual question answering. In PCVPR, 2019.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset, 2022.
Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang. Dualfocus: Integrating macro and miero perspectives in multi-modal large language models. arXiv preprint, 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modas models with better captions. arXiv preprint, 2023b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lueas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. In ICLR.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint, 2023c.
Wei-Lin Chiang. Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%, chatgpt quality. See htlps://vicuna. Imsys. org (accessed 14 April 2023), 2023.
11

---

Published as a conference paper at ICLR 2025
W Dai, Ji, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung, and S Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint.
W Dai, Li, D Li, AMH Tiong, Zhao, W Wang, B Li, P Fung. and S Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023. arXiv preprint, 2023.
Runpei Dong, Chumrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint, 2023.
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, et al. Pp-oer: A practical ultra lightweight ocr system. arXiv preprint, 2020.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. CVPR, 2023.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint, 2023.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint, 2024.
Yushi Hu, Otilia Streteu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In CVPR, 2024.
Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. Do Ivims understand charts? analyzing and correcting factual errors in chart captioning. arXiv preprint, 2023a.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36, 2023b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Alyssa Hwang, Andrew Head, and Chris Callison- Callison-Burch. Grounded intuition of gpt-vision's abilities with scientific images. arXiv preprint, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In ICML., 2021.
Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing, 1981.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 2017.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. arXiv preprint, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre-training with frozen image encoders and large language models. In ICML. PMLR, 2023b.
Bootstrapping language-image
Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, and Yunchao Wei. Stablellava: Enhanced visual instruction tuning with synthesized image-dialogue data. arXiv preprint, 2023c.
12

---

Published as a conference paper at ICLR 2025
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint, 2023d.
Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In CVPR, 2024.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. ECCV, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint, 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chumyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint, 2023c.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint, 2023.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Jory, and Enamul Hoque. Chartqa: benchmark for question answering about charts with visual and logical reasoning. arXiv preprint, 2022.
R OpenAL. Gpt-4 technical report. arxiv 2303.08774 View in Article, 2023a.
OpenAL. Gpt-4v(ision) system card. Citekey: gptvision., 2023b.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using million captioned photographs. Advances in neural information processing systems, 2011.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018
Noam Shazeer. Glu variants improve transformer. arXiv preprint, 2020.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. Answer them all! toward universal visual question answering models. In CVPR, 2019,
Amanpreet Singh. Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint, 2023a.
13

---

Published as a conference paper at ICLR 2025
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint, 2023b.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 482, 2024.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, arXiv preprint, 2022a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence- torenne-tt--euunnee learning framework. 2022b.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint, 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint, 2022c.
Penghao Wu and Saining Xie. V*; Guided visual search core mechanism in multimodal Ilms. arXiv preprint, 2023.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinstruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint, 2022.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint, 2023.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In ECCV, 2016.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. arXiv preprint, 2023a.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint, 2023b.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guogiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training gpt4-style language model with multimodal inputs? arXiv preprint, 2023.
Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu-Jui Fu, William Yang Wang. Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint, 2024.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint, 2023a.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint, 2023b.
14

---

Published as a conference paper at ICLR 2025
A DISCUSSION WITH CLOSELY RELATED WORKS
The efforts of ViperGPT (Suris et al. 2023) share the same basic idea with our work that decomposes complex visual problems into reasoning steps. In comparison with their training-freee framework combining external VLMs using a code LLM, we focus on training an end-to-end VLM to enhance its visual reasoning ability to solve complex problems. V (Wu & Xie 2023) is concurrent work who also solves problems by progressively acquiring visual cues. Their two-part framework first utilizes a VQALLM model to list visual objects, followed by a dedicated search model to acquire the objects. On the other hand, our approach focuses on using one model to actively perform reasoning and to identify or mark the most useful visual information, which may offer the potential for solving more complex reasoning tasks in addition to the detailed identification, such the challenging geometric math problems. DualFocus (Cao et al. 2024) was released around the same time as ours. They also construct training dataset that includes intermediate cues (bounding boxes) and trained the model with two stages. Compared to their work, our CoM training places more emphasis on answering questions in single reasoning process and marking images to assist in solving complex problems. VPD (Hu et al. (2024) converts programs obtained from LLM and execution engine into CoT and distills them into the VLM, enabling the model to reason when solving visual problems. This approach is similar to our method on visual reasoning. However, our ultimate goal is to train VLMs to solve complex visual problems by actively reasoning (i.e., CoT) and manipulating (c.g.. zooming in or drawing auxiliary lines) images which is consistent with the human behavior in realistic scenarios. VisProg (Gupta & Kembhavi 2023) relies on in-context learning of LLMs to generate programs, which are then executed to get final answers. Ferret-v2 (Zhang et al. 2024) add the Dense Referring and Dense Detection tasks into the training stages and adopts an additional DINO encoder to enhance the fundamental visual capability of VLMs. The study from (Bhattacharyya et al.) trains a VLM to combine the low-level visual features with high-level inferences to reason and generate the final response videos, where the low-level features are introduced by surrogate tasks during training.
B DISCUSSION FOR FUTURE WORKS
Humans solve difficult questions with a period of thinking before answering. In contrast, LLMs/VLMS generate outputs immediately after prompting tokens. The Chain of Thought (CoT) mechanism serves as an effective substitute for this thinking process. Similar to human reasoning, a backtracking mechanism could provide a reliable approach for forming correct and concise reasoning paths. However, as the integration of backtracking results in high time complexity, developing an efficient backtracking strategy that improves the accuracy of reasoning paths without sacrificing performance is a crucial direction for future work.
C LIMITATION AND IMPACT
Though we build a robust framework with remarkable LLMs and reliable visual tools, there are still limitations. First, the diversity of solving steps is still insufficient, and inaccuracies in visual tools (e.g., coarse grounding boxes, OCR errors on slanted text) result in many negative paths, which could be better utilized. We propose improving these issues with dedicated prompts and enhanced visual tools. Second, re-inputting manipulated images with fixed prompts can slow down the process. This can be optimized by incorporating physical manipulations directly into the vector space calculations. This work presents general visual reasoning mechanism that alleviates the problems caused by existing conclusion-alignmenn training for VL.Ms, introduces a data production framework involving I.LMs and visual tools reliable annotators, and devises memory-based compatible VLM architecture. We expect this work to bring three benefits to the community. First, the proposed visual reasoning mechanism may push the progress of VLMs in solving complex visual problems. Second, the introduced data production framework may be applied to widespread training scenarios to promote the development of current data-driven machine learning. Third, we hope that the memory- baaee architecture will be helpful for VLMs in multi-turn long contexts.
15

---

Published as a conference paper at ICLR 2025
D ADDITIONAL ANALYSIS
D.1 THE NUMBER OF INFERENCE TURNS AND THE QUALITY CONTROL
We have detailed the statistics for the training and testing data in the Appendix E.3 including the total number of reasoning chains, the average number of reasoning steps, and the average number of manipulation types. To clarify the average and maximum numbers of the multi-image inference turns, we have also conducted statistics on the number of turns divided by multiple images. The results are as follows: in the training data source from Tex and ST-VQA which may involve generating new images such as through zooming, the average number of turns is 1.42, and the maximum number of turns is (we restrict the maximum number of turns to 4 during training to prevent OOM). In the test set of Tex,VQA, our model produced average of 1.54 turns involving multiple images. is worth noting that not every image requires manipulations such as zooming, and some can be answered through reasoning with evidence direct observation.
During the collection of our 70K CoM training data, we discard the wrongly recognized data (i.e., we refer to these data as the negative paths in our paper), as these data can not terminate to the golden answer node during the DFS traversal. We used this filtering strategy to ensure that only the correct data capable of reaching the golden answer (i.e., positive paths) was included in the 70K training data. Since we recursively search for the answer by following the intermediate path composed of grounding boxes until we reach the leaf answer node, this approach generally produces the correct path, except in a few cases where the grounding boxes may be too large. Most of the data we have constructed relies on grouping results as intermediate reasoning steps. After our manual verification, we found that most of the reasoning paths are indeed correct. Therefore, most of the generated CoM training samples can be guaranteed to be error-free. As using Reinforcement Learning to penalize the negative paths during training is another optimization strategy, we look forward to utilizing these negative paths as negative rewards in future work.
D.2 THE DISTRIBUTION OF THE SUCCESSFUL AND UNSUCCESSFUL PATHS
Since most of the questions in the constructed dataset ask about the details of images, we have compiled the distribution of image classes for the reasoning data sourced from TextVQA (which includes an image classes label). The statistics are now presented in Figure 7 of the revised paper. Specifically, we analyze the distribution of image classes where a successful reasoning path was not obtained using GPT-4 and visual tools, and among those classes, the distribution of image classes where a successful reasoning path was achieved. To facilitate display, we show the top 200 image classes. To improve visualization, we applied 1o to the positive frequency values on the y-axis, excluding zero. We find that: (1) For most image classes, the proportions of successful and unsuccessful paths are approximately consistent, indicating that the image category does not have a significant impact in this problem. (2) The success rate for constructing reasoning paths is higher for common and prominent objects, such as "person" and "vehicle", while the success rate is lower for smaller, less common objects like "necklace" and "kettle." This suggests that the lower success rate for such objects is due to the limitations in the accuracy of the tools used to find the positive path.
3.5
on unsuccessful paths successful paths
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Figure 7: Distributions based on top200 image classes from unsuccessful paths, where the positive frequencies on yaxis are taken as 1-) to facilitate the visualization.
16

---

Published as a conference paper at ICLR 2025
E DETAILS OF DATA PRODUCTION
In this section, we further introduce the details CoM data production, with the overall algorithm of pseudo code, example of the solving steps generation with L.M and corresponding guidelines, an example of the reasoning chains completion with visual tools. We also list the details of data statistics for the synthesized training data as well as the evaluation data of CoM-test, followed by limitation analysis for the current data production method.
E. ALGORITHM FOR THE AUTOMATE DATA GENERATION PIPELINE
We provide the pseudocode of the CoM synthesis algorithm to clearly explain the process of data generation, thereby facilitating understanding and reproduction
Algorithm
Synthesising Chain of Manipulations
M anipulations {fo: + fi E M}
1: Define:
Linguistic Annotator V //We e use GPT4 in this work
Visual Annotator Vy //We use PaddleOCR and Grounding in this work
2: Input: Image I, Question Q, Answer
3: // Linguistic Annotation
4:
Prompt , with guideline pL to generate reasoning steps:
(s (sleps1, sleps2,...)
5 = V2(Q)P"),
where
(3)
steps: = (Fi, desc;)
5: Define tree T
6:
for = 1 to do do
Extract 24, Y: instantiated with f. in step, 8: Extract referential boxes from 2;
7:
9:
for b in B do Leverage V to acquire corresponding visual content y/ (x,,)T,),, and apply y to
10:
T .append())
(4)
11: end for
end for
13: Traverse T to obtain positive chains that leads to given answer with terminal return
[s,, s2.-.. DDF((T(A))
(5)
14:
Return (2,
E.2 THE COM-TEST BENCHMARK AND EVALUATION METRIC
To measure the correctness of CoM chains, we introduce keypoints-aware metric. The intuition that we care about the key elements including actions (i.e., manipulation name), targets (i.e., manipulation input), and visual contents (i.e., manipulation returns) each step the path, as well as the logical execution order of manipulations. Given pair of chain chain-answer annotation (c, a) and corresponding model prediction (c', aa), we first sequentially extract the key elements from e and d to construct two ordered lists, and then replace the elements in the lists with their fixed indices in Bag-of-Elemente = cJ to result lists Levenshtein Distance 8. Levenshteing k') k')/N between the two lists, where N is the maximum length between and k'. We adopt this simple discretization strategy with low time complexity to concentrate the key points well the solving order. We further consider the linguistic matching of paragraphs by calculating the BLEU (Papineni et al.] (2022) score between two chains Sp BLE c'), and the final score is weighted combination
and k' We thus calculate the score as the normalized
(0.6 8 0.4 x 8/)
17

---

Published as a conference paper at ICLR 2025
E.3 DATA STATISTICS
We develop strategy to extract predicate phrases based constituency parsing with StandordCoreNL.P in which extract verb, conjunetion-conmeeeed verb phrase, preposition-conneceed verb phrases.
Besides the standard CoM data incorporating manipulations with explicit visual evidence, the proposed data synthesizing framework is compatible of producing implicit visual reasoning steps step, (desc,) without involving the manipulations. We thereby also build this partial CoM data on the corpus consisting of absurd visual questions (i.e., asking unanswerable questions based on the given image) to further resist the toxic ballucinations. Specifically, given an image I with a question Q. we prompt GPT-4V (OpenAl 2023b) solve the question step-by-step acquire the reasoning chains.
Data Source
#QAs
#Chains
#Steps//hain
#Manipulations Types/Chain
VA (itenn 2019
10782
13766
2.93
2.41
ST-VQA (Singh al.
2019
4814
3959
2.88
2.43
TDIUC-count (Shrestha et al.
2019)
53547
54523
2.35
0.74
TDIUC-absurd Shrestha et al.
2019
11677
11677
4.09
CoM-test
4609
8612
3.26
Table 5: Detailed statistics the the training data and evaluation data synthesised with CoM production.
0.06
0.00
0.01
0.00
Figure Distribution of the top-50 generated manipulations out of total 465 based on 4-shot prompting, where the first three bars scaled with 20% for smooth visualization of all data.
18

---

Published as a conference paper at ICLR 2025
Wannow
OGR Nottett
nude IoE
DCR manpuutton,
recopmized
cacustt manpulation
spectted by tarper lat image. and retorn
groundng wobb
grovniding manduuatio locates the obsectto specelld sageet phrase comere and
reauing counddig houpees
by top-lett
coorsinatcs.
and zoomon wtt.e/ooooooooeeeeeeeeeeeo j
Hit crop and zoomin manquuotton whet usets to identiyy smotl and subte otjects in image, that frer craps the coneent inago using the box BBb padkied by the tip loft and botton right coordiarrss and than acon do crapped image by two timer trusty return the resuting image ing
sew at vicsal assiwart capabble gaosealing soling waes for ieggeetentee vleak questions to each aleps you can optiosully use Mannipuiitionn to aperaio the image, which can be used acquine speciil information from mhe image or acquine the processed new vnago (pecase awore these manpulations wi actuall perforeed what you gonerate soling stops), The manpuiation be from the prederned ones, can be now one create yoursat (shood there indeed be need, where the predelined manpulttonss with thee acising
diacrpptioee
MoAMMAN
image plaase geowonte
alaps, where the oudput of each smep a apple consiating a Marisalaion floave to foce the ournett alap dowant invole any manqwlation) obtal
Desoiation: Manioclation the
mangulation
information
Descrition.
wheen is centence decoribing the ournet sovving step in Please oshere the following format given on ippot of 10: xox The essentia sotving (Manipulaion Desolasion) 'iin There
No (Manipulaton Desolasion) (DEMKNNSTATTOMSSI (SAVESTION)) The escenbal cotving stope are:
Maripulations Definition and Linguistic Annotation Guideline
INPUTS
Question:
tuikinng
Step 1o groundingg 1 tuiddingt--bbl..1
building image bounding the building
(grounding 2lrumber the building 'bbx 11obbxx
Idbertily police building
number "bbx 2",
Step 3: (OCR) trumber region toba -xtr_11
Rlozggiiee number the bito and retumn recogniied number lal.
Question: What rumber polcco
building?
Answer: 43
OUT Sa
btx
Bbx
Atrosshos
Locate the bu the image and retum the bounding box the builinng as Tbbx 1";
kientlly the number of the palice station on the building box brx If return the bounding the number "box
Recognize the number rogian "bbx_2 and nelurn the retoognized number tat
An Example Figure 9: An example shows the configuration, inputs, outputs of the linguistic annotation and visual annotation.
the lingulstic annotation results
annotation results
E.4 DETAILS OF THE LINGUISTIC/VISUAL ANNOTATIONS
In this work, we adopt the GPT4-turbo as the linguistic annotator for generating gobblems-solvingg steps, and the API call was conducted during the period of 2023.9-2023.1.. For the visual annotators. we leverage the currently best- best-performing tools, GroundingDINO and PaddleOCR, to acquire all visual contents requested by the manipulations. For clear description for the production setting and results, we illustrate the guiding prompt, and example-based linguistic annotation results as well as the visual annotation results in Figure 9
E.5 LIMITATION ANALYSIS FOR THE DATA PRODUCTION
For the implemented data framework, we engage the remarkable LLM to provide basic solving steps, adopt two reliable visual tools (i.e., GroundingDINO and PaddleOCR) to acquire corresponding visual contents, and then perform the traversal to achieve feasible reasoning paths, which ensures the correctness and robustness of data synthesizing. However, we also find that there are three major limitations caused by the employed models and could be improved in future:
19

---

region on Published as a conference paper at ICLR 2025
The lack of diversity in linguistic reasoning steps. The 5-shot prompting to the GPT-4 gains a stable solving steps, but it also results in the descriptions for executing manipulations or general thinking similar. We suggest that this can be addressed by employing diversified prompts or requirements.
The inaccuracy of visual tools. We find that there a considerable amount of negative paths caused by the failures of visual tools, such as the rough granularity of bounding boxes and the error recognition of slated letters long sentences. This issue can be relieved by improving the semantic understanding capabilities of visual tools.
F DETAILS OF TRAINING
F.1 LAUNCHING PROMPTS
Please solve the problem gradually via chain of manipulations, where in each step you can selectively adopt one of the following manipulations GROUNDING(a pbrase)--oxes, OCR(an image region)->texts, CROP_AND ZOOMIN(a region given image) manipulation, if that seems helpful. {QUESTION)
CALCULATE(a computable target) target)--numeen or invent a new
Please tackle given question in a stepbystep manner. For each step one of the following manipulations (depicted - phrase)->boxes OCR(an (an image or a region)->texts,, ttetts, CROP LLL_ADD2OOON given image)-new Limage. CALCULATE(a computable -nummbes,,,,,,, or develop new manipulation yourself (if it is indeed required). {QUESTION]
can be optionally used: GROUNDING(a
Please go through the question incrementally with chain of manipulations (optionally use manipulation when needed) such as GROUNDING(a phrase) OCR(an image or a region)->txxts CROP_AND ZOOMIN(a region on given image)->new aamae,, CAL- CULATE(a computable (onget)-nuumbers, and create new manipulation if necessary. {QUESTION)
F.2
TRAINING SETTINGS
Parameters
Stage1-1
Statel-2
Stage-2
Hardware Environment
3,840 A100xdays
256 A100xdays
160 A100xdays
Objective
next token prediction 1.5B
next token prediction
next token prediction
Images
40M
576K
Batch size
8192
1024
160
Iterations
120,000
60000
14000
Optimizer
AdamW le-4
Adam W
AdamW
Learning rate
lc-5
le-5
Warm up steps
7200
1200
280
Trainable weights
6.5B visual expert
6.5B visual expert
6.5B visual expert
Table 6: Training details of all stages.
G DETAILS OF QUALITATIVE ANALYSIS
G.1 QUALITATIVE ANALYSIS
We investigate the capabilities of CogCoM on scenarios that requires different types of detailed reasoning, including recognizing textual details, reading time, understanding charts and counting objects. The results are shown in Figure The first case demonstrates that CogCoM finds the region corresponding to the plane logo through two steps of grounding and then achieves the answer based on zooming in the cropped region. The second case illustrates the ability of CogCoM in reading time, by locating the device that displays time and then transforming the time into words based on the read timne manipulation. In the fourth example, CogCoM first identifies all visible truck wheels through grounding, and then leverages the counting manipulation consolidate the total number.
20

---

Published as a conference paper at ICLR 2025
G.2
REPRESENTATIVE CASES
TRAFALGAR
CoM:
Given a question, please salve the question step by step with chain of munipulations where each seep you can optionalyy use one of the following manipulations (defined as Namet Impurt- Returr))) on the image: GROUNDINCI phrnse) oboses, OCRian image region- CROP_AN AND 200MENG given imtage) inew_image CALCULATER computable target) ionumbers,
manipulation created by yourself
Question: What written on the trffee sign near the bus?
Model: Using CROUNDING((h traflic sign nour the bus) locate the traflie sign neur the bus image, which (1084,755 127,926, (676 (46,6444,4400 7.0.7 Unilizing OCRppets region 11084,157 926 926,2031) to read the sexts writeen on the traffic sign within region [[084,157 922,7031).
resulting Taxi, So ultimately, the conclusive snswer to the question discussioon is Taxi.
Visual Grounding:
Queszion: Find the region image that "a traffic sign' describes.
Model:: (1,,,14,739922000
Grounded Captioning:
Question: The goal of this task generate description for the image. The specifted by (178,145,739.29))
Model: Amm-int--igg
Image Captiuning:
Question: this task, you are given an image and you will need to generute some test to describe it. Plense give me a concise answer.
Answer: Taxi bus with minbow colors side.
Malti Choice:
Question: Select color does appear minhow sign (1) yellow, pink, (4) black, give dirrcct
Answer: black
Detuiled Captioning:
Question Esplain the visual content of the imagee in great devail.
city where white with red and blue parked. The bus beans the text "Trafalgar' bold red letters acroes its side, indicating it's likely part of tourist travel compuny. The bus positioned next to traflic sign that reads "TAXI suggesting might be waiting for passengers or ready to embark on journey. In the background, there are buildings and trees, adding depth to the urban landscape. The overall and
Answer:
The
captures
impression one typical day
setting, with wailing passengers
city
Figure 10: CogCoM demonstrates flexible capabilities for adapting different multimodal scenarios, including evidential visual reasoning, Visual Grounding, Grounded Captioning, Image Captioning, Multi Choice, and Detailed Captioning.
21


Reviews:
[{"review_id": "HYKPBsF6dK", "reviewer": "Reviewer_sBKx", "summary": "The paper focuses on visual reasoning problems using large vision language models. The paper focuses on the problem caused by the first alignment training stage in such vision language models, where the model does not focus on the low level details and instead focuses only on the high level details of the image. The paper proposes to solve this problem by teaching the model to perform intrinsic manipulations (e.g., grounding, zoom in) with results (e.g., boxes, image). The paper shows promising performance on various visual reasoning problems.", "questions": "* The paper should include a broader discussion of prior work.\n* The paper should include more extensive ablations.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper shows promising results.\n* The paper is well written and easy to understand.\n* The paper includes extensive qualitative examples.\n* The proposed CoM dataset has the potential to be useful to the broader community.", "weaknesses": "* Novelty: The core idea of the paper is very similar to that of \"Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models, CVPR 2024\" and \"Visual Programming: Compositional Visual Reasoning Without Training, CVPR 2023\". The Visual Program Distillation work trains the VLM to perform several grounding tasks for more accurate visual reasoning. Somewhat similarly, \"Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models, arXiv 2024\" uses a visual sampler to extract relevant regions in the image.\n\n* On the other hand, prior work such as \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" used surrogate tasks to extract low level visual information. A discussion of the relationship to prior work is necessary.\n\n* Limited size of the CoM dataset: The proposed CoM dataset consist of only 6k images. This might limit its utility due to the lack of visual diversity.\n\n* Ablation studies: The proposed method uses several datasets during the second stage of training along with the proposed CoM data -- this includes MultiInstruct (Xu et al., 2022), LLaVAR (Zhang et al., 2023b), and ShareGPT4V (Chen et al., 2023c). It would be beneficial to include additional ablations to highlight the utility of each of these datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQHlqJCWc0", "reviewer": "Reviewer_pnHj", "summary": "This paper introduces Chain of Manipulations (CoM) to address the reasoning questions that require a detailed look of the visual information. CoM is a step-by-step strategy allowing the model to take manipulation operations like Crop/ZoomIn over the image, thus reserving more capacity for the image regions/bboxes that needs to be looked at with more detail. The contributions include both two datasets and a trained model, which shows SoTA performance on 9 datasets in 4 categories.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method, defining operations of image manipulation to solve queries that require a detailed look is intuitive, is intuitive and clear.\n2. Intensive experiment results compared with up-to-date models are shown on various tasks, leading to SoTA performance.\n3. writing is clear and easy to follow", "weaknesses": "1. One thing not very clear - in inference time, once the model is trained, is step-by-step CoM still needed? In other words, is CoM a method for collecting training data, or is it for VLM inference as well? Fig-6 right breaks down the questions into eight groups based on the time overhead - is this time the inference time using one single VLM call or multiple calls using CoM is needed?\n2. In principle, how does this work compare to visual programming [1,2]? Is the defined manipulations like ZoomIn/Crop a specific case of the operations as in these works? More discussions will be helpful.\n3. While the VQA results show huge improvement, the improvements on grounding on RefCOCO(+/g) are not as significant. Any discussions/analysis on this?\n4. The proposed data contains 70k automatically generated data and 7k human-annotated data. How does each of them contribute to the model performance? For example, are there results using each type of the data separately? Or are there more discussions on why both types are needed and only one type is not sufficient?\n[1] Gupta, Tanmay, and Aniruddha Kembhavi. \"Visual programming: Compositional visual reasoning without training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2] Surís, Dídac, Sachit Menon, and Carl Vondrick. \"Vipergpt: Visual inference via python execution for reasoning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ydmGZoEBDe", "reviewer": "Reviewer_NibN", "summary": "This paper presents CogCoM, a model that can reason in a multi-turn way when answering VQA questions. The paper first introduces an approach to collect multi-turn data given only input-output (single-turn) VQA pairs, by using GPT-4, and then uses this multi-turn pseudo-labeled data to finetune a VLM, giving this VLM the capability of reasoning in a multi-turn way. This results in better results in a variety of benchmarks.", "questions": "- Do the math problems also benefits from the same \"modules\"? (cropping, detection...), or new ones were identified by human annotators?\n\n- Is there any analysis on the distribution of the correct paths found during the data collection process? If only 35%, do those 35% belong to the same category of questions? Maybe there are entire categories of questions that GPT-4 just never solves, and that end up not being part of the training data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1 - The paper's contributions are very sound. The idea of collecting data automatically, but having a way to pseudo-verify its correctness (at least end-to-end correctness) is reasonable.\n\n2 - The paper shows good results across a large variety of benchmarks.\n\n3 - The paper additionally contributes extra human-annotated data for mathematical visual reasoning, which can be useful for the community.", "weaknesses": "1 - From a scientific point of view, it is hard to tell how significant the contributions are. There is one ablation (which is informative) where they remove the multi-turn training data from their training mix, and it seems like it helps overall (but very little for two out of the three tasks shown). However, it is unclear how much of the contribution comes from the single-turn vs. multi-turn setting, or the data, or the combination of both, or just any other modeling decision when training the final model. If I am not mistaken, the results that are shown from previous methods are all single-turn, which makes the comparisons unfair. \n\n2 - Motivation. There are no comparisons to the module-based approach (ViperGPT-like) that is actually used to collect the pseudo-annotations. If this approach is good to collect the pseudo data, why would it not be used at inference time? Would it perform worse? Is it slower or more expensive?\n\n3 - Presentation\n\n  - There are many grammar and orthography mistakes.\n  - The related work is only presented in the appendix. This makes it harder to understand the specific contributions of this paper.\n\nMinor weaknesses:\n\n4 - It is mentioned that the positive paths \"are guaranteed to be error free\". This is not necessarily true. The model could get to the correct answer through a wrong path, or even by having the last reasoning step completely ignore all previous information. \n\n\n\n- Many related literature not mentioned? Or compared\nRelated work should not go all of it in the appendix!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ctd0vd57u", "reviewer": "Reviewer_qQQK", "summary": "The authors provide CogCoM, a technique that augments VLMs with ‘chain-of-manipulation (CoM)’ reasoning. In CoM, given image and text input, VLMs generate visual programs that evoke 6 external modules that can perform OCR, grounding, counting, calculating, zooming-in, and drawing a line. The outputs of external modules are composed into the reasoning path of VLMs. The authors automatically collect 70K training annotations for CogCoM on TextVQA/ST-VQA/TDIUC datasets, by prompting GPT-4 to use GroundingDINO and PaddleOCR, and also collect 7K training annotations for geometry problems with human annotation. The authors compares CogVLM augmented with CogCoM to other VLMs as well as CogVLM in VQA benchmarks (GQA/TallyVQA/TextVQA/ST-VQA), a proposed benchmark for CoM reasoning chain (CoM-test), visual grounding (RefCOCO, RefCOCO+, RefCOCOg), and other general benchmarks (MM-Vet, POPE). CogCoM improves CogVLM performance in many benchmarks, especially on CoM-test.", "questions": "- In Fig 2., should the “Large Language Model” be the “Vision-Language Model”?\n- In the title, should ‘visual’ be ‘vision’ for consistency?\n- Have you also experimented with adding CogCoM to different VLMs? Do you have experiment results showing whether the base VLMs already need to be strong to generate CoM, or can weak VLMs benefit from CoM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Introduction of CogCoM, a technique that teaches VLMs to generate visual programs that execute visual expert modules to augment the reasoning chain of VLMs.\n- CogCoM improves CogVLM performance in multiple benchmarks.\n- Code/Model/Data will be publicly available.", "weaknesses": "- **No comparisons with other visual programming methods.** There have been many works where LM or VLM generate programs to execute visual experts for different visual reasoning tasks. Earlier works include VisProg (https://arxiv.org/abs/2211.11559) / ViperGPT (https://arxiv.org/abs/2303.08128). More recent works include GPT4Tools (https://arxiv.org/abs/2305.18752), LLaVA-Plus (https://arxiv.org/abs/2311.05437), Visual Sketchpad (https://arxiv.org/abs/2406.09403) and V* (https://arxiv.org/abs/2312.14135). As generating programs with visual experts itself is not new, qualitative/quantitative comparisons would make the contribution/usefulness of CogCoM more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)