ScaleCap: Scalable Image Captioning via Dual-Modality Debiasing
Authors: Long Xing
Qidong Huang
Xiaoyi Dong
Pan Zhang
Yuhang Zang
Yuhang Cao
Jinsong Li
Shuangrui Ding
Weiming Zhang
Nenghai Yu
Jiaqi Wang
Feng Wu
Dahua Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision Language Model
Image Caption
Primary area: foundation or frontier models, including LLMs
TL;DR: We present ScaleCap, a scalable image captioning framework.

Abstract: This paper presents ScaleCap, a scalable image captioning strategy that generates
comprehensive and detailed image captions. The key challenges of high-quality
image captioning lie in the inherent biases of LVLMs: multimodal bias resulting in
imbalanced descriptive granularity, offering detailed accounts of some elements
while merely skimming over others; linguistic bias leading to hallucinated de-
scriptions of non-existent objects. To address these issues, we propose a scalable
debiased captioning strategy, which continuously enriches and calibrates the caption
with increased inference budget. Specifically, we propose two novel components:
heuristic question answering and contrastive sentence rating. The former generates
content-specific questions based on the image and answers them to progressively
inject relevant information into the caption. The latter employs sentence-level
offline contrastive decoding to effectively identify and eliminate hallucinations
caused by linguistic biases. With increased inference cost, more heuristic questions
are raised by ScaleCap to progressively capture additional visual details, generating
captions that are more accurate, balanced, and informative. Extensive modality
alignment experiments demonstrate the effectiveness of ScaleCap. Annotating
450K images with ScaleCap and using them for LVLM pretraining leads to consis-
tent performance gains across 11 widely used benchmarks. Furthermore, ScaleCap
showcases superb richness and fidelity of generated captions with two additional
tasks: replacing images with captions in VQA task, and reconstructing images
from captions to assess semantic coverage.

Published as a conference paper at ICLR 2026
SCALECAP: SCALABLE IMAGE CAPTIONING VIA DUAL- MODALITY DEBIASING
Long Xing 2: Qidong Huang'; ; Xiaoyi Dong Pan Zhang?, Yuhang Zang', Yuhang Cao?, Jinsong Lia33 Shuangrui Dinga.s 2.3 Weiming Zhang', Nenghai Yu², Jiqqi Wanga.,, Feng Wu', Dahua 2in
l University of Science and Technology of China, Shanghai Artificial Intelligence Laboratory, The Chinese University of Hong Kong, "CPIl under InnoHK (xing lxing_longe, hqd0037a mmall. ustc, edu.cn
ABSTRACT
This paper presents ScaleCap, a scalable image captioning strategy that generates comprehensive and detailed image captions. The key challenges of high-quality image captioning lie in the inherent biases of LVLMs: multimodal bias result- ing in imbalanced descriptive granularity. offering detailed accounts of some elements while merely skimming over others; linguistic bias leading to halluci- nated descriptions of non-existent objects. To address these issues, we propose a scalable debiased captioning strategy, which continuously enriches and calibrates the caption with increased inference budget. Specifically, we propose two novel components: heuristic question answering and contrastive sentence rating. The former generates content-specifie questions based on the image and answers them progressively inject relevant information into the caption. The latter employs sentence-level offline contrastive decoding to effectively identify and eliminate hallucinations caused by linguistic biases. With increased inference cost, heuristic questions are raised by ScaleCap to progressively capture additional visual details, generating captions that are more accurate, balanced, and informa- tive. Extensive modality alignment experiments demonstrate the effectiveness of ScaleCap. Annotating 450K images with ScaleCap and using them for LVLM pretraining leads to consistent performance gains across 11 widely used bench- marks. Furthermore, ScaleCap showeases superb richness and fidelity generated captions with two additional tasks: replacing images with captions in VQA task, and reconstructing images from captions to assess semantic coverage. Code is available at https:://// git.b..tthuuuug.. com/ com//ooperr52//5aaeeesa
INTRODUCTION
In the realm of large vision language models (LVLMs) (Bai et al. 2025 2023 OpenAl 2023a bj
Liu et al.) 2023 Dai et al. 2023). the quality and richness of image-text pairs play pivotal role in determining the effectiveness of pre-training (Chen et al. 2023 Liu et al. 2023 Li et al. 2022).
Particularly, longer and more descriptive captions have shown increasing importance in supporting fine-grained language alignment, moving from carly-stage captions with only a few generic
words (Chen et al.] 2015 Li et al. 2022) to recent efforts that generate paragraph-level, context- rich descriptions (Chen et al. 2023 Sun et al., 2024). As the field pushes toward building ever
more capable foundation models (Meta 2025 Grattafiori et al.) 2024 Bai al. 2023 Touvron et al., (203),, the need for vast quantities of high-quality multimodal data becomes increasingly
urgent. However, relying on human annotation (Xu et al.) 20224 Urbanek et al. 2024 or proprietary
APIs (Dong et al.) 2024 Chen et al. (023) to produce such detailed captions proves prohibitively expensive and fundamentally non-scalable. This challenge has spurred growing interest in developing scalable captioning strategies based on open-source LVLMs, offering a more cost-effective and flexible path for constructing large-scale, high-quality captions.
"Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
ScaleCap: O
The inage portrays o neticalously designed, madern home office setup that exudes o minimolist aesthetic, charasterizd ciean lines and functional elegance... The fisoring comprised of light-colored weed planks. amonged neatly poraliel office double-desk artfulyy placed, cading touch in mnnmlist design, offering arrple workspace for two indriduals
space, wooden fleor. heart this conterpiace that defines the room's is finctionalty The desks white, deekk, and
lending wore and netural
Two black chairs, with a unique wever design are positioneed front the desks, prosiding orgonanic secting for the uxens Theae chairs festure sturdy metal froms and a comfortsble waven seat, designed wupprt lang hours of coptiaating
desktop computer iountainouu londscape with tronqui bie that can inspine creativity and re eeoortion. The keyboard and mouse are placed commnientlyy front of the maniter. neady for use A mail petted plant. liely succulent or soo ferm oods a touch of greentry to the left side of the desli, enhancing the reon's 's air qudiity and be bringing a bit of nature indoors. On the right desk, a white laptop is possisned, its scneon showing a vbront and colorful inage, possibly o photogroph or graphic design project
up, complete
keyboord, offening
view
right nulti tiered, and offers compariments and shelves for organizing and staring office supplies.... The poster above the left desk features block and whine image a womon holding surfboond eveking cence adventure and sportonanehip, The paster above the right desk depicts dundation black and white, syrbolizing realience and the beautty of nature. A anail white ceranic cip is placed on the right dids the right desk, perhope holding o beveroge or serving os o pan holder, eddng a practioal yet subric detai the workdpese.... ...
black pristur,
placed
sightly
dask setup,
casy
for printing
centrol shelf
Detailed Description
Not detoiled Description
Mive2-VL-72B- 2
O
6PT !
@
O
The depicts medern, minnalist office The reom features light greon woll and light weoden Horning creating bright
The depicts a medern home office setup with a clean and organized aesthetic. Two white desks ore erronged in on L-shope, cornected by sheiving unitt the conter. Tws black, ninmalist chairs whth a wirefrons des gn are ploced in front of soch desk
ohnosphere...
There are two white desks posín oned side by sids, forming on L-shope. The clsk on the left has desktop computer setup, including a maniter disploy nig scenic landecops, a keyboord, and mouse dissloyinng o different inage, and a cup coffee proced nexi the
desk on right laptop open.
landscape. desk, there's all-in-ane desitop computer diesloying a small potted
Thes brock chairs
flowars, Canon right desk, there's laptap shelving
unqus, bockneet. providing contensenory look. there white shelvnn unit with three shelves. The top shelf holds a black printer. The niddle shelf has a Ten basks....
design
placed the plant on the left desk A white cup on the right desk
the derks.
Twos fromed pictures one hung on the light green woll. The left picture shows o person holding a surfboord doove head of the beoch. right sicture and dondslion lolened weod hiles, potterred my particll visble the foreground
picture an hear, potted plant has light colored rup with subtlc patteen, adsing
shows person from behind, holding surfboard abeve their greanery
light-
edding
to workspace. texture room
Figure I: Comparison between the captions generated by our ScaleCap and those produced by other advanced VLMSs. The parts of the caption that are bolded refer to the detailed descriptions of the object, while the parts that do not mention the target object are included in the ellipsis.
Despite growing interest, open-source LVLMs still generate suboptimal captions due two intrinsic biases: multimodal and linguistic. First, multimodal datasets often contain imbalanced annotations, causing models to over-desrribe certain objects while glossing over others, leading to inconsistent granularity and reduced caption completeness. Second, inheriting language habits from LLMs (Leng) jtt al. 2024; Liu al.] 2024c). LVLMs tend to favor generic phrasing and frequent co-occurrence patterns, resulting in visual hallucinations: descriptions of non-existent objects or attributes that misrepresent the image. Together, these biases hinder the generation of high-quality, faithful, and fully detailed captions.
2024) have explored the use of To mitigate these limitations, recent efforts (Li et al. 2024 Sun et al.
auxiliary tools or expert modules, such as object detectors (Fang et al. 2023) or image taggers (Zhang et al. 2024) to enrich captions or reduce hallucinations (Yin et al. 2024). While these designs can offer targeted improvements, the overall caption quality is ultimately bound by the precision and coverage of the supporting tools. Given the combinatorial diversity of real-world objects and their attributes, it is unrealistic to rely on handcrafted or category-specific modules as a general solution. These tool-dependent approaches thus fall short in achieving the breadth and adaptability required for generating truly comprehensive and scalable image descriptions.
In contrast to tool-based approaches, we argue that general-purpose LVLMs already possess sufficient perceptual capacity for rich captioning-iff guided properly. In particular, we observe that the lack of detail is not necessarily due to insufficient visual understanding but rather stems from suboptimal information extraction during generation. As shown in Figure. 2 when we explicitly ask for more details about an object that is only roughly described in the original caption, the model can provide precise descriptions. Notably, this perceptual capacity is not confined to large models. We find that even compact LVLMs with only 7B parameters can match the descriptive quality of much larger models when equipped with the right prompting. This observation highlights a promising and cost-effective path toward scalable caption generation: leveraging smaller models with proper guidance, rather than relying on brute-force scaling.
guided
Motivated by this insight, we propose ScaleCap, scalable debiasing strategy that stimulates the model to revisit and refine the caption through a structured, recurrent process. ScaleCap contains two complementary components: heuristic question answering and contrastive sentence rating. The first.
2

---

Published as a conference paper at ICLR 2026
Stastistical Results
728 denception
The image dep a scene from on onimated show
Miss the detaid
image
food
background includes televston stand, curtain..
there are Several plates of feed
detsil
what be type of bread pastry, some of which appear be Topped with jan butter
Generste simple queaition dbout food
Find out of abjects missing detailed rescrppions.
Initial Caption
Describe more detals about the food.
Orgarize Instructions
There are feee plates of bread scottorne on the tobls, with some butter an surface, making very
The food the tcble includes sesenal pieces of what appears a tyse bread or pastryy some appxar toppecl butter
Describe details about Xox
728
and 728 L.MMe perception capabikiiis
98% the now rosponses previde the objects detalle
The reason for certain object detail omissions in LVLM captions is mainly due to the absence of guiding heuristic questions rather than insufficient perceptual capability. We also observe that 7B and 72B LVLMs exhibit similar perceptual capabilities.
Figure 2:
component prompts general-purpose LLM to generate content-specific follow-up questions based on an initially generated caption. These questions target under-deseribed or ambiguous elements, such object attributes spatial relations, and are then answered by the LVLM to progressively inject additional visual details into the caption. This iterative aacstion-answering loop enables a scalable enrichment process, allowing the model to uncover increasingly fine-grained details with each refinement round. To ensure the factuality and fluency of the evolving caption, both the initial caption and its subsequent refinements are evaluated by the contrastive sentence rating module. This second component addresses hallucinations through an offline sentence-level contrastive decoding strategy, avoiding the coherence issues commonly seen in online decoding. Candidate sentences are generated independently and scored to identify high-quality visually grounded variants, ensuring that the final caption is not only rich in detail but also coherent and faithful to the image.
We comprehensively evaluate the effectiveness of ScaleCap across three complementary settings. First and most critically, we use ScaleCap to annotate a large-scale dataset of 450K images and apply it to pretrain multiple LVLM architectures. Across all settings, models trained with ScaleCap consistently achieve the best performance on 11 widely used multimodal benchmarks, demonstrating its broad applicability and pretraining benefits. Second, under the Prism framework, we assess caption informativeness via downstream performance and find that ScaleCap- outperforms even larger models like Swen2-VL-722 - highlighting the efficiency of our strategy. Finally, we evaluate semantic coverage through image reconstruction. showing that ScaleCap captions better preserve visual content than existing open-source models and GPT-4o. These results collectively demonstrate that the captions generated by our approach are highly informative and accurate.
Qwen2-VL-7B
2 SCALECAP
In this section, we first introduce the details of the proposed scalable captioning pipeline ScaleCap, then present ScalcCap-450K, large-scale, high-quality dataset constructed using ScaleCap.
The overall pipeline of ScaleCap is illustrated in Figure 3 which integrates heuristic question answering and contrastive sentence rating in a scalable generaatoon-effiemmnt framework. Specifically, given an input image, the model is prompted to generate an initial caption, and the contrastive sentence rating module applied to extract high-quality sentences from it. We denote it as the "golden sentences", which is the starting point of the ScaleCap. Building upon these golden sentences, the heuristic question answering module generates series of content-relevant questions to explore additional visual details. Each answer is then evaluated by the contrastive sentence rating module to filter out hallucinated or low-quality content. As more questions are raised, the caption is progressively enriched with finer-grained and more balanced descriptions. At last, use capable LLM to integrate the complex and abundant visual information into a complete and structured image caption. To manage inference overhead, the process is governed by a pre-defined scale budget, which limits the maximum number of questions that can be asked. ScaleCap's 's scalable refinement strategy flcxibly balances caption quality and computational cost, producing informative and faithful outputs.
3

---

Published as a conference paper at ICLR 2026
Scale Caption
Contrastive Sentence Rating
Initial Caption
Frobability Contrest
The image depicts
The bockground shons typical desert scene...
Token probabllity
Ratlng and Filter
LLM
forward
The imag depicte Mhitee
M Golden Sentences
image temited...
rEER car..
UMLM
background
Keep sentences without hallacinations,
longe rear wing. which typical for race No...
TThe car festures
probebiity iity
Typical
Heuristic Question
Object Instructions
Dongi Coornge d
Coption Scaling
Go iden Sentences
Final Caption
Reting Filter
Describe details about the 2. Describa more details about the herdights
The race car di..
Object Detail Summerizztion
The image portnoyd
The hecdigghss
imited..
Golden Sentences
Position Instructions
Number of Znste
The can is edomed array spensor logon it's affiiiiion...
1. Describe more details about the pastiion of the cor. Describe detoils position hoodlights.
Adjust N trade off the richneess of image detail and cost
Soldin Sentenoes
Reting Position betail Summerization
The posthion
and Fileer
heodlights
Figure 3: Overview of ScaleCap. ScaleCap is composed of two synergistic parts: heuristic question answering and contrastive sentence rating. The first module utilizes general-purposs LLM to create guiding questions, and the second module addresses hallucinations by an offline contrastive strategy.
2.1 HRURISTIC QUESTION ANSWERING MODULE
Heuristic question raising, To extract richer object- and position-level details, we generate sim- ple, structured instructions (c.g.. "Describe more details about the airplane") based on golden sentences. We use powerful LLM Mr in an in-context manner for instruction generation. Let Sa = {S1. S2, ................................ S denote the golden sentence set. For each Sk. we construct an object prompt Tia with in-context examples to guide the LLM in generating a set of object-related instructions I 1. 1h ...2..... I A. where each 1h queries additional details about an object in Sg. Together,
1 covers all objects mentioned in Sg. This process can be mathematically represented as
1h = -Mi(Tia, k= 1,2..... 1,
Likewise, we generate positional instructions to capture the spatial relationships between objects and the overall image layout. Building on the object instruction set lo, we construct the position instruction set Ip by simply adding a position-specifie prefix to each object instruction, yielding prompts like "Describe more details about the position of the airplane."
For the full set of golden sentences SG = (S1, ................................ Sy), generate both object and position instructions per sentence, forming Io = U 1 and Ip Ui 1.. Together, these instruction sets support a comprehensive understanding of object appearances and their position. To manage inference overhead, the process is governed by a pre-defined scale budget N, which limits the maximum number of object and position instructions. In the following experiments, unless otherwise specified, N is generally set to large value include all instructions.
Efficient visual answering. As discussed in Prism QQia et al. 2024). LVLMs exhibit comparable perceptual capabilities across scales, with differences mainly in reasoning. Since the constructed J. and I are straightforward and require minimal reasoning, a small-scale LVLM can effectively handle these instructions. Thus, we directly apply Io and Ip to a lightweight LVLM to extract fine-grained image details low cost. Formally, for any instruction 1 the object-specific details are obtained as DR = MM (I, 18)), where both the image I and instruction I& are input to the model. By processing T, and I, collect object details D,, and position details D,p-
4

---

Published as a conference paper at ICLR 2026
Data Processing and Analysis
LAION 58
ShareGPT4V
ScoleCop-40Ok
Average Character Length of Coptions
2000
Our caption has the longest longthi
Fittering
(d) Baskktion (orpCorp
Images
1000
311
moskrinn
Images
ScaleCop
10C0
The aeple
Maderste
(c) Diverse dote scurces
(6 Godaen Sertanc
(b) Aponopriate richnees image
resckation
200k Images
(b) Casczaing Caprion Amplricchasn
ShargPTey Dessefutie ScaleCap
Figure 4: Data processing and analysis. During the image collecting and processing stage, we primarily focus on the diversity and richness of image content. In the resulting ScaleCap-450k, the captions are significantly longer than those in other datasets.
2.2 CONTRASTIVE SENTENCE RATING MODULE
Basic Formulation. Let My denote the LVLM parameterized by 0. Given image / and a captioning instruction T, the model generates caption as: C My (I, T). where C' = {c1, 2.....,C ) is sequence of tokens generated autoregressively. At each decoding step I, the token CN is drawn from the conditional distribution: Yt ~ Pe (y: cer)), with y/t denoting the predicted token and ccr the preceding context. The probability distribution is conditioned on both the image / and the textual input [T, cze), guiding the model to iteratively generate each token.
Offline Contrastive Probability Analysis. To detect hallucinated content without disrupting the natural language distribution, we adopt an offline contrastive probability analysis, in contrast to prior online decoding methods (Leng et al.) 2024, Wang al., 2024 Huo et al.) 2024). Given the initial caption tokens C = {c1, -., Co. J. we compute two sequences of token probabilities: caption tokens with and without the image input. The probability sequence P conditioned on the image / is:
Pt = po (y: = c. [T, cct)), I E [1,
Here, Pt denotes the likelihood of generating token c2 given the image I, the instruction prompt T, and previous tokens Cec. We then compute the counterpart P' by conditioning only textual input:
P' (ph.p2,..... ... P.
po(y = C. [T. ced)), t E [1,
This sequence reflects the model's inherent linguistic prior-hov how likely it is to generate c, without visual evidence. Since large vasion-language models (LVLMs) inherit strong language modeling capabilities from I.MMs, they may over-rely on textual co-occurrence patterns, leading to hallucinated outputs (Leng et al., 22024) [iu et al., 2024c). To quantify this effect, we define the contrastive probability sequence APas as:
AP = P P' (API. Ap2....
Apr),
APp-
Ps Ph.
A high Apr indicates that token es benefits significantly from the visual context and is thus more likely grounded in the image. In contrast, a low Ap& suggests that the token is generated primarily based on language priors, signaling potential hallucination.
Sentence- Level Rating. To mitigate hallucinations without disrupting fluency, we filter at the sentence level rather than the token level. The initial caption C segmented into sentences C' = C+ {c) ck difference App from Equation forming a sentence-wing wise sequence A Ps = (Apt,...,,Apl Apk,)) )-
C. using punctuation (e.g., periods), where m is the number of sentences. Sentence .., c consists of ką tokens, and each token associates with contrastive probability
To assess whether sentence is language-biased, we compute the maximum Aut over critical tokens Sentences with strong visual grounding are retained as the Golden Sentences Sg:
Sc [C& max (Apt. Api ... Apt. >>
(5)
where is tunable threshold, with higher values leading stricter filtering.
Identified via part-of-speech tagging exclude function words such adpositions.
5

---

Published as a conference paper at ICLR 2026
InfaVQA
ChurtQA
SEED
48.5
75.0
69.5
ScaleCap
ScaleCap
ScaleCap
47.0
DensePasion
74.5
DensePusion
69.0
DensePuston
--5.5
66.5
74.0
44.0
68.0
42.5
73.5
67.5
41.0 1d. 150 200 250 300 350 400 450
ion
73.0
100
150 200
250 300 350 400 450
67.0
100
150 200 250 300 350 400 450 Number of Captions (k)
Number of Captions (k)
Number Captions (k)
Figure The scaling trend under different number of pretraining data.
2.3
CAPTION INTEGRATION
Our ultimate goal is to form complete and structured image caption. Leveraging the strong sum- marization and logical reasoning capabilities of the LLM, we instruct to organize and consolidate the relatively fragmented content from the detail sets using two prompts, To and Tp, for object-level details and position-level details respectively. tion, also provide the LLM with the golden sentences caption backbone. This helps the LLM maintain awareness of the overall caption structure, enhancing the quality of the summarization. As result, generate the following summaries:
ensure coherence and structure during summariza-
C. (S,, To, Do), Cp (SG, Dp).
(6)
where C, summarizes the object-level details, and C, summarizes the positional details of the objects in the image. Finally, we utilize the LLM once more to integrate C. and Cp into a comprehensive final caption. The final caption is generated as follows:
F (M Tiinal, Co. Cp),
where Tymnal is the integration prompt. All prompts used in ScaleCap can be found Appendix D
2.4 SCALECAP-450K DATASET
Based on ScaleCap, we create hyper-detailed image caption dataset as Figure. presents. We collect 450,000 images and then annotate them using ScaleCap to generate high-quality image-text pairs.
Data Source and Processing. In collecting images for our dataset, we primarily focus on two aspects: diversity and richness of image content. Given that the SharcGPT4V-1000 already includes a wide range categories, such as artworks, landmarks, etc., inherently offers a certain level of diversity. Therefore, we opt to directly incorporate these images into our dataset. To further enhance the dataset's ' diversity and to obtain more content-rich images, we additionally select 350k images from the LAION-5B (LAION 2022) dataset which encompass wide range of subjects, styles due to its web-based origin. During filtering, we retain only images with high resolution and moderate complexity. The specific tools and details used in filtering can be found in the Appendix
Caption Model Selection. As previously mentioned, ScaleCap leverages the collahoration between a Vision Language Model and a Large Language Model generate high-quality captions. For LVLM. as we discussed above, small model is capable of capturing visual content, so we use Qwen2-VL-7B by default. When it comes to the LLM, the question-raising task is relatively simple, while the integration of complex and abundant visual information within thousands of tokens requires advanced reasoning capability, so we resort Qwen2-72B based on an empirical study.
PRETRAINING EXPERIMENTS
To comprehensively evaluate the effectiveness the ScaleCap-450k dataset, we conduct extensive pretraining experiments. The experimental details are as follows.
3.1 IMPLEMENTATION DETAILS
Our model structure follows LLaVA-NeXT (Lu et al., 2024b). comprising a vision encoder, an MLP projector, and an LLM. To thoroughly evaluate approach, experiment with three configurations: (1) Owen2.5-7B + Qwen2.5- -iTT, (2) Qwen2.5-3B + Qwen2.5- --T,,, and (3) InterLLM.55-7 + CLIP- VVT-L/14-336. Training involves three stages: initial pretraining BIII--58KK, further pretraining

---

Published as a conference paper at ICLR 2026
Table 1: Comparison with different datasets on 11 benchmarks. ScaleCap-450k significantly improves pretraining efficiency, achieving the best results on nearly all benchmarks with the same amount of data. Further stringent ablations about image sources can be found in Appendix
Pretraining Data
Info
Doc
Chart
MM
Math Vista
LLaVA Bench
Model
VQA
VQA
QA
Star
MMVet
MMB
MMMU
SEED
A12D
Average
Vanilla
46.2
81.5
75.5
47.0
47.0
72.8
46.7
74.6
45.2
69.9
71.6
61.6
Qwen2.5-72 7- Qwen2.5 VIT
ShareGPTa-4450L
47.5
82.9
76.0
48.8
46.2
72.9
48.9
75.2
43.8
71.3
72.7
62.4
DenseFusion 450k
49.4
84.8
77.1
49.2
47.5
70.3
52.4
73.1
44.3
70.6
73.9
63.0
ScaleCap-4S00
51.8
85.7
77.8
48.8
49.7
74.7
35.9
75.6
46.1
71.6
74.0
64.7
Vanilla
39.1
76.3
72.1
44.8
41.6
66.4
39.9
69.1
37.4
67.1
69.7
56.7
Qwen2. 5-3B
ShareGPT4V-450
42.4
78.3
73.0
44.8
43.7
66.8
43.3
69.5
38.5
67.9
70.2
58.0
Qwen2.5 VIT
Denselusion 450k
44.5
81.1
73.8
43.5
45.9
69.4
39.9
68.7
42.1
68.2
70.8
58.9
ScaleCap
47.2
81.7
74.7
44.9
46.1
68.3
45.6
70.0
41.3
69.1
71.8
60.1
Vanilla
36.2
72.3
68.4
47.8
43.5
68.7
41.3
72.2
41.2
72.7
73.9
58.0
M2.5-7B CLIP-VIT-L
ShareGPT4V
36.9
48.6
66.6
41.5
72.6
58.3
DenseFusion-450k
39.1
75.1
70.0
48.7
44.1
66.9
47.2
72.2
40.1
73.6
73.0
59.1
ScaleCap-t5Ok
39.6
75.5
71.3
48.7
44.5
70.3
48.0
73.4
42.1
74.0
74.7
60.2
Table 2: Caption informativeness comparison results in Prism Framework. ScaleCap significantly outperforms the quality of captions generated by Qwen2-VL-7B and (oven2-VL-7225
Table 3: Ablation study of Object Instructions and Position Instruc- tions benchmarks subset.
Caption LVLM
MM
MM
Info VQA
Charl
Text VQA
Text VQA
MM
Chart
Average
Methol
Strategy
QA
QA
Prism
QwenZ VL-7B
53.3
47.7
49.3
68.5
$1.7
$4.1
Only Object Instr
52.9
54.5
69.1
58.8
Prism
VV-72-V1-7228
57.3
48.7
50.0
69.5
54.4
56.0
Only Position Instr
52.3
54.3
65.7
57.4
ScalCap
Qwen2-VL-78
58.8
50.3
53.8
72.9
55.3
58.2
ScaleCop
53,2
58.8
72.5
61.5
using high-quality image caption, and final instruction-cuning with Open-
IM (Chen & Xing 2024). Baselines for comparison include a two-stage Vanilla method (without further pretraining). ShareGPT4V-450k (Chen et al., 2023). and DenseFusion-450k (Li et al. 2024).
To ensure fair evaluation, differences are limited to the further pretraining datasets only. Details of the settings can be found in the Appendix
3.1.1 MAIN RESULTS
Comparison with different pretraining datasets. As shown in Table T comprehensive experiments under three settings demonstrate that pretraining with ScaleCap-450k consistently yields the best performance across most benchmarks. For instance, in Qwen2.5-7B setting, ScaleCap-450k improved Info VQA scores by 4.3% over ShareGPT4V-450k and 2.4% over DenseFusion-450k. On natural image QA benchmarks like MMVet, ScaleCap-450k achieved a 7% gain over ShareGPT4V-450k and 3.5% over DenseFusion. Similar gains were observed with other LLMs.
Compared to DenseFusion, which uses captions from multiple expert models that often miss object details due to limited attribute coverage, ScaleCap leverages general-purpose models for higher- quality captions. These detailed descriptions of objects significantly enhance modality alignment during pretraining. Consequently, the better-aligned visual features are more readily understood by the I.M., enabling finer-grained image comprehension and improved benchmark performance.
The pre-training data scaling performance. We then study the data efficiency of captions generated by ScaleCap. We conduct pretraining using varying data volume from 100K to 450K, with the Qwen2.5-3B setting. The results show that, given the same pretraining samples, modality alignment using the ScaleCap dataset significantly outperforms the DenseFusion dataset. Moreover, as the pretraining data volume increases, the advantage of ScaleCap becomes even more pronounced. These findings indicate that captions generated by ScaleCap enable more efficient modality alignment. Additionally, the steep upward trend the ScaleCap curve suggests that further expanding the data volume will continue to yield substantial performance gains.

---

Published as a conference paper at ICLR 2026
Table ScaleCap, equipped with GPT-4o in the Prism frame- work and supplemented with image-vish visible responses, outper- forms other advanced proprietary models.
Table 5: Ablation study on the sum- marization model scale in ScaleCap on benchmarks subset.
Method
Sonne13.5
GPT4V GPT4o Gemini Qwen2 VL-72B
ScaleCap (GPT40xGPPT4))
Summarization Model
MMVeL
MMS
Qwen2-7B Qwen2-725 7213
43.6
40.3
MMVel]
66.0
67.5
69.1
70.4
74.0
76.1
58.8
49.5
DIVE INTO SCALECAP
4.1
VALIDATING INFORMATIVENER OF SCALECAP VIA VQA
Setup. Prism (Qiao et al. 2024) is framework designed to disentangle the perception and reasoning processes of LVL.Ms. It separates the problem-solving pipeline into two stages: the perception stage, LVLMS extract information from images and convert it into textual form without access to the question, thus producing general-purpose captions; in the reasoning stage, LLMs generate answers based the extracted text. With fixed LLM, the benchmark performance directly reflects the informativeness contained within the caption. We use Qwen2-72B as the LLM in the Prism framework to answer visual questions based on captions. For comparison, Generic Instruction used in (Qiao et al. 2024) as a prompt to Qwen2-VL-7B and Qwen2-VL-72B to generate detailed caption possible, establishing strong baselines.
provide the carefully designed
Main Results. As illustrated in Table 2 we can observe that ScaleCap outperforms both Qwen2- VL-7B and
across all benchmarks, demonstrating outstanding performance. These results strongly validate the informativeness of caption generated by ScaleCap, demonstrating that its heuristic question answering effectively extracts fine-grained image details. This significantly enhances the caption quality initially generated by VL-72- even surpassing Qwen2-VL-72B VL-72B by a substantial margin. These findings indicate that ScaleCap can generate captions of significantly higher quality than those produced by large LVLM trained on extensive image-text data.
The Potentials ScaleCap. ScaleCap a flexible captioning pipeline that allows for the replacement of both the LVLM and LLM with any open-source proprietary models. This flexibility suggests significant potential, especially when equipped with powerful models like GPT-4o, which could enhance caption quality. To test this, we compare ScaleCap with GPT-4o against other advanced models, including Sonnet 3.5 and Gemini-2.0-PP irrelevant captions first, with the LVLM's 's direct -isibe response also attached supplemental information to the LLM. As shown in Table ScaleCap not only outperforms its direct answering baseline but also all proprietary LVLMs, indicating its high potential.
In experiments, ScaleCap generates the question-
4.2 VALIDATING INFORMATIVENESS OF SCALECAP VIA RECONSTRUCTION
Image Reconstruction Comparison
Caption
Ranking Score
The image disicss exquiitilyy amrnged
2.15
FLUX
L76
Generate: Caption
The fecel paint pristine white plate with slander
Generate Image
User Study
GPT
Scalecop
Original Image
ScoleCap
GP
Qwen2-vL--28
Figure 6: Human evaluation of image similarity with original image over 50 samples and 25 volunteers. Volunteers rank the images based on their similarity to the original image.
Leveraging the capabilities of powerful modern text-to-imate generation models, the similarity between a reconstructed image and its original can effectively reflect the extent to which a caption covers the image's 's content. To gain an intuitive understanding of caption quality, we utilize one of the best image generation models, FLUX. FLUX can adhere to very detailed instructions, such as "wearing a silver cross-shaped necklace around his neck." " As result, it is an ideal tool for validating caption quality. When the caption is highly detailed and covers every object in the image, the FLUX-genee -egnerrrd image shows high degree of similarity the original image. Conversely, if the caption contains errors or lacks detail, the similarity will be much lower. We randomly sampled 50 images and used ScaleCap, GPT4o, and Qwen2-VL-777 VL-72B to generate captions. FLUXX then generate
8

---

Published as a conference paper at ICLR 2026
MMVet
MMStar
Accuracyr%,
-66
58
64.2
55.4
56,7
58.6
49
50
49
49.3
49.6
49.9
50.3
50
42.9
51.5
51.7
ML
Qwen22/1-76
Qwin2
39.0 41.0
54-
48
30
51.8
50
02
10 15 20
all
16
0.2 6 10 15 20
all
Recogailiio Category
Konwledge
Number of Instructions
Number of Instructions
(b)
Figure 7: (a) Caption scaling. We adjust the number of instructions N used in ScaleCap to explore changes in benchmark performance within the Prism framework. (b) As the LVLM ScaleCap scales up, perception capability saturates at 7B, while world knowledge continues to increase.
corresponding images based on these captions. Finally, we invite 25 volunteers to rank the similarity of the generated images the originals. A model that ranks first in the three categories receives three points. As shown in Figure. 6 captions generated by ScaleCap result in images that more accurately reflect the original images than those generated by GPT4o, significantly surpassing Qwen2-VL-72B. This further demonstrates the superior quality of our approach.
4.3 ANALYSIS ON SCALECAP COMPONENTS
Caption quality increases with scaling of inference budgets N. In ScaleCap, we adjust the number of heuristic questions trade off between the richness of image detail and computational cost, To intuitively present this effect through the Prism framework, we conduct experiments on MM and MMStar. The results in Figure show that initially increasing the number of heuristic questions leads to sharp rise in benchmark performance, indicating that these questions significantly enrich image detail. However, when the number exceeds 20, the performance curve begins to plateau, suggesting that most objects mentioned in VQA tasks are already covered. This trend aligns well with the general scaling laws observed in previous work.
LVLM with 7B size is sufficient and efficient. In this section, we study the scale of the LVLM used for visual information extraction. Here we use the Qwen2-VL, series from 2B to 72B and evaluate the benchmark performance under the Prism setting above. Due to the high inference cost of large LVLMs, we use a randomly sampled 300-question subset from MMStar, ChartQA, and TextVQA. As shown in the first part of T find that LVLM from 2B to 7B shows promising improvement, but the further improvement for 72B is minor. This is consistent with the conclusion in Prism and proves our analysis that 7B size model is capable enough for extracting most visual information.
Large LVLM introduces more world knowledge in the caption. Despite the minor gap between 7B and 72B models, we further analyze the difference in Figure 7b different categories of questions in Vetet. We find the models perform similarly at the recognition question, but larger models get better performance on questions that related to world knowledge, which reasonable because model with larger size contains more knowledge and could introduce it in the caption.
Small LLM struggles with complex visual information integration. Then we study the influence of the LLM used for heuristic question raising and information integration. As shown in the second part of Table d the 7B LLM shows significantly worse performance than the 72B model. To position the performance bottleneck, we first compare the question-raising quality and find the gap is minor. So we use the same context before the Caption Integration Phase and evaluate the integration quality between models in Table 5 and observe a consistent performance gap with Table 6 This indicates substantial difference in summarization capabilities among model sizes. During summarization, the combined object details can result in context length of up to 20k tokens, which causes Qwen2-7B to miss important information due to its limited ability to handle long contexts.
Object and Position Instructions equally important. In ScaleCap, questions raised for both the object and its position. Here we study their effectiveness in Table within the Prism framework. Removing either Object Instructions Position Instructions results in notable drop in performance.
Ablating Contrastive Sentence Rating Strategy. Here we study the effectiveness of the Contrastive Sentence Rating Module CHAIR, a benchmark specialized for hallucination evaluation. As

---

Published as a conference paper at ICLR 2026
Table 6: Ablation on LVLM and LLM scale in benchmarks subset. As the LVLM scales up, caption quality saturates from 7B; as the LLM scales up, caption quality continues to improve.
Table 7: Hallucination evaluation results. Golden Sentence Selec- tion strategy performs the best.
LVLM
MM
Vet
MM
Star
Chart
QA
Text VQA
Average
Method
CHAIRs
CHAIRJ
LLaVA-v1.5
48.8
13.9
Qwen2-VL
Qwen2-728 722
54.0
43.6
58.8
44.4
50.2
+VCD
46.8
13.2
Qwen2.722
49.5
Qwen2-VI. V22-BB
Qwen2-72B
59.0
52.3
69.4
54.1
58.7
+Golden Sentence
33.6
11.3
Qwen2 VL.-BB
Qwen2 7B
45.5
40.0
53.0
47.5
46.5
Qwen2. PLe 7B
44.2
7.5
Qwen2-V1. V1.
Qwen2-72B
58.8
49.5
72.5
53.2
58.5
GGolden Sentence
25.8
shown in Table 7 strategy significantly mitigates hallucination in LLa outperforming baselines such as OPERA (Huang et al., (024) and VCD (Leng et al. (024). also achieves notable hallucination reduction capable models like Qwen2-VI.-7B, proving the effectiveness and generalization of the Contrastive Sentence Rating Module in hallucination elimination.
CONCLUSION
In this work, we present ScaleCap. scalable image captioning framework. By integrating heuristic question answering and contrastive sentence rating, ScaleCap progressively enriches and calibrates captions with increased inference budget. resulting in more detailed and balanced descriptions. Extensive experiments demonstrate that captions generated by ScaleCap not only excel in downstream tasks such as VQA and image reconstruction, but also significantly boost LVLM pretraining when used at scale, paving the way for high-quality captioning systems.
ACKNOWLEDGMENTS
This project is funded in part by Shanghai Artificial Intelligence Laboratory, Shanghai Innovation Institute, the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s s InnoHK, Dabua Lin a PI CPII under the InnoHK.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work builds on large vision-language models (LVLMS) to develop scalable and debiased image captioning. Our approach primarily uses publicly available datasets such as LAION-SB and ShareGPT4V, which are filtered to ensure appropriate resolution, diversity, and complexity. No personally identifiable, sensitive, or private data were intentionally collected. We acknowledge that large-scale image-text datasets may contain biases or mislabeled content, which can propagate into downstream models. To mitigate these risks, our method explicitly targets multimodal and linguistic biases by filtering hallucinations and promoting balanced detai. In addition, the image data undergo rigorous cleaning to further enhance reliability. The generated captions are intended to support rescarch in multimodal alignment and are not designed for high-stakes vecision-making scenarios such as medical or legal applications. We emphasize that any deployment of this work should carefully consider fairness, potential misuse, and compliance with privacy and copyright regulations. This research adheres to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our work. The detailed architecture of ScaleCap, including heuristic question answering and contrastive sentence rating, is described in Section 2 with all prompts provided in Appendix D Implementation details of pretraining experiments, including model configurations, training stages, and hyperparameters,
reported in Section and Appendix [ The construction of the ScaleCap-450K dataset, including data sources, filtering criteria, and caption generation process, documented in Section 2.4 and Appendix F Ablation studies and benchmark comparisons further validate the robustness of our findings. Together, these resources provide sufficient transparency for independent verification and extension of our results.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 222300828266... /2966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502.2500..192 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.1888 18930, 2024.
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, et al. Sam 3: Segment anything with concepts. arXiv preprint arXiv:2511. 1671199,, 2025.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568 2021.
Lin Chen and Long Xing, -een-llava-next: An open-source implementation of llava-next series for fa- cilitating the large multi-modal model community. /tttps://github. com/ xi acachen98/
Open LLaVA NeXT 2024.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference Computer Vision, pp. 370-387. Springer, 2024.

---

Published as a conference paper at ICLR 2026
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprins arXiv:1504.00322. 00325, 2015.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.03227, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.063 0,65000, 2023. URL https
7/api. Seappissemantcccchiiee wcw.chlarccccccccccooooooooaaaae Terg/CorpussDii 25861522222622
Henghui Ding,
Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 20224-20234 2023.
Hongyuan Dong. Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Bench- marking and improving detail image caption. arXiv preprint arXiv:2405. 19092, 2024.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544
2023.
Yuxin Fang, Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19358- 19369, 2023.
Tinglei Feng, Yingjie Zhai, Jufeng Yang. Jie Liang, Deng-Ping Fan, Jing Zhang, Ling Shao, and Dacheng Tao. Ic9600: A benchmark dataset for automatic image complexity assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, (01):1-17, 2023. doi: 10.1109/TPAMI. 2022.3232328.
Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. Imageinwords: Unlocking hyper-detailed image descriptions, 2024.
Yunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung- Yi Lin, Ming- Yu Liu, and Yin Cui. Visual fact checker: Enabling high-fidelity detailed caption generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14033-14042, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.222 21783. 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entan- gled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375-14385, 2024.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and petrospection-alllcatioo on Computer Vision and Pattern Recognition, pp. 13418-13427, 13427, 2024.
In Proceedings of the IEEE/CVF Conference
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang, Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vasion-language models. arXiv preprint 2024.
Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, and Qingmin Liao. Catch: Complementary adaptive token-level contrastive decoding mitigate hallucinations in Ivims. arXiv preprint arXiv:241 122713, 2024.
12

---

Published as a conference paper at ICLR 2026
Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, et al. Veclip: Improving clip training via visual- enriched captions. In European Conference Computer Vision, pp. 11 111-127. Springer, 2024.
LAION, LAION-5B: A New Era of Open Large-scale Multimodal Datasets, 2022. URL https: Accessed: 2025-04-22.
//laion. ait.lo/////bllg//aaaa/ laioon 5bb|
Saehyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, and Sungroh Yoon. Toward robust hyper- detailed image captioning: multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint arXiv:2412. 15484, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882 2024.
Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. Beyond fixed: Training-free variable-length denoising for diffusion large language models. arXiv preprint arXiv:2508.008/7 2025.
Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin. Visual self-refine: pixel-guided paradigm for accurate chart parsing. arXiv preprint arXiv:2602. 16455, 2026.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified dision-languaee understanding and generation. In International Conference Machine Learning, pp. 12888-12900. PMLR, 2022.
Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang. Xinlong Wang, and Ling-Yu Yu Duan. Densefusion-Im: Merging vision experts for comprehensive multimodal perception. arXiv preprint arXiv:2407.08303. 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large lision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00053. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.084. 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: training-free method for alleviating hallucination in Ivims. In European Conference Computer Vision, pp. 125-140. Springer, 2024c.
Yuhong Liu, Beichen Zhang, Yuhang Zang, Yuhang Cao, Long Xing, Xiaoyi Dong, Haodong Duan, Dahua Lin, and Jiaqi Wang. Spatial-ssrl: Enhancing spatial understanding via self-supervised reinforcement learning. arXiv preprint arXiv:2510.27666 2025.
Meta AI. LLaMA 4 Models, 2025. URL htthtttt////m www. llama wwt.:/oc.cooceeeecee sllaamalmmammm 4/ Accessed.
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg. Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. In European Conference on Computer Vision, pp. 291-309. Springer, 2024.
OpenAI. Chatgpt. htttp:://chtt
2023a.
OpenAl. Gpt-4v(ision) system card. 2023b.
13

---

Published as a conference paper at ICLR 2026
Zhangyang Qi. Zhixiong Zhang. Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Understand 3d scenes from videos with vision- lisoon-lagguage models. arXiv preprint arXiv:2501001428, 2025a.
Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao. VIn-rl: Vision- language navigation via reinforcement fine-tuning arXiv preprint arXiv:2506.1722) 2025b.
Rui Qian. Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Self-supervised object-centric learning in videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023.
Rui Qian. Shuangrui Ding, and Dahua Lin. Rethinking smage-to-video adaptation: An object-centric perspective. European Conference on Computer Vision, pp. 329-348. Springer, 2024a.
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:119336-19966 2024b.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video ilms with active real-time interaction via disentangled perception, decision, and reaction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24045-24055, 2025.
Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang. Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms. arXiv preprint arXiv:2406. 14544444, 2024.
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13009-13018, 2024.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556 1556-2565, 2018
Yanpeng Sun, Jing Hao, Ke Zhu, Jiang-Jiang Liu, Yuxiang Zhao, Xiaofan Li, Gang Zhang, Zechao Li, and Jingdong Wang. Descriptive caption enhancement with visual specialists for multimodal perception. arXiv preprint arXiv:2412. 14233, 2024.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint XXXx:23307.09888 2023.
Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero- Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (67000266799 2024.
Yoshitaka Ushiku, Masataka Yamaguchi, Yusuke Mukuta, and Tatsuya Harada. Common subspace for model and similarity: Phrase learning for caption generation from images. In Proceedings of the IEEE international conference on computer vision, pp. 1668-26776 2015.
David Wan. Jaemin Cho, Elias Stengel -skkin, and Mohit Bansal. Contrastive region guidance: Improving grounding in vision-language models without training. In European Conference on Computer Vision, pp. 198-215 Springer, 2024.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large laiiun- models with instruction contrastive decoding. arXiv preprint 1003 2024.
14

---

Published as a conference paper at ICLR 2026
Long Xing, Qidong Huang, Xiaoyi Dong. Jiajie Lu, Pan Zhang. Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision- lsaounlanguage models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247 2024.
Long Xing, Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jinsong Li, Shuangrui Ding, Weiming Zhang, Nenghai Yu, et al. Scalecap: Inference-time scalable image captioning via dual-modality debiasing. arXiv preprini arXiv:2506. 19848, 2025.
Hu Xu, Po-Yao Huang, Xiaoqing Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zeitlemoyer, Wen-tau Yih, et al. Altogether: Image captioning via re-aligning alt-text. arXiv preprint arXiv:2410.1725. 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12944 12953, 2024a.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 00022-14033 2024b.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fin-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13807-13816 2024c.
Pan Zhang. Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. model for advanced text-image comprehension and composition. arXiv preprint ar arXiv:2309. 15119.1112 12, 2023.
A lisoun-angguage large
Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1724-1732, 2024.
Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He. Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Sec: Advancing complex video object segmenta- tion via progressive concept construction. arXiv preprint arXiv:2507.15852 2025.
Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 291-300, 2024.
A RELATED WORK
Hallucination in LVLMs. Hallucination remains a significant challenge in LVLMs (Bai et al. 2024
Cui al., 2023 Liu et al. 2024a), despite the rapid development of multimodal. Object hallucination occurs when large vision- language models produce textual descriptions that mention objects or attributes that are not actually present in the corresponding image. This issue is commonly seen in tasks like image captioning and visual question answering, where essential ensure precise correlation between the visual and textual elements (Guan et al.) 2024 Li et al., 2023). Currently, a range methods have been proposed to address hallucination. These methods can be broadly categorized into two types: one requires training, with numerous approaches utilizing diversified training data to enhance the instruction tuning phase (Yu et al. 2024a), while others leverage preference data through DPO or other reinforcement learning strategies to mitigate hallucinations (Sun) et al. 2023) Yu et al.
2024c Zhu et al. (002)) Another approach is training- free (Huo et al. 2024
Wang et al. 2024
Wan al. 2024
Kan et 2024h, with notable works such as OPERA (Huang
15

---

Published as a conference paper at ICLR 2026
Table 8: Ablations about image sources. We apply ScaleCap to annotate the images in DenseFusion- 450k, yielding the CaneCCap-DDnsuuiionnnnea33 dataset for stringent comparison.
Model
Pretraining
Info VQA
Doc
Chart
MM
Math Vista
LLAVA
MM Bench
MMB
MMMU
SEED
ADD
Avg
Data
VQA
QA
Star
Vet
Qwen2.5.3B
Vanilla
39.1
76.3
72.1
44.8
41.6
66.4
39.9
69.1
37.4
67.1
69.7
56.7
Qwea2.5-VIT Dascapppeesseuuiioot55k
DenseFusion-450k
44.5
81.1
73.8
43.5
45.9
69.4
39,9
68.7
42.1
68.2
70.8
58.9
46.7
81.7
74,9
44.1
46.0
73.8
43.9
69.9
40.8
69.2
71.2
60.2
ct al. 2024), which employs a novel MLLM decoding method based on an over-trust penalty and a Earropection-allocatioo strategy that addresses the internal causes of hallucinations. VCD (Leng et al. 2024) contrasts the output distributions derived from original and distorted visual inputs to mitigate over-reliance on statistical biases and unimodal priors. These methods work during the decoding phase, which is online, and detect hallucinations at the token level. We propose a sentence-level hallucination detection method, which is performed after the entire sentence is generated. This approach can enhance the coherence of the sentence and improve the stability of detection.
Image Caption. To enhance LVLMs, early works focused on large-scale image-text datasets. CC3M Sharma [C3MaShrrma et al.] (2018) and CC12M (Canngihni et al. (2021) leveraged web-crawled alt-text
but lacked line-grained details, while manually annotated datasets like SBU et al. (2015) and
COCO-Captions Chen et al. (2015) offered higher quality but struggled with contextual richness.
Later efforts Fan et al. (2023); Lai et al. (2024); Yu et al. (2024b); Rasheed et al. (2024); Garg
et al. (2024): Onoe et al. (2024) improved captions using LLMs. LLavA Liu al. (2023) intro-
duced human-ammaan annotaaed captions and bounding boxes to guide GPT-4 but remained annotation- hhavy. Chenn
ShareGPT4V Chen et al. (2024) constructs a large-scale dataset with 1.2M highly descriptive cap- tions, demonstrating significant improvements in LVLMs' performance across multiple benchmarks. DCE|Sun et al. (2024) enhances captions with fine-grained attributes and 3D spatial relationships using open-source visual specialists, optimizing cost-effective annotation for complex scenes, Per- ceptual Fusion Li al. (2024) leverages high-resolution image processing and multi-expert signal fusion (detection, OCR, tagging) to train a scalable caption engine for open-domain visual perception. There also work that improves caption quality and mitigates hallucinations by building specialized pipelines. For example. VFC et al., (024) integrates an open-voue vece-vocybulary detector together with multiple LVLMs to synthesize high-quality captions, while CapMAS(Lee et al. 2024) uses an LLM to break the initial caption into independent sentences and then has an LVLM verify them. These methods rely on LVLMs or dedicated models as verifiers, which helps reduce hallucinations to some extent. However, they provide very limited improvement in visual information, and in some cases may even reduce In contrast, our dual-modality debiasing strategy achieves substantial gains in both aspects.
B ABLATIONS ABOUT IMAGE SOURCES IN PRETRAINING EXPERIMENTS
In our previous experiments Table 1. we did not strictly control for ScaleCap and other datasets to use exactly the same set of images. As a result, factors such as image resolution and type could potentially influence the comparison. To mitigate this concern, we conducted an additional ablation study on image sources under the Qwen2.5-3B + Qwen2.5- ViT setting. Specifically, we applied ScaleCap to annotate the images in DenseFusion 450kk yielding the ScaleCap-DenseFusion Dew.eeesiinnnnn555 455kk dataset. The corresponding experimental results are presented in Table The results show that, under the condition of using exactly the same set of images, pretraining with Saalleaap-Deneeuusione yields an average improvement of 1.3% over DenseFusion-4500, and outperforms the baseline on 10 benchmarks. This stringent comparison demonstrates that the superiority of the ScaleCap-450k dataset does not stem from image diversity or resolution, but rather from the exceptionally high quality of the captions generated by ScaleCap itself
C LLM SIZE EFFECT ON QUESTION GENERATION
We explicitly evaluated the robustness of our question-generraioon component to both LLM size and domain shift. Using Qwen2VL-7B to produce initial captions for 500 natural images, we then generated questions with Swen2-7B-Insttrr and Qwen2-72B-Inge Instructt under the same ScaleCap prompt. The accuracy of the generated questions is verified through a strict format matching process. The result is in Table g Across domains, the question-format accuracy remained consistently
16

---

Published as a conference paper at ICLR 2026
Algorithm ScaleCap: Scalable Image Captioning via Dual-Modality Debiasing
Require: Image I, LVLM Mv. LLM Mr. Scale Budget N, Threshold
Require: Prompts:
Tinst, Tiot finat
Ensure: Final Caption (inal
Stage 1: Initial Generation & Debiasing
1: Cinit < MMy (I, Tistt
2: Sa CONTRASTIVSSANTTNNERRTTTTCCCCCC
1,T)
Stage Heuristic Question Answering
3: 1o 1p f 0
4: for each sentence Sk E SG do
5:
1 + ML(Tict Sk)
6:
Th ADDPOSITIOPRFFIX(5
7:
Io f LUI
Ip + 1, 1
8:
end for
9: Do. Dp + 0
Qqueue + SELECT(Lo, N) SELECT(I,:.N)
11: for each question 9 E Qqueve do
Araw (M.
13:
Aclean <- CONTRASTIVESENTTNNCERATTTTG (A I,)
14:
if q E Lo then
15:
Do + Do U Adean
16:
else
17:
Dp + D2 U Aclean
18: end
19:
end for
Stage Caption Integration
20: Co - (z To, D.)
Mp Tp. Dp)
22: Cfinal +-Mi(Sa, Tysnats Co. Cp)
23: return Cfinal
24: procedure CONTASSTVSSENTNNCRRAINGG....
25:
Split Craw into sentences {51, $2, $m Sfittered / for each sentence in Craw do
26:
27:
28:
Compute probs with image: P po(yl [T, (cel)
29:
Compute probs w/o image: p = pa(
[T, cerl)
Compute difference: AP= if max (AP) > T then
p
31:
Sfittered 8 filtered U {
33:
end if
end for
35: return Sfittered
36:
end procedure
high: for natural images, Qwen2-7B and Qwen2-72B achieved 97.5% and 98.7%; for Qwen2-72B on Infographic and STEM images (additional 500 images), accuracy reached 98.8% and 97.7%, respectively. This suggests that question generation is relatively simple, robust task.
D PROMPTS USED IN SCALECAP
Prompts used in ScaleCap and Prism presented in Figure 8 and Figure 9
17

---

Published as a conference paper at ICLR 2026
Table LLM size effect on question generation
Instruction Generation Model Domain
Instruction Accuracy
Qwen2-7B
natural
97.5%
Qwen2-72B
natural
98. 98.7%
Qwen2-72B
Infografic
98.8%
Qwen2-72B
STEM
97.7%
T, Instructions summarize object details:
Descriptions:
Collect all details about each object from the descriptions, including detoiled appearance, structure, matorial, ond special marks or logos. include any analysis
T Instructions for summarize position details:
Descriptions:
Extract and abstract only the position information obout each object from the decriptions. Do not include andlysis your opinions.
Instructions used for other LVLMs in Prism:
Describe the fine-grained content of the image, including objects, relationships, instance location, and any
Instructions used for LLM in Prism:
You are an excellent text based reasoning expert. You are required to onswer the question based on the detailed description of the
Description:
Question:
Final Instructions used for summarizing object and position detsils:
Basic Context:
Object Information:
Position Information:
0
Following the logic Bosic orgonize all details provided Object Information Position Information give very comprehensive description obout image. include any onalysis or your opinions.
Figure 8: Prompts used in ScaleCap and Prism.
E LARGE LANGUAGE MODEL USAGE
During the preparation of this paper, large language models (LLMs) were used exclusively as tools for language refinement and did not contribute the methodological analytical aspects of the study. Their main function was to support the writing process by improving grammar, adjusting phrasing, and harmonizing style across different sections of the manuscript. This assistance helped us present complex technical details with greater clarity and coherence, ensuring that our arguments and findings were communicated effectively the reader. addition, L.MMS were employed to identify and correct minor linguistic issues or awkward sentence structures that could otherwise obscure the scientific content. We emphasize that their involvement was strictly limited to these editorial tasks. The design of the research framework, the execution of experiments, the analysis of results, and the formulation of conclusions were carried out entirely by the authors. By restricting the role of LLMs to editing and proofreading, were able take advantage of their strengths in natural language processing while retaining complete intellectual responsibility for the scientific and technical contributions of this work.
18

---

Published as a conference paper at ICLR 2026
Your task is to convert each Object mentioned in given sentence into carresponding instruction, and the resulting instructions are output as "Descrize more defails about the [Object] Ensure your instructions do not cover the raw question, options, or thought process unswering the instructions. You should ignore the Objects that appear inferences, such as the senfences that begins 'it might be' Sentence:
are probably'.
The image depicts mon in suit and tie jumping in the air above bed in bedroom
Instructions:
Describe details about the suit. Describe more details about the tie. Describe more detuils about the bed. Describe detuils about the bedroom.
Describe more
Sentence:
The train appeors to be the main subject of the image, showcasing its sleek design ond modern oppearance
Enstructions:
Describe details about the trein.
Sentence:
The table has few other items on it, including camera, a jar of jom, and a spoon, suggesting that there might be some people ready to
eat
Instructions:
Describe more details
table. Describe more details about the comera.
Describe more details about the jom. the spoon.
Describe dctails
Sentence:
The text "You see the world are" is playful and thought-provoking statement, encouraging viewers appreciate their unique qualities and perspectives
Instructions:
more
the text.
Sentence:
1. "*Prehest the Oven** Prehest your 35D'u0060F (175'00060C).
bescribe more details about the oven.
Describe more details about the preheat temperature.
0
Instructions:
Figure 9: Prompt Tiat used ScaleCap to generate object instructions.
F DATASET PROCESSING
Data Source and Processing. In collecting images for our dataset, we primarily focus on two aspects: diversity and richness of image content. Given that the ShareGPT4V-I00k already includes a wide range of categories, such as artworks, landmarks, etc., it inherently offers certain level of diversity. Therefore, opt to directly incorporate these images into dataset. To further enhance the dataset's ' diversity and to obtaín more content-rich images, we additionally select 350k images from the LAION-5BLAICH available content on the internet, encompassing a wide range subjects, styles, and domains due to its web-based origin. This ensures high level of diversity in the collected data. During filtering, we retain only images with high resolution and moderate complexity. During the image selection process, we filter out images with short-edge resolution of less than 600 pixels to preserve the richness of visual content. To further ensure the complexity and informativeness of the images, we employed (Feng et 2023) to score image complexity. Images were filtered based on a complexity range of [0.4, 0.8), which helped exclude both overly simplistic and excessively complex images, thereby maintaining the overall quality of the dataset. During the image selection process, we also conduct manual sampling filter out potentially harmful images.
(2022) dataset. The LAION-5B dataset is sourced from publicly
Caption Model Selection. As previously mentioned, ScaleCap leverages the collaboration between a Vision Language Model and Large Language Model generate high-quality captions. For LVLM, as we discussed above, a small model is capable of capturing visual content, so we use (wee2--VL-BB by default. When it comes to the LLM, the question-raising task is relatively simple, while the integration of complex and abundant visual information within thousands of tokens requires advanced reasoning capability, resort Qwen2-72B based empirical study.
19

---

Published as a conference paper at ICLR 2026
G PRETRAINING DETAILS
Training Setting. We follow the training strategy consistent with ShareGPTTV, dividing the overall training process into three stages. (1) Initial Pretraining Stage. In this stage, we train the projector from scratch using the BLIP-558K dataset for pre-alignment. The objective is to establish an initial mapping between visual and textual modalities. We adopt a learning rate of le-3 and batch size of 256. (2) Further Pretraining Stage. We initialize the projector with the weights obtained from the initial pretraining stage. Both the projector and the LLM are jointly optimized using high-quality image caption in this stage. This stage leverages high-quality image-text pairs to achieve effective alignment of the visual features extracted by the vision encoder and the corresponding text features Stage. in the LLM. We set the learning rate to 4e-5 and the batch size to 256. (3) Instruction-tumi IM(Chen Xing (024)
During this stage, we use open-source datasets Open-LLa -eeLLVV---e---itttete to finetune the projector and the LLM. Previous research has demonstrated the effectiveness of using We---LLLV--Ne-----ee
open-source datasets such as such Doc VQA and SynDog-EN We adopt this dataset directly for instruction tuning. During this stage, both the projector and the LLM are jointly fine-tuned. A learning rate of 2e-5 and a batch size of 128 used.
1M, which includes diverse data sources
Baselines. (I) Vanilla. We select the model trained through only two stages, namely the Initial Pretraining Stage and the Instruction-tuning Stage, as our basic baseline. This model does not undergo any additional further pretraining and corresponds to the default configuration of the original LLaVA-NeXT. (2) ShareGPT4V-450k, For fair comparison, the amount of data is strictly aligned with that of ScaleCap-450k Specifically, ShareGPT4V-4500 consists of ShareGPT4V-1000 samples randomly selected from ShareGPT4V-PT. (3) DenseFuusion-40kk It is a subset randomly sampled from DenseFusion- IM, which is high-quality dataset constructed using diverse perception experts as image priors to provide explicit information on visual elements. During the training, we use different datasets only in the Further Pretraining Stage, while keeping other settings exactly the same to ensure a fair comparison.
and 350k
Our experiments are conducted on 8 A100 GPUs The initial pretraining stage using BLIP-558K took approximately 2.5 hours, the further pretraining using ScaleCap-450K required 13.5 hours, and the final iustruction-tuning stage took 17.2 hours.
H CURRENT LANDSCAPE OF MULTIMODAL MODELS
Multimodal learning has accelerated rapidly in the past two years, driven by systems that couple
language with perception and action at scale (Liu et al. 2023 Bai et al. 2025 Li et al. 2025
2026 Xing al. 2025 2024 Liu al. 2025 Carion al. 2025 Qian et al. 2025 2024b
Ding
et al. 2023; Qian et al. 2023 These ad vanceees ZZang et al. 2023 2025 Qi et al. |202aa b Qian
al. 2024a) have produced broad empirical breakthroughs across benchmarks and real-world tasks.
Compared to earlier eras that focused mainly on images paired with text, today's 's models are larger,
trained on more diverse corpora, and natively support additional modalities such video and audio.
USER STUDY INSTRUCTIONS
To directly compare the richness and accuracy of captions generated by different LVLMs, we use the captions produced by each model as prompts for FLUX to generate corresponding images. In the following questionnaire, please rank the images based on their similarity to the original image.
20


Reviews:
[{"review_id": "WXDbl3YWNS", "reviewer": "Reviewer_wjkS", "summary": "The paper proposes ScaleCap, a captioning pipeline that iteratively enriches captions while reducing hallucinations by combining (i) heuristic question answering and (ii) contrastive sentence rating. A tunable scale budget controls how many questions are asked, trading cost for detail. Using ScaleCap, the authors build ScaleCap-450K and show consistent gains across 11 benchmarks, plus benefits in Prism-style perception tests and an image-reconstruction study. The author argues that the approach improves informativeness and alignment even with smaller LVLMs.", "questions": "Refer to the weaknesses,", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear, modular method that targets two real failure modes. The offline contrastive, sentence-level rating is a neat way to down-weight language priors without destabilizing decoding.\n- Strong empirical coverage and breadth. Pretraining with ScaleCap-450K beats ShareGPT4V-450K and DenseFusion-450K on most of the 11 benchmarks.\n- Practical efficiency levers. The pipeline uses a small LVLM for perception (object/position Q&A) and a budget N to scale detail, giving users cost–quality control.", "weaknesses": "- Practical efficiency levers. The pipeline uses a small LVLM for perception (object/position Q&A) and a budget N to scale detail, giving users cost–quality control.\n- Prompt dependence in question generation. The method hinges on a “powerful LLM” to craft good object/position prompts; robustness across domains or weaker LLMs isn’t deeply probed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7WftiUAcL", "reviewer": "Reviewer_x2KT", "summary": "The paper introduces *ScaleCap, a scalable image-text paired data distillation pipeline designed to produce long, instance-complete, and visually grounded captions from LVLM without external detectors/tools. ScaleCap has two core components:\n(1) Heuristic Question Answering (HQA): an LLM generates targeted, object/attribute/position questions from an initial caption; a LVLM answers them to surface missing details under a controllable budget.\n(2) Contrastive Sentence Rating (CSR): an offline filter that compares token probabilities with vs. without the image and retains sentences whose “critical tokens” are better supported by the image than by language priors, aiming to suppress hallucinations.\nUsing this pipeline, the authors build ScaleCap-450K, a 450k-image caption dataset sourced mainly from LAION and ShareGPT4V images with resolution/complexity filtering.", "questions": "1. For frontier models, does the way ScaleCap introduces more prior information help the LVLM itself in the pipeline, such as fine-tuning the LVLM with generated data?\n2. If CSR is computed with a different LVLM than the one used to answer HQA (or than the pretraining backbone), do conclusions hold?\n3. A clear pseudocode algorithm will be more helpful for reading and understanding.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The two identified blockers map neatly to HQA (add missing visual facts) and CSR (filter unsupported text). The mechanism is easy to reason about and implement with standard LVLM/LLM primitives.\n2. The observation that 7B-class LVLMs are often sufficient for perception while larger LLMs help during long-context integration provides concrete guidance for cost-constrained systems.\n3. Improvements appear across multiple LVLM backbones/settings, suggesting the dataset’s benefits aren’t model-specific.", "weaknesses": "1. The core claim that balanced, instance-complete detail drives the gains isn’t cleanly isolated from caption length or sheer verbosity. Controlled studies (equal length across methods; fixed token budgets redistributed among object/attribute/position details) are missing.\n2. CSR accepts sentences based on a max-over-critical-tokens Δ probability threshold. This may be unstable (single-token spikes; POS-tagging noise). Robustness analyses (pooling variants, τ-sweeps, cross-dataset calibration) are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxTqULvmLI", "reviewer": "Reviewer_jUSR", "summary": "This paper introduces ScaleCap, a pipeline designed to generate comprehensive and detailed image captions for pretraining VLM. The authors identify multimodal bias and linguistic bias as key challenges. ScaleCap addresses these through Heuristic Question Answering and Contrastive Sentence Rating. Using ScaleCap, the authors create the ScaleCap-450K dataset  and show that pretraining LVLMs on this dataset consistently improves performance compared to using datasets like ShareGPT4V and DenseFusion.", "questions": "1.Could the authors quantify the computational cost—such as average time per caption or total GPU hours required to generate a single caption using the full ScaleCap pipeline, and how does this compare to baseline caption generation costs?\n\n2.While the paper compares pretraining benefits against other datasets, how does the quality of ScaleCap captions or the performance of models pretrained on them compare against models directly improved using training-based hallucination mitigation techniques like RLAIF-V or LLaVA-RLHF?\n\n3.Considering the introduction's potentially restrictive view on tool-based captioning , could the ScaleCap pipeline itself benefit from integrating specific tools, perhaps for grounding question generation or verifying answers, rather than solely relying on the VLM and CSR", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The paper is generally well-written and clearly structured, making the proposed pipeline and experiments easy to follow.\n\n2.The work proposes a complete pipeline aimed at improving the quality, detail, and factuality of LVLM-generated captions.\n\n3.Experiments effectively demonstrate that the ScaleCap-450K dataset, generated by the proposed pipeline, leads to superior LVLM pretraining outcomes compared to existing large-scale caption datasets like ShareGPT4V-450k and DenseFusion-450k.", "weaknesses": "1.The core components of ScaleCap, namely Heuristic Question Answering and Contrastive Sentence Rating, appear to have limited novelty, primarily combining or refining existing techniques rather than introducing fundamentally new concepts.\n\n2.The proposed ScaleCap pipeline suggests a high computational cost for generating each caption, involving multiple model inference stages including initial captioning, filtering, question generation, iterative question answering, answer filtering, and final integration using a large LLM. This multi-step process appears resource-intensive, potentially limiting its practical scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BnXEgSlSlV", "reviewer": "Reviewer_VAQo", "summary": "This paper proposes ScaleCap, a detailed image captioning pipeline that uses multiple models, including an LLM and an LVLM. It first generates an initial caption and identifies golden sentences, highly likely non-hallucinatory sentences within the generated caption, based on contrastive sentence ratings. Contrastive sentence rating is a strategy for evaluating the factual precision of a sentence by comparing output token probabilities between multimodal decoding and language-only decoding. Then, ScaleCap further obtains visual information by iteratively requesting detailed descriptions for each object mentioned in the golden sentences (referred to as heuristic question-answering module). By alternating between contrastive sentence rating and heuristic question answering, ScaleCap collects reliable sentences and finally summarizes them using an LLM. The authors demonstrate the effectiveness of ScaleCap through reconstruction and pretraining experiments.", "questions": "oes the proposed method outperform existing methods across multiple evaluation metrics?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is clear and well-organized overall.\n2. The problem addressed in the paper is interesting and timely.", "weaknesses": "1. The authors use image reconstruction and pretraining experiments to demonstrate the effectiveness of the proposed method. However, there are still many evaluation approaches remaining for assessing detailed image captions. For example, GPT-based methods [1,2] and QA-based methods [3] could be leveraged. I strongly encourage the authors to incorporate these additional evaluation metrics.\n2. Several previous studies have proposed detailed image captioning systems that involve multiple deep learning models [3, 4]. Comparisons and discussions with these works are needed to better position the proposed method within the existing literature.\n\n[1] Petryck et al., \"ALOHa: A New Measure for Hallucination in Captioning Models\"  \n[2] Chan et al., \"CLAIR: Evaluating Image Captions with Large Language Models\"  \n[3] Lee et al., \"Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage\"  \n[4] Get et al., \"Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)