Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models
Authors: Tiezheng Zhang
Yitong Li
Yu-Cheng Chou
Jieneng Chen
Alan Yuille
Chen Wei
Junfei Xiao
Venue: NeurIPS 2025 poster ()
Keywords: Auto-Encoder
Diffusion Model
Vision-Language-Vision
Primary area: deep_learning

Abstract: Building state-of-the-art Vision-Language Models (VLMs) with strong captioning capabilities typically necessitates training on billions of high-quality image-text pairs, requiring millions of GPU hours. This paper introduces the Vision-Language-Vision **(VLV)** auto-encoder framework, which strategically leverages key pretrained components: a vision encoder, the decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large Language Model (LLM). Specifically, we establish an information bottleneck by regularizing the language representation space, achieved through freezing the pretrained T2I diffusion decoder. Our VLV pipeline effectively distills knowledge from the text-conditioned diffusion model using continuous embeddings, demonstrating comprehensive semantic understanding via high-quality reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the intermediate language representations into detailed descriptions, we construct a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and significantly reduces data requirements; by primarily utilizing single-modal images for training and maximizing the utility of existing pretrained models (image encoder, T2I diffusion model, and LLM), it circumvents the need for massive paired image-text datasets, keeping the total training expenditure under $1,000 USD.

Vision-Language-Viiion Auto-Encoder: Scalable Knowledge Distillation from Diffusion Models
Tiezheng Zhang', Yitong Li,, Yu-Cheng Chou',
Jieneng Chen',
Alan Yuille',
Chen Wei,,
Junfei Xi
Johns Hopkins University
Tsinghua University
'Rice University
Project Lead
https://lambert-
Abstract
Building state-of-the-art Vision-Language Models (VLMs) with strong captioning capabilities typically necessitates training on billions of high-quality image-text pairs, requiring millions GPU hours. This paper introduces the Vision- LanguuLa Vision (VLV) auto auto-encoder framework, which strategically leverages key pretrained components: a vision encoder, the decoder of Text-to-Image (T2I) diffusion model, and subsequently, a Large Language Model (LLM). Specifically, we estab- lish an information bottleneck by regularizing the language representation space, achieved through freezing the pretrained T21 diffusion decoder. Our VLV pipeline effectively distills knowledge from the text-conditioned diffusion model using continuous embeddings, demonstrating comprehensive semantic understanding via high-quality reconstructions. Furthermore, by fine-tuning pretrained LLM to decode the intermediate language representations into detailed descriptions, struct stute-of-the-as (SoTA) captioner comparable to leading models like GPT-4o and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and significantly reduces data requirements; by primarily utilizing single-modal images for training and maximizing the utility of existing pretrained models (image coder, T21 diffusion model, and LLM), it circumvents the need for massive paired image-text datasets, keeping the total training expenditure under $1,000 USD.
VLV
VLV va. GPT-4o
eiting on skde of cours hoiding to art whiled soiming the rgt side
Reconstruction with Caption (FID)
book apau The and collar
couch
6.64
green
lamns. These
boolshed
6,20
frommeed pint im
sheve
thes nig
coach Thers wooder tacleis srigts coach. sake pakerad plpers othe
Training Cost (Est. GPU hour)
imag
dodd oweell armos magt andivallde
smooth
The
S1K
compueet mesatly aponnit
OPP-44c A wide dysid goi cut and sorufly white dlgg
streddod
1000
groon couch
spiling ing muafing
and liking
icc snurned 11
Inference (Captioned Images per USD)
wostenyoonne warl Diitrte finvtt micelt oag mard diaar oneayy The vooden flor autiwwe too (fowed
75000
300
Figure I: VLV matches GPT-4o's descriptive fidelity at three orders of magnitude lower cost. Left: VLV captures all salient objects, matching GPT-4o in coverage without hallucinations, yet better preserving their spatial layout. Right: On the thD-cost-throughpur plane, VLV reaches comparable FID, trains for orders-of-nne onders-of-magnitude less, and delivers vastly higher captions-per-dolle inference-proveee proving that detail-rich descriptions need not demand massive budgets.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Multimodal representation learning aims to capture meaningful semantic relationships between vision and language. Broadly, existing approaches can be categorized into three major paradigms based on the interaction between textual and visual modalities: (1) aision-languag models (VLMs), where images serve as inputs and text as outputs [50, 37. 5, 84, 68, 55, 52]; (2) contrastive learning frameworks [51, 83, 10], where image and text embeddings are aligned into a shared latent space through a contrastive objective; and (3) text-to-image generative models, where textual descriptions condition the generation of visual content [55, 52].
Traditionally, the first two paradigms, vision-language modeling and contrastive learning, have been predominantly utilized for learning robust multimodal embeddings. In contrast, text-to-image generative models, such as diffusion-based architectures [27], are generally considered generative tools rather than effective mechanisms for multimodal embedding learning. Although intuitively these generative models must implicitly encode detailed semantic relationships to produce coherent images, their potential for multimodal tasks like image captioning has not been fully realized.
Recent research suggests that text-to-image generative models indeed capture rich, nuanced seman- tic structures [66, 69], highlighting potential opportunities in applying the
approach [82]. Rooted in cognitive science, this idea has long argued that perception works by imagining the hidden causes of a signal and selecting the one that best "explains" it. Motivated by this insight, our work demonstrates how pretrained text-to-image diffusion models can effectively transfer their inherently rich multimodal representations to downstream vision-language tasks such as cap- tioning and VQA, where text-to-image diffusion models "imagine" the image, whose corresponding multimodal representation serves the "best explanation".
Specifically, we introduce novel architecture termed the Vision-Language Vision" (VLV) autoen- coder. In this framework, an open-source pretrained diffusion model, specifically Stable Diffusion 2.1 [55], is used as powerful frozen diffusion decoder. We distill knowledge from this decoder into a bottleneck representation through a regularization process on the language embedding space produced by an encoder [72]. Next, these continuous intermediate representations are decoded through a pretrained LLM decoder [77] after alignment, generating detailed captions. Our approach achieves captioning performance competitive with leading proprietary models, including GPT-4o [1] and Gemini 2.0 Flash [22], while utilizing significantly smaller, open-source models.
Our methodology also exhibits strong scalability: We obtain substantial performance improvements when scaling the training dataset from 6M to 40M images. Notably, by primarily leveraging single- modal images, the data collection approach is much less of burden compared to extensive paired image-text datasets. Adding up maximizing the utility of existing pretrained models, training costs remain below $1, $1,000 USD (less than 1,000 GPU hours), significantly enhancing accessibility and promoting broader innovation within the vision-language research community.
Additionally, explore emergent properties of the proposed VLV autoencoder: a) semantic richness, where learned embeddings encode detailed semantic aspects, including object 3D pose and orientation, resulting in robust spatial consistency; and b) compositional generalization, achieved by concatenating caption embeddings from distinct images, allowing the model to disentangle foreground objects from backgrounds effectively and compose novel, coherent, and visually plausible images.
In summary, the primary contributions of this work are as follows:
We introduce Aision-Language able and efficient knowledge distillation from pretrained ext-to--mage diffusion models. This approach learns language-semant representations only using amsge-based training.
(VLV)
a novel framework for seal-
The construction of a lightweight yet effective LLM-based caption decoder, achieved by strategically integrating pretrained models, resulting in negligible training overhead.
Comprehensive experimental results validate that the proposed captioner exhibits highly competitive captioning performance relative to SoTA VLMs, such as GPT-4o, and surpasses other open-source models of comparable parameter counts.
An investigation into the emergent properties of the VLV framework, specifically highlight- ing the preservation of spatial semanties and advanced multi-image compositionality. These findings underscore the efficacy and potential of the learned representations.

---

2
Related Work
Visual Autoencoder (VAE) has long served as foundational method for unsupervised representation learning [19, 24, 26, 32, 61]. Variants such as VQ-VAE! [60, 54] and discrete VAE [53] extend this idea by learning discrete and structured representations. Although widely used for image tokenization in multimodal learning [21, 55, 80, 81, 41], their quantized latent spaces often entangle semantics and require co-trained decoders to be effective. Recent works, like De-Diffusion [69], replace the latent bottleneck with text sequences decoded by pretrained diffusion models, aiming for interpretability. ViLex [66] further pushes this by directly training visual embeddings through generative supervision from frozen diffusion backbones, bypassing token-level representations entirely. Despite these flosed-sourced advances, our VLV model the first to efficiently build a Dision-lanuuag-ee auto-encoder with all open-source modules with minimal training cost.
Vision-L.anguage Captioners. Recent advances in vision-language models (VLMs) have significantly advanced image captioning by leveraging large-scale image-text pretraining and powerful multimodal architectures. Some previous works [79, 37, 36, 62, 73] aligned visual encoders with language
decoders, while Flamingo [5], Kosmos [47, 29], and Shar [11] highlighted few-shot and interleaved vision-text capabilities. Recent models like GPT-4o [2], Gemini [22]. Qwen-Vi VL [8, 76], and LLaVA [42] combined instruction tuning with powerful language backbones for fluent captioning. Large-scale systems such as PaLI-X [12], mPLUG-2 [74], Intern VL [16], and CogVLM [63] scaled
model and data size to achieve top performance on COCO [14], Flickr [48]. NoCaps [3], and
TextCaps [57], while IDEFICS [33]. OpenFlamingo [6], Fuyu-8B [9]. and Baichuan- [39]
offered strong open-soured alternatives. Emerging models like Emu3 [65], NVLM [17], Pixtral [4], and Molmo [18] further demonstrated the effectiveness of diverse multimodal modeling strategies. Despite these advances, most models depend on massive image-text pairs and costly training. In contrast, our VLV framework distills knowledge from a pretrained diffusion model using single-modal image data, enabling high-quality captioning without requiring web-scale, high-quality labels.
Representation Learning with Difflusin Models. growing body of work has explored leveraging diffusion models for representation learning across diverse modalities and tasks [49, 67, 30, 59]. De-Diffusion [69] and ViLex [66] used frozen T2I models for aunuagge-aligned embedding learn- ing. Other works. like DreamTeacher [35], distilled diffusion model features into discriminative backbones, while DiffMAE [70] recast denoising as masked autoencoding. Several studies also demonstrated that diffusion models can serve directly as zero-shot classifiers [34] or that their intermediate activations encode linearly separable features [71]. the vision-language domain, SPAE [81] and RLEGG [85] bridged image-language understanding using semantic autoencoding and synthetic contrastive supervision, respectively. ODISE [75] and DIVA [64] used diffusion priors to boost open- vocabulary segmentation and CLIP's perception, while RepFusion [78] explicitly mined time-step features for classification. Finally, simplification studies like Deconstructing DDMs [15] revealed that even stripped-don down DAEs retain strong representational power. Unlike prior methods that require co- co-training of text and vision modules, handcrafted bottlenecks, or synthetic supervision, method directly transfers generative knowledge into latent space that supports both high-fideling reconstruction and competitive caption generation with minimal compute.
Method
In this section, we introduce our proposed pipeline, which employs vision- larion--annuac-viiiion (VLV) autoencoding to distill high-fide fidelity semantic information from images and subsequently decodes these semantics into descriptive captions using a multi-modal language model. We begin by outlining the pipeline architecture in $3.1. Next, in $3.2, we describe how we leverage pretrained diffusion model to encode images into compact, continuous semantic embeddings, eliminating the need for explicit image-text pairs during training. Finally, in $3.3. we detail how these embeddings are decoded into natural-language captions via alignment with a pretrained large language model (LLM).
3.1 Pipeline Overview
VLV aims to extract high-fidelity semantic information from images through a pretrained T2I diffusion model. Previous similar work [69] utilizes discrete text token of CLIP as latent representation directly and Guubee-SSoftax [31, 45] for optimization, resulting in training inefficiency and lack of fine- grained semantic details. In contrast, we train model using continuous embedding space for
3

---

Reconstruction
Diffusion Decoder
Traneformer
CLIP Text Encoder
Image Encoder
Encoder Decoder
MLP
U-Net
Caption Embeddings
Learnble Queries
Stage-1: Vision-Language-Viisie Vision Auto-encoding
Caption Decoder
with ousteor central Eocus has Wide, expressive amoer eyes Ballnd the cat, plaza with light brown stone povement posuisted by numerous pigcons, The backstop incluches a leeg,, omett cachwital on the lull. a tall deta led tower in the cantie, and palm trens on the right. The sun is shining lrighlyy with a slight flare visible in the upper...
dressed
bowlle
VLV Encoder
CLIP Text
Auto-
backdrop. The
Encoder
MLP
regressive
df
A
d
Stage-2: Captioning with Auto-regressive Decoder
Figure 2: Method Overview. Our method has two stages: 1) vision-language-viiste autoencoding for learning language semantics, 2) representation decoding into discrete language tokens through multi-modal LLM alignment. Our model has three major modules (i) VLV Encoder: visual backbone augmented with a lightweight multi-modal adapter maps an input image into continuous caption embedding with compact semantic information; (ii) Diffusion Decoder: frozen text-to-image diffusion model reconstructs the image; (iii) Caption Decoder: pretrained large language model with MLP projector decodes language-centric representations into comprehensive captions.
better training convergence, stability, and efficiency and decode the embeddings discrete language tokens like multi-modal LLMs to generate text tokens given encoded visual embeddings of images.
Our VLV encoder extracts continuous caption embeddings directly from images. Training is fully self-supervised: a frozen text-to-image diffusion model serves as the decoder, reconstructing each image from its caption embeddings. Because the text-to-image diffusion model is fixed, the encoder must embed all information necessary for faithful reconstruction, effectively distilling the diffusion model's 's rich visual knowledge into lightweight vision backbone, while eliminating the need for paired image-tex text data. Next, we fine-tune VLV encoder together with an LLM--ased decoder that maps them latural- captions. Since the caption embeddings obtained by the VI.V encoder are compact and encode only implicit semantics, we utilize a pretrained LLM to decode them into deseriptive image captions. The autoregressive architecture of the LLM and its rich linguistic knowledge enable it to generate natural, coherent sentences with flexible length. This alignment uses paired image-text data specified in $4.1.
3.2 Knowledge Distillation from Diffusion Models
Following sulf-supervised learning framework, this stage adopts symmetric auto-encoder archi- tecture that encodes to and decodes from latent tokens as information bottleneck. Given an image x xW x3 visual backbone produces visual tokens
N. xD. A linear projection followed by LayerNorm [7] maps them to o' E N. These tokens are concatenated with N, dummy which multimodal Trans- former encoder converts to contextual states hg = Enne (o) Since there no caption for supervision in this stage, we inject N₂ learnable query tokens q E NR on the Transformer decoder side; cross-attention with hE yields h Dee(g, hE) NDD. lightweight MLP projects these states to the channel dimension of the frozen CLIP text encoder in the diffusion model, producing the caption embedding receives as conditioning and is optimised only indirectly. Specifically, with latent 2n = E(x) and its noisy counterpart 2, /Ox - VI- 04 c, the frozen U-Net predicts the noise co t, 2); the encoder parameters are updated by the standard denoising loss
prompt embeddings Éprompe to form X [I'; Eprompt| E +
Φ(h) E X The text-to-image diffusion model D remains frozen; it
Licaoise = Ez,e,elee ce ce(2t,t,2)
(I)

---

The auto-encoder architecture forces visual encoder to distill all information required for faithful reconstruction into the compact caption embedding z. Instead of using image-text paired data, visual encoder learns the inverse 12T mapping process through pretrained T2I diffusion decoder, which contains rich cross-modal knowledge. Rather than discrete text token and Gumbel--oftmax, we use implicit and continuous embedding as latent for remaining detailed semantic information in compact way without losing fidelity. The faithful encoding performed in Stage-1 forms the foundation for high-quality understanding and captioning in Stage-2, ultimately enabling accurate reconstruction.
3.3
Caption Decoding from Language-centrie Representations
The aim of this stage decode intermediate representations into readable. high-quality captions. Previous structure design has fixed-length word tokens, contradicting with the inherent difference of complexities among all kinds of images, e.g., a picture of an apple and picture of a big city should have semantic complexities of different levels. The setting limits the effectiveness and flexibility of image encoding, result in losing the potential of faithful reconstruction. Thus we introduce our LLM-based VLV Caption Decoder, which can decode unlimited and length- -lethlflexible natural language descriptions of images from compact semantic embeddings.
we train our VLV encoder E and LLM decoder G with our image-text pair (zr,y). We first obtain the caption embeddings z E X via VLV encoder (E). Since is in the CLIP text-embedding space, we pass it through the frozen CLIP text encoder T, obtaining contextual representations c = T'(2) R R.d, A lightweight trainable MLP & dr - 12 then projects these vectors to the hidden size of causal language model G:
As shown in Section 3,
(cc) E RNN,xdu,, During training with paired image-text pair {zz, V1:T), the projected vectors e are prepended to the ordinary token embeddings of the caption, forming the input stream [e; Embed ),- Embed J. With positions corresponding to e masked out, we compute the autoregressive loss only on real words:
LIM =
Σ log Togr e, y<t);
where 0 = (F, s/, G} are the only trainable parameters; the CLIP text encoder T remain untouched. At inference, we compute = E(x) + c = T(z) + = ((() and feed the projected vectors e (without any text tokens) into the language model G, which autoregressively samples the caption. Thus this stage bridges the previous visual semantics to natural language with only a lightweight projection head, while fine-tuning E and G and keeping T frozen. This design lets compact latent embedding be flexibly decoded into human-readable captions of arbitrary length, while preserving fine-grained image semantics. And the progressive training-ant and-inference strategy achieves superior performance, as demonstrated empirically in Table 4.
4 Experiment
In this section, we first describe the experimental setup for both stages of VLV in $ 4.1. Next, we report quantitative results on text-to-image (T2I) generation (64.2.1), a human study of caption quality ($4.2.2), and vistal-question-answeiin
(VQA) benchmarks ($4.2.3). Finally, $4.3 presents two ablation studies: (i) a trainable-parann paraamerrrr study, varying the number of learnable queries for representation learning from information bottleneck and the progressive training strategy (i.e., progressively unfreezing encoder modules) in training the captioning decoder; and (ii) scalability study in the aspects of training data scale and captioning decoder model size.
4.1 Experimental Setup
Data Collection. From LAION-2B WwWN---Beenaestheeeee a subset of LAION-SB [56]. we curate a 40M image subset. For training stability we keep only images whose shorter side is greater than 512, aspect ratio in the range of 0.5 to 2, and watermark probability less than 0.5. The resulting images are used train the VLV antodee under image-only supervision, without any accompanying text. Next, we query Gemini-2.0 Flash [58] to generate captions for 6M images in our dataset, producing aligned image-text pairs that fine-tune the lightweight language decoder. An overview for crafting our image-text pairs dataset used in alignment training is shown in appendix. Despite using only 0.4% % (40M/10B) of the WebL.I dataset [13] used by De-Diffusion [69], our method still learns strong language-oriented semantics through the vision-language- vision auto- nnto--encoding pipeline.
5

---

Original Images
VLV
Stage-1
VLV Stage-2
Figure 3: Reconstruction with language semantics. For each original input image (top), we feed its caption embedding directly to the frozen diffusion decoder and obtain reconstruction (middle) that preserves high-level semantics and the-grained appearance cues. The same embedding is then decoded by the LLM; prompting Midjourney with that caption yields an image of high fidelity.
Guidance Scale
4.0
Users
Gemini 2.0 Average
Original [14]
16.62
9.90
12.69
14.49
Recap-7B [38]
12.26
7.70
10:16
11.82
Qwen2.5-7B [8]
5.00
5.07
5.03
LLaVA-v1.5- (42)
9.99
12.93
14.91
Doven2.5.VL--2. [8]
12.61
6.98
9.19
10.59
GPT-4o [2]
5.20
5.25
5.23
Florence-2 Large [72]
10.61"
7-51
9.95
11.35
VLV (Ours)
5.17
5.18
5.18
VLV (Ours)
11:47
6.64
8,56
9,90
Gemini 2.0 Flash [58]
12.82
5.87"
7.57"
8.77*
Table 2: Benchmark Captions Through Users
GPT--4 [2]
12.16
7,96
9.25
and VLM Rating. We asked human users and a state-of tfrte-- vision-languagg model (VLM), i.c., Gemini 2.0 Flash, to rate captions generated by different models, employing a scoring rubric ranging from 1 to 6. The evaluation criteria en- compassed semantic accuracy, linguistic fluency, and relevance to the corresponding images.
Table 1: Benchmark Captions Through Text- to-Image Reconstructions. We evaluate the cap- tions through FID scores (L), with image recon- truction. We use Stable Diffusion 3.5 Medium to reconstruct images with captions. Best results with open-source models are bolded: * best for all.
Training Details. When training our VLV auto auto-encoder, we initialize the image encoder part with Florence-2 [72] pretrained weights. The additional Ng 77 learnable queries are randomly initialized. We use AdamW W [44] optimizer with (B,, 52) = (0.9. 0.99) and decoupled weight decay of 0.01. Training runs for 200K steps with batch size 512 on 8 RTX 6000 Ada GPUs (~ 4 days). The learning rate starts at 5e-5 and follows cosine schedule [43]. We use Qwen-2.5 [77] pretrained models for initializing the L.M decoder. We train the captioning decoder with 100K steps, having the batch size of 64. The learning rate decays linearly starting at 1c-5, We use FP32 in autoencoder training to make models converge with stability, while the LLM decoder training uses BF16.
4.2
Main Results
4.2.1
Text-Conditioned Reconstruction with Captions
We assess caption quality by feeding each decoded caption Stable Diffusion 3.5 Medium [20] and computing the Fréchet Inception Distance (FID) [25] between the synthesized and original images on 30K samples from the MS-COCO 2014 validation split [14]. Captions are generated
with four state-of-the the-art art VLMs: Florence-2 [72], Qwen2.5-VL [8], Gemini 2.0 Flash [58], and GPT-4o [2]. Image synthesis employs the rectified flow-matching sampler using 40 inference steps and classifier-free guidance [28] scale from 1.0 to 4.0. As Table 1 shows, our captions achieve an FID essentially indistinguishable from GPT-4o's (difference < 0.5) and markedly lower (better) than those of Florence-2 and Qwen2.5-VI. indicating that our captions convey visual semantics on par with the strongest public baseline; only the closed-source Gemini 2.0 Flash attains a marginally better score. Figure 3 shows qualitative results on generated images by both caption embeddings and corresponding decoded captions, illustrating the faithfulness of our caption embeddings.
4.2.2 Captioner Arena: Rating with VLMs and Humans
We benchmark caption fidelity by comparing state-of-the-art vasion-language models (VLMs) with human raters under the identical three-criterion rubric-cone ccoverco no hallucination, and spatial- layout consistency- and the 7-point rating scale (0-6) introduced in Appendix. A random sample

---

Accuracy (%)
Benchmark Shot
Florence-2 Large
Owen-2.5-VLL7BB
Gemini 2.0 Flash
VLV (ours)
0
58.74
61.74
62.52
58,55
VQAv2
4
60.05
62.37
63.36
61.72
32
63,28
63.77
64.05*
63.60
46.80
45.83
46.34
45.31
OK-VQA
55.36
55.11
56.48
54.10
32
59.72
61.20
62.31
60.25
Table 3: Few-shot VQA Evaluation(Texttorn We evaluate the VQA accuracy (%) on VQAv2 and
OK-VQA under zero-shot few-shot settings. DeepSeek-V answers only from caption text. By 32-shot, VLV matches the best open-source model (Qwen-2.5) and sits within I percentage of the overall leader (Gemini 2.0 Flash), despite being far cheaper to train and run.
N₂
FID
Trainable modules
1.0
2.0
3.0
4.0
16
5.72
MLP only
14.27
9.71
11.87
13.22
32
5.60
MIP LLM decoder
12.22
7.55
9.58
10.84
77
5.30
MLP+ LLM decoder + VLV encoder
11.47
6.64
8.56
9.90
Table 4: Ablation Studies. Left: Effect of the number of learnable query tokens (N₂).. Right: Effect of unfreezing modules in Stage-2; both reported by FID (L).
# Images(M)
1.0
2.0
3.0
4.0
Decoder
1.0
2.0
3.0
4.0
6 11.38
9.01
10.33
10.81
Qwen-2.5 0,5B
14.70
9.37
11.26
12.45
18
10.14
7.57
8.41
8.78
Qwen-2.5 1.5B
12.25
7.30
9.16
10.26
40
9.71
7.22
7.70
7.84
Qwen-2.5 3B
11.47
6.64
8.56
9.90
Table 5: Scalability in Data and Decoder Scale. FID (4) computed at guidance scales 1 1-4 for (left)
training-dat data size and (right) caption-decode size. VLV demonstrates strong scalability.
of 200 images from the MS-COCO 2014 validation split [14] is paired with captions produced by Qwen-2.5 VL, GPT-4o, and VLV. Each mmage-caption pair is then evaluated by one VLM judge (Gemini 2.0 Flash) and three independent human raters. For every pair the judge returns a single score E .O..... ; the same rubric is applied by the human raters. Table 2 shows that VLV matches GPT-4o within < 0.05 points on the 0-6 scale, surpasses Qwen-2.5-VL- preferred by one of the three human raters. These results confirm that our caption embeddings yield human-level captions while remaining competitive with the strongest commercial VLMs.
by 0.15 on average, and
4.2.3 Text-Only QlustioonAAnserte with Captions
Because our caption embeddings capture both global semantics and fine- grained appearance cues, assess their effectiveness open-ended Asioo-language tasks using VQAv2 [23] and OK- VQA [46] validation sets. Following Wei et al. [69], each caption is inserted as image context in a large-language-mode (LLM) prompt, which the I.LM then completes answer the visual question. An answer is deemed correct only if exactly matches the ground truth. We evaluate our captions with DeepSeek-V V3 [40] in both zero-shot and few-shot settings, without any additional fine-tuning. Table 3 shows the zero- 4-, and 32-shot accuracies using captions generated by different VLMs. In strict zero-shot, VLV trails the best baseline by roughly three percentage points, yet gains the most from extra in-context examples (about five points on VQAv2 and fifteen on OK-VQA),so that by thirty-two shots it lies within single point of the state of the art. Although VLV not the top scorer in every setting, it reaches comparable while training at lower cost, underscoring its scalability.
4.3 Ablation Studies
We conduct two complementary ablation studies in this section. (1) partmaabee- analysis. We probe the impact of trainable parameters by (i) varying the dimensionality of the learnable queries when training VLV auto-encoder and (ii) selectively unfreezing individual modules of the VLV encoder while training the LLM decoder. (2) Scalability analysis. We test how performance scales by (i) scaling the training corpus from 6M to 18M and 40M images, and (ii) increasing the size the autoregressive captioning decoder from 0.5 B to 1.5 B and 3 B parameters.
Progressive Training Leads Better Performance. Herein, we train VLV with different trainable parameters settings to explore the trade-off between performance and training cost. Stable Diffusion
7

---

Figure 4: Representation Learning Beyond Text: Spatial Preservation. The figure compares the original images (left) with those reconstructed by our embeddings. The accurate 6D poses of individual objects and the relative spatial configurations among multiple objects demonstrate the method's strong capability in capturing spatial structure.
Average Deviation
0.16
Im)ges(M)
18
Angle (L)
0.1564
0.1287
0.1227
0.1016
0.12 0.14
Center ())
0.1625
0.1498
0.1402
0.0988
Scale ())
0.1775 0.1773 0.1835
0.1222
Table 6: Quantitative Comparison of Spatial Aware- ness. With more supervision images, VLV demonstrates improved spatial awareness. We evaluate this by mea- suring the L1 distance deviation between the bounding boxes of original and generated images with identical labels, as detected by Gemini 2.0 Flash [22].
10
25
Images (M)
Figure 5: Continual Spatial Represen- tation Learning VLV enables continual
3D spatial representation learning.
2.1's CLIIP text encoder accepts at most 77 tokens, and our default uses this full budget (N,=77). We halve the number of learnable queries to N2=16, 32 and gauge the impact by reconstructing MS-COCO 2017 test images from the resulting caption embeddings and reporting FID. In our second stage training, we progressively unfreeze the modules, starting with MLP first followed by the LLM decoder and finally the VLV encoder to see how many extra parameters are worth optimizing. Table shows how reconstruction FID and caption quality improve smoothly with more trainable weights, clarifying the trade-off between performance and training cost.
Scalability of VLV. During training of the VLV auto-encoder we save intermediate checkpoints after the model has processed 6M and 18M images. To assess scalability, each checkpoint is used to extract caption embeddings for the 30 K images in the MS-COCO 2014 validation split described in $4.2.1. These embeddings are passed to the frozen diffusion decoder to reconstruct the images, and the resulting FID scores reported in Table 5. We further probe model capacity by replacing the Qwen-2.5 3B caption decoder with its 1.5 B and 0.5 B variant while keeping all other components fixed (same table). In both cases FID degrades smoothly data or decoder size is reduced, confirming that VLV benefits predictably from more training images and a larger language decoder.
4.4 Emerging Properties
4.4.1 Representation Learning beyond Text: 3D Visual Awareness
Besides rich details, we also find our embeddings have scalable spatial awareness. During training, as the diffusion decoder is exposed to a larger pool of images, the model steadily refines its spatial priors. To quantify this effect, we use Gemini 2.0 Flash to recover 3D bounding boxes for the primary objects in original images and compare them with boxes reconstructed from caption embeddings.
Table 6 show a consistent reduction in pose estimation errors, and the examples in Figure illustrate that VLV not only captures the poses of individual objects more accurately but also better preserves their spatial relationships. These results demonstrate that VLV effectively translates larger training image sets into sharper spatial understanding, visualized in Figure 5.
8

---

Figure 6: Emerging compositionality with multi-image semantics. Given two input images- a Siherian cat the left edge of the frame and either (above) Van Gogh-style painting (hottom) Mount Fuji landscape-we truncate and concatenate their caption embeddings and feed the composite vector Stable Diffusion 2.1. The generated outputs faithfully preserve the cat's spatial layout while transferring the desired artistic style or background, without any extra fine-tuning or text prompts.
4.4.2 Compositionality with Multi-image Semantics
VLV semantic representation space exhibits strong compositional properties across multiple images, illustrated in Figure In the leftmost example, we begin with images: (i) photograph of Siberian cat positioned on the left side of the frame, and (ii) a Van Gogh-style painting. By truncating the trailing tokens of each caption embedding and concatenating the resulting vectors, we create joint embedding that is fed to Stable Diffusion 2.1. The synthesized output preserves the spatial layout of the cat while adopting the Van Gogh style, indicating that our embeddings encode both content (e.g., object identity and position) and style (e.g., artistic rendering). Notably, this compositional behavior emerges without any additional fine-tuning or reliance on text prompts. Further style transfer examples, including cartoon and Disney-style Shiba Inus, well as try-on scenarios like Shiba Inu or a man wearing sunglasses and a man trying on hoodie or simple compositional of two objects like a Shiba Inu sitting in front of Fuji Mount and a sunglasses on hat.
5 Conclusion
In this paper, we presented the Lssion-Language-issinn (VLV) auto-encoder, novel framework for scalable and efficient knowledge distillation from open-source pretrained text-conditioned diffusion models. By leveraging a strategically designed two-stage training process, VLV distills semantic- rich representations from frozen diffusion decoders into compact, continuous embeddings, and subsequently translates these embeddings into detailed natural language captions using an open-source pretrained Large Language Model. Our experiments demonstrate that VLV achieves state-of-the-e captioning performance comparable to leading models such as GPT-4o and Gemini 2.0 Flash, while dramatically reducing training costs and data requirements. Notably, our method primarily utilizes manglemmoal images, significantly enhancing accessibility by maintaining training expenditures under $1,000 USD. Additionally, we explored the emergent properties of our framework, highlighting its strong spatial consistency and advanced compositional generalization capabilities. We believe efficiency, effectiveness, and interpretability of VLV pave promising pathways for future research in scalable and cost-effective multimodal learning.
Limitations & Future Work. As our training data is filtered with aesthetic score, VLV performs poorly on OCR (Optical Character Recognition) tasks due a lack of data with texts or watermarks; augmenting with document and street-view images or adding a lightweight OCR branch should somehow improve the performance on OCR scenarios. Another thing is that we are using the Stable Diffusion 2. as the generation decoder in our pipeline which is outdated also limits the transferable knowledge, limiting our upper bound. so re-distilling from recent state-of-the-an diffusion models such as SD 3.5 or FL.UX is an incoming work. Moreover, extending VLV to video modality is also worthy to explore since videos offer more dynamics and could emerge stronger spatial representations well physics- based learning for understanding comprehensive world semantics.

---

Acknowledgments
We thank the anonymous reviewers for their insightful comments. This work was supported in part by grant from Amazon through the A12AI program, and by the Office of Naval Research under award 6000142412696.
10

---

References
[1] OpenAl GPT 4o Team. Gpt-4o system card, 2024.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint
2023.
[3] Harsh Agrawal, Karan Desai, Yufei Wang. Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In ICCV, pages 8948-888 8957889972222 2019.
[4] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervel, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024.
[5] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35:23716-23736, 2022.
[6] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openftamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 01390, 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.000 004500 2016.
[8] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report, arXiv preprint arXiv:2502. 13923, 2025.
[9] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar. Introducing our multimodal models, 2023.
[10] Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
[11] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi- modti- models with better captions. In ECCV, pages 370-387. Springer, 2024.
[12] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo. Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprini arXiv:2305. 18565, 2023.
[13] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language- image model. arXiv preprint arXiv:2209.0679) 2022.
[14] Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: arXiv:1504.00 1504.00355 2015.
[15] Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self supervupeeii learning. arXiv preprint arXiv:2401.14400 14404, 2024.
[16] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguiste tasks. In CVPR, pages 14185-24199 2024.
[17] Wenliang Dai, Naycon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shocybi, Bryan Catanzaro, and Wei Ping. Nvim: Open fronticr-class multimodal 1mss. arXiv preprint arxi 1402, 2024.
[18] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of the-the-tt multimodal models. arXiv preprint arXiv:2409.1714. 17146, 2024.
11

---

[19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings the 2019 conference the North American chapter of the association for computational linguisties: human language technologies, volume (long and short papers), pages 4171-41888 2019.
[20] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URI. https://arxiiv org/abx/2405.0:206. 2, 2024.
[21] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883, 2021.
[22]
Google.
Introducing gemini 2.0: new ai model for the agentic era, 2024. Accessed: Dec 2024.
[23] Yash Goyal, Tejas Khot, Douglas Summers- Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pages 16000-16009, 2022.
[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurIPS, 30, 2017.
[26] Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. NeurIPS, 6, 1993.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33:6840- 6851, 2020.
[28] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.12598, 2022.
[29] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. NeurlPS, 36.72096-72109, 2023.
[30] Drew Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K Lampinen, Andrew Jaegle, James I. McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. Soda: Bottleneck diffusion models for representation learning. In CVPR, pages 23115-23127, 2024.
[31] Eric Jang, Shixiang Gu and Ben Poole. Categorical reparameterizztion with gumbel-softmax. arXiv preprint 1rXiv:1611.01 144, 2016.
[32] Diederik Kingma, Max Welling, al. Auto-encoding variational hayes, 2013.
[33] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. NewrlPS, 36:7168:3 36:71683-71702 2023.
[34] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusson model is secretly a zero-shot classifier. In ICCV, pages 2206-2217,2023
[35] Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Dreamteacher: Pretraining image backbones with deep generative models. In ICCV, pages 16698 16708, 2023.
[36] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre-training with frozen image encoders and large language models. In /CML, pages 19730-19742. PMLR, 2023.
[37] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified dision-langual understanding and generation. In ICML, pages 12888 12900. PMLR, 2022.
[38] Xianhang Li. Haoqin Tu, Mude Hui, Zeyu Wang. Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. What if we recaption billions of web images with Ilama-3? arXiv preprint 240:24406000777 2024.
12

---

[39] Yadong Li. Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song. Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Da Pan, Shusen Zhang, Xin Wu, Zheng Liang, Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang. Kaicheng Yu. Tao Lin, Jianhua Xu, Zenan Zhou, and Weipeng Chen. Baichuan-ome ommi technical report. arXiv preprint axiv:2410008866. 2024.
[40] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
[41] Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. NeurIPS, 36:4382 4395, 2023.
[42] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. NeurIPS, 36:34892-34916, 2023.
[43] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, arXiv preprint (eXiv:6608.0388. 2016.
[44] Ilya Loshchilov and Frank Hutter. Decoupled axXiv:171..05001, 2017.
weight decay regularization. arXiv preprint
[45] Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint 160p:11110007222 2016.
[46] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pages 3195-3204 2019.
[47] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[48] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phras correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[49] Konpat Preechakul, Nattanat Chatthee. Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In CVPR, pages 10619-10629, 2022.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, pages 8748-8763. PmLR, 2021.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[52] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint ar arXiv:2204.06122 1(2)3, 2022.
[53] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh. Scott Gray, Chelsea Voss, Alec Radford. Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In JCML, pages 8821-8831. Pmlr, 2021.
[54] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals, Generating diverse high-fidelity images with va-vae-2. NewrlPS, 32, 2019.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffuusion models. In CVPR, pages 10684-10695, 2022.
[56] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. NeurIPS, 35:25278 25294, 2022.
[57] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanprect Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part II 16, pages 742-758. Springer, 2020.
[58] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11885 2023.
13

---

[59] Changyao Tian. Chenxin Tao, Jifeng Dai, Hao Li, Ziheng Li, Lewei Lu. Xiaogang Wang. Hongsheng Li, Gao Huang, and Xizhou Zhu. Addp: Learning general representations for image recognition and generation with alternating denoising diffusion process. arXiv preprint arXiv:2306.05223, 2023.
[60]
Aaron Van Den Oord. Oriol Vinyals, et al. Neural discrete representation learning. NewrIPS, 30, 2017.
[61] Pascal Vincent, Hugo Larochelle, Isabelle Lajoic, Yoshua Bengio, Pierre AntrinAntonne Manzagol, and Léon Bottou, Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. JMLR, 11(12), 2010.
[62] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022.
[63] Weihan Wang. Qingsong Lv. Wenmeng Yu, Wenyi Hong. Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogylm: Visual expert for pretrained language models. NewrIPS, 37:121475- 121499, 2024.
[64] Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv preprint arXiv:2407.0117, 2024.
[65] Xinlong Wang. Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need, arXiv preprint arXiv:2409. 2024.
[66] XuDong Wang. Xingyi Zhou, Alireza Fathi, Trevor Darrell, and Cordelia Schmid. Visual lexicon: Rich image features in language space. arXiv preprint 2rX 2024.
[67] Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan. Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. Infodiffusion: Representation learning using information maximizing diffusion models. In ICML, pages 36336 36354. PMLR, 2023.
[68] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 2021.
[69] Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, and Jiahui Yu. De-diffusion makes text strong cross-modal interface. In CVPR, pages 13492-13503, 2024.
[70] Chen Wei, Karttikeya Mangalam, Po-
Po-Yao Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang. Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In ICCV, pages 16284- 16294, 2023.
[71] Weilai Xiang, Hongyu Yang, Di Huang, and Yunhong Wang. Denoising diffusion autoencoders are unified self-supervised learners. In ICCV, pages 15802-15812, 2023.
[72] Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for a variety of vision tasks (2023). URL https://arxiv. org/abs/231 1.0644,, 2023.
[73] Junfei Xiao, Zheng Xu, Alan Yuille, Shen Yan, and Boyu Wang. Palm2-vadapter; progressively aligned language model makes strong vision-language adapter. arXiv preprint arXiv:2402. 10896, 2024.
[74] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al. mplug-2: A modularized multi-modal foundation model across text, image and video. In ICML, pages 38728 38748. PMLR, 2023.
[75] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pages 2955-2966, 2023.
[76] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:24/2.151 151 2024.
[77] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao. Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang. Qin Zhu, Rui Men. Runji Lin. Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhearu Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2511: 2024.
14

---

[78] Xingyi Yang and Xinchao Wang, Diffusion model as representation learner. In ICCV, pages 18938-18949, 2023.
[79] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv:2205.01917 2022.
[80] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2006. 10789, 2(3):, 2022.
[81] Lijun Yu, Yong Cheng. Zhiruo Wang. Vivek Kumar, Wolfgang Macherey. Yanping Huang, David Ross, Irfan Essa. Yonatan Bisk, Ming-Hsuan Yang. et al. Spae: Semantic pyramid autoencoder multimodal generation with frozen 1lms. NeurIPS, 36:52692-52704, 2023.
[82] Alan Yuille and Daniel Kersten. Vision bayesian inference: analysis by synthesis? Trends in cognitive sciences, 10(7):301-308, 2006.
[83] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss language image pre-training. In ICCV, pages 11975-119999 1996, 2023.
[84] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, pages 5579-5588, 2021.
[85] Liming Zhao, Kecheng Zheng, Yun Zheng, Deli Zhao, and Jingren Zhou. Rleg: Vision-language repre- sentation learning with diffuusin based embedding generation. In ICML, pages 2023.
PMLR,
15

---

Appendices
A Data Processing
17
VQA Analysis: Are "Ground Truth" labels really ground truth?
18
Vision-Language- Vision Autoencoding Does Help
18
Caption Evaluation with SoTA Multi-modal LLM (Gemini)
19
E
Qualitative Results: Reconstruction from Captions
20
Dataset & Model License
22
F.1 Training Datasets
22
F.2 Testing Datasets
22
F3 Pre-trained Models
22
16

---

A Data Processing
This section details data collection and filtering procedure. We annotate subset of the corpus with Gemini 2.0 Flash[58). Figure 7 shows the whole pipeline how we obtain our data for Stage-1 and Stage-2. 2. Figure 8 provide the token length distribution of our captions used for training Stage-2.
Art
Nature
Ratio X
Water Mark
Celebrity
Animal
Landmark
are perceptive visual assistant. Given an image, your task is describe in exhaustive detail. Your description should include the overall scene and setting, objects present and their relative positions, people or animals (including appearance, clothing, expressions, and puses), colors, textures, lighting, shadows, any actions occurring, and mood atmosphere. Describe you without summarizing inferring beyond visible content. Provide only caption without any additional text line breaks.
life painting depicts blue, cylindricul iar with slightly domed resting flat, light orange or peach-colored surfuce. the right of the jar, two reddish-orange peaches with hints of yellow positioned, one slightly behind and the right of the other....
Gemini 2.0 Flash
Celebrity: A mature woman with short, light
blonde or grevish hair, widely recognized as Jamie Lee Curtis, stands facing slightly her left, with her placed on hips. She wearing black outfit that appears to a jumpsuit or top and pants, with planging neckline revenling some eleavage....
Figure 7: Data Filtering Principles. We filter and collect 40M images from M--ON-B--e----ee We apply filtering based on the image resolution and aspect ratio to ensure the image quality and then prompt Gemini 2.0 Flash with image-conditioned templates generate rich, descriptive captions.
Distribution Caption Token Lengths
Mean: 226.82
Median: 226.00 00
600000
500000
Requency
400000
300000
100000
Number of Tokens
Figure 8: VLV Captions' Length Statisties. Histogram token counts for all captions (our ~6M image-text paired data, used for stage-2 captioning). Most captions fall in the 170 280 token band, with mean == 226.82 (red dashed) and median 2 226 (green dashed).
17

---

with three bright orange oranges, each giisennig with water droplets. The bowl, which appears to be made of metal or a similer reflective material, has dark, almost black interior. The oranges of the bowl, with the one the foreground taking up the majoriyy the frame. The other two oranges are partially visible in the background, slighlly out of focus. The water droplets on the oranges create sparkling effect. adding Image. The lighting is soft dittused. casting gentle shadows and highlighling the textures of the oranges and the bowi. The overall mood is fresh and inviting, emphasizing the natural beauty the fruit. colors oranges contrasting against the dark background of the bowl. The compoeition is simple yet effective, drawing the viewers attention to the oranges and their natural beauty, The focus sharp the oranges the foreground. while background slightly blurred, creating depth of field that draws the viewer's eye to the main subject The image evokes senee of simplicity and neturainess, highlighting the beauty of everyday otjects. The overall impression
Caption by VLV: close-up
positinedd in the center
of freshness and vibrancy the
saturated, with bright orange
freshness, simplicity, and natural beauty.
Caption Gemini: A close-up captures several bright orange oranges with tiny water droplets clinging to textured surfacee, stusted
Question: What states are these grown in?
grey metalic bowl. bow's rim is visible along the bottom left quadrant of the image, showing slight discoloration and wetness from the condensation water. The oranges fashion, with one light is distriutted unevenly. with highlights on the wet parts of the oranges and the bowl, and shadows cest areas where they touch. The overall color palette dominated by oranges, with silver gray
Answer: Florida, California, Arizona
arranged others partially visible around suggesting group.
clustered
Gomini Answer: Florida, California
Ground-Truth:
providing neutral centrast. due to the visible pares on the orange peels and the water droplets adding a reflettive quality. The maod is fresh and inviing, potentially evoking sense of healthy eating retreshing etrus. the left side the photo is text, oriented vertically in a thin, sans-sert! font, that reads l'Abi Booth Photography.
texture appears rough
['Morrdd califoriaa. "forida california", "Nordda califomiaa. 'foridaa california". "california", "california", "califomia, 'califomia', "flordda, "Rorida"]
Figure 9: OK-VQA Example. Both caption and Gemini caption do not mention the states information. But our caption not only capture the oranges but also the number of oranges. Our answers contain the right ones highlighting in LimeGreen.
B VQA Analysis: Are "Ground Truth" labels really ground truth?
Following Wei et al. [69], we evaluate on OK-VQA with DeepSeek- V3 [40] under the strict exact- match metric. Our raw score is 45.31% (2,295 / 5,064), trailing the Gemini 2.0 Flash caption baseline of 46.34% by 1.03% (52 questions). Among the 526 cases where Gemini marked correct and our model wrong, we compute moswer-answa cosine similarity in CLIP space and relabel pairs with similarity 0.8, recovering 94 additional correct answers. The adjusted accuracy therefore 47.17% This shows that the apparent deficit stems mainly from lexical mismatches rather than missing visual content. We show an example (one of the 94 cases) Figure 9.
C Vision-Language- Vision Autoencoding Does Help
We do an ablation study of the stage-1 Vision- Eision-Language- Vision autoencoding. To be specific, we only train our Stage-2 with pretrained our VLV Encoder, and assess the generated captions with T2I tasks. Table 7 reports the resulting FID scores (4) on MS-COCO 2014. Skipping Stage-l (first three rows) yields very poor fidelity, even larger decoders cannot compensate, whereas with Stage-I training (grey row) drops FID to 12.2, confirming its critical role.
Decoder
1.0
2.0
3.0
4.0
Qwen-2.5 0.5B
57.29
64.02
66.92
68.18
Qwen-2.5 1,5B
49.15
56.66
59.15
60.14
Qwen-2.5 3B
45.63
51.03
53.28
54.40
Qwen-2.5 2.5 3B + Stage-1
12.22
7.55
9.58
10.84
Table 7: Effect of Stage-1 training on FID (!). The gray row demonstrates that our Vision-Languaae- Vision auto-encoding pipeline makes the encoder distill the knowledge from the text-conditioned diffusion model effectively and efficiently. This leads the effectively
18

---

"The image shows black while cal lying on top laptop. cal appears be resting or lounging with body streiched out scroes the surface of the laplop. The laplop is open, dieplaying Its keyboard, and the cat's head is posticned near the upper part of the laptop screon. The background seoms to ba indoors, possiblyy on a dask or table. The cat has a fuffy coal with a mix of black and white fur, and its eyes are partilly diosed glving a relased appearance."
Qwen-2.5 caption:
Retlng::
Explanation: "The caption acourately describes the image, covering the main elements like
laptop, and setting. There are haluzinations.
their refetionatip, also mentions
apoeerance and and white with long and whiskars lying strotched out insice an open laptop. cat's body cocupies the entire space where the screen would be, with its heed resting on left side the laplop. Its ears are parked up, and has an alart expression with its yollow eyes slightly narrowed. The keyboard of the loptop is visible in the foreground. featuring a silver oolor with white keys. The lighting bright and even, casting soft shadows around the edges the cat's fur. The sexture of the cats fur appears silky and The moed lighthearted and quiiey, suggesting
GPT-40 caption:
smooth, while the laptop has metalic sleek
and playful of laplop resting place
Rating:
Explanation: "The caption acourately descr the key elements of the image, including can's appaarancc,. postton, and on, as well as the laptop's features and overall setting. It also correctly identiiles the spatial relationsips between the abjects.
VLV caption: "A clase up shot shows a black and white cat lylng on top of laptop, with the iaptap's is kaybaard visible in the foreground. The cal's for is mix of black and white, with prominent while patch around its face and chest. eyes
its turned siigttly striking yellow, and whiskers
white. The posture is relaxed, plain white
beckground The laplop is silver and a gray keybuard. and laptop's surtace. wall. lighdin is satt dittused, creating suttle shadows on the avenall mood is calm and domastic. The focus sharp on the cal slightly blurrad The textura of the cat's fur appears soft and fufty contrasting with the smooth surface of the loptop. The image is well-zom
the laplnp, while the background
with the ual posiiooedd in the venter of the frame, drawing the viewer's attention to its teatures and the laptop's keyboand." Rating:
Explanation: "The caption provides comprahonsive reserptton image, covering cat's appearance, laplop, overall sieene aucurately describess the colors, lestures, and spotisl reletionehipe belween the objects, There are no halucinaliona""
Figure 10: Captioner Arena Example. All captions show the correct objects without hallucinations. Both our caption and GPT-4o caption show the spatial relationship while 2wen-2.5 VL does not.
D Caption Evaluation with SoTA Multi-modal LLM (Gemini)
We assess caption quality by querying Gemini 2.0 Flash with a tailored rubric. Figure 10 displays an evaluation case with, together with Gemini 2.0 Flash's rationale, confirming that our captions are on par with those from GPT-4o.
Your role is to serve as an impartial and objective evaluator of an image caption generated by Large Multimodal Model (LMM). Based the single image input, assess the caption on three main criteria:
1. Coverage of image elements how well the caption mentions the salient objects, their attributes, actions, and contextual details.
2. Absence of hallucinations the caption must not invent objects, attributes, counts, spatial relations, or other details not present or implied by the image.
3. Object spatial layout consistency whether spatial relationships (left/rghtt, above/below, front/behind, center, buckground/foregronnd) are described accurately.
Any incorrect or invented spatial relation is hallucination.
Omitting obvious spatial relation reduces coverage.
Stating a relation that is ambiguous or uncertain in the image is also halluci-
nation.
Evaluation protocol
Start with a brief explanation of your evaluation process. Then assign one rating using the scale below. Output only the rating number-no extra text, symbols, or commentary.
Comprehensive coverage, correct spatial layout, no hallucinations
5 Very informative, correct spatial layout, no hallucinations, minor omissions
4 Moderate coverage, correct spatial layout, no hallucinations, several omissions
3 Limited coverage, minimal spatial detail, no hallucinations
2 Informative but contains at least one hallucination (object or spatial)
1 Limited coverage and at least one hallucination (object or spatial)
0 Not informative and/or multiple hallucinations
19

---

E
Qualitative Results: Reconstruction from Captions
We show some qualitative results of our captions MS-COCO 2014 validation split in Figure 11, Figure 12, Figure 13, Figure 14. In each figure, we show the original images and the reconstructed images generated by Text-to-Iman Image generation models with our VLV captions. We show the gen- eration results using Midjourney, FLUX. 1-dev and Imagen 3. The reconstructed images preserve comprehensive semantics, demonstrating our VLV can do high-quality, comprehensive captioning.
Raw Image
Midjoumey
FLUX. 1-dev
ImageFX
medium shot captures black bear standing in grassy, rocky area. The bear is positioned in the center of the frame, facing slightly to the right, with its head turned towards the right side of the image. The bear's fur is a solid, glossy black, and its ears are small and rounded. appears to be standing on its hind legs, with its front paws resting on the ground. ground around the bear covered color, ranging light gray to dark towards the top the frame. The lighting the image appears to soft and diffused, with harsh shadows. The colors muted and natural, with the black of the bear and the browns of the rocks and grasses dominating the scene. The overall mood of the image peaceful and natural, capturing moment of stiliness in the bear's environment. The bear's posture and expression suggest sense of calm and observation. The background is slightly blurred, drawing the viewer's attention to the bear the foreground. The image is well-composed, with clear facus on the bear and its surroundings. The texture the bear's fur and the rough surface of the rocks are visible, adding depth realism to image.
Caption:
grasses dry grasses small, brown rocks. The rocks vary in and green
taller grasses reaching up
Figure 11: VLV capture spatial layout. The caption shows bear's 's layout (in the center of the frame) in this image as well as the bear's 's posture (head turned towards the right side), showing VL.'s ability of capturing spatial layout.
Raw Image
Midjoumey
FLUX.1 1-dev
ImageFX
Caption: medium, eye-level shot depicts train car with the number \"1843V' painted it, positioned train track in front of train station. The train the side, The number 1'1843"" is prominently displayed on both the front and side of the car. The train track is visible beneath the train ear, and the train station is visible the background. The train station appears to be made of concrete and steel. The sky is overcast, and lighting soft diffused. The overall mood of image is calm and serene. The train the background, and the sky overcast. The lighting is soft and diffused, and the overall mood the image calm and serene.
predominantly gray with yellow accents on the front and red stripe running along
subject of image, and positioned
foreground. The train station visible the
Figure 12: VLV can capture text (OCR). VLV has reasonable OCR ability, even though the training set is heavily filtered (we filter the data by watermark probability less than 0.5). There is still potential to improve OCR performance with further training on more OCR-oriented data.
20

---

Raw Image
Midjournyy
FLUX.1-dev
ImageFX
Caption: The image shows luxurious outdoor bed setup on a sandy beach under a clear blue sky. The bed elevated on wooden platform and is covered with white bedspread. A neatly folded stack of white towela sits on top of the bed. The bed framed by white curtains that are tied back on either side, creating a sense of enclosure and privacy. The curtains are draped way that allows light filter through, adding soft glow to the scene. the background, the turquoise the horizon, with sand light-colored and appears be well-maintained. The lighting bright and sunny, casting soft shadows and highlighting the textures of the sand, wood, and fabric. The overail mood is serene and luxurious, evoking sense of relaxation and vacation. The composition is well-bslanced, with the bed as the focal point and the beach setting providing natural and inviting backdrop. There are no people or animals visible in the image. The atmosphere is calm and peaceful, suggesting perfect day the beach. The colors are vibrant and natural, with the white of the and curtains contrasting against the blue the sky ocean. The textures are varied, from smooth sand to the rough wood of the platform and the fabric curtains.
stretches
lounge chairs beach umbrellas scattered along the shoreline. The
Figure 13: VLV capture complex objects. Caption enumerates almost every object and correctly describe their spatial relationships, highlighting VLV's comprehensive scene understanding.
Raw Image
Midjourney
FLUX.1 F-Ux.1-dev
ImageFX
Caption: medium shot captures and a light-colored ski jacket with black and red accents, paired with dark gray ski pants. A large black and gray backpack is slung over his shoulders. He is using ski poles to nuvigate the slope, and his expression one of concentration and enjoyment. The background features a dense forest bare trees, suggesting winter setting. The ground is covered in snow, and lighting bright, casting shadows that depth to the scene. The overall mood one of outdoor adveettre
skiing down snow-covered slope. He is wearing red beanie, dark sunglasses,
sports. The texture with red the beanie and jacket providing pop color against predominantly white and gray The man's ski tracks are visible in the snow, indicating movement. The stmosphere cold and crisp, typical of winter day. The image is well- well-coompseed with the man in the foreground and the forest in the background, creating a sense of depth. The lighting is bright and natural, enhancing the colors and textures of the scene. The overall impresaion is one of winter sports and outdoor adventure. The man's expression is one of enjoyment and concentration, adding to the overall mood the image.
visible, adding the realiam
image. colors muted,
of the
Figure 14: VLV capture human posture. Captions show details of human well his posture, demonstrating VLV's fine-grained posture awareness.
21

---

Dataset & Model License
F1 Training Datasets
LAION-SB
License: Creative Common CC-BY 4.0 http:::/laaion.aa/llg///aaaa
F22
Testing Datasets
MS-COCO
License: Creative Common CC-BY 4.0 httt::///cccccattasstttt http::///ocoddaaaattttooetttttes
org/Htermsofuse
VQAv2
License: CC-BY 4.0 40bttpsss//taaallggoog///eeeee html
Dataset website:
OK-VQA
License:N/A.
Dataset website: httpps:/ookvqa.allee /rrg
F3 Pre-trained Models
stable-diiffusinn-..-bbbas (used for image generation).
http://hii.gttt...oo.oooaaaaaaaa / LICENSE-KODEL.
Qwen-2.5 (used in 2 for LLM decoder).
https://hugginngface co
Qwen-2.5-VL (used in image captioning).
https://github. com/
walpauub..om///aaaaeaaa QwwenM//weeee.. 5- G.VL.bblob/main/IICENEE
Florence-2-Large (used in image captioning).
http:://uuggingfacc.cc//otoooo http::/aww....atoooo//ooo/ee/eea
L.LVA-v1.5 (used in image captioning).
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction (Sec. clearly enumerate our key contributions especially in the last paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a subsection of limitation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

This work is entirely empirical and does not include novel formal theorems or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: To ensure reproducibility, we include implementation details the main paper, list full hyperparameters in the appendix, and provide comprehensive dataset descriptions and evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The dataset is partially open-sourced, and will release the implementation code upon the paper's acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
httpssubmissoooooioiiooooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtppbcccc/aacccccccccccccccccct weticc/ggii/////ee /Ctpe//ddeeebmeeeecee wwaibbmmsssioPPaa ssionnolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The section of experiment and method fully describes training and test details, and Appendix further lists all remaining hyperparameter choices and tuning methodology.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix reports the results of mean and std over multiple runs. We conduct rigorous comparison with baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information about the computer resources in the implementation subsection.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:://eee..eeeeeeee httpppu/ltttistsiiiiiiiiaaaaaaee
Answer: [Yes]
Justification:
We have reviewed the ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[NA]
The manuscript is purely technical and focuses on the model improvement; it does not include dedicated section on potential societal or ethical impacts.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our models operate only in benchmark and pose direct high-risk dual-use concerns.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All data and models used are properly cited.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer:[NA]]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htaps:://neuris... for what should or should not be described.
29


Reviews:
[{"review_id": "ANnTDoxqpT", "reviewer": "Reviewer_yw4J", "summary": "This paper presents the Vision-Language-Vision (VLV) auto-encoder, a novel and efficient framework for leveraging pretrained text-to-image diffusion models to learn rich multimodal representations. The proposed two-stage pipeline first distills semantic information from images using a frozen diffusion decoder, then decodes the intermediate continuous embeddings into natural language captions with a pretrained large language model. Notably, the approach requires only single-modal image data and remains cost-efficient—under $1,000 USD for training—while achieving captioning quality comparable to state-of-the-art commercial models such as GPT-4o and Gemini 2.0 Flash. The authors further demonstrate that the learned representations exhibit strong spatial consistency and compositional generalization, underscoring the effectiveness of their distillation strategy.", "questions": "1) The authors state in lines 204–241 that “Although VLV is not the top scorer in every setting, it reaches comparable [performance] while training at lower cost, underscoring its scalability.” It would be helpful to clarify how cost-efficiency alone substantiates the claim of scalability. Typically, scalability refers to a model’s ability to handle increasing data or model size while maintaining or improving performance—how does lower training cost directly support this notion?\n\n2) In Table 1 and Table 2, it is unclear which version of the Qwen2.5 language model was used (e.g., 1.5B, 3B, or larger such as 7B). While line 202 mentions Qwen2.5, it does not specify which model size corresponds to the key experimental results. Clarifying this would help contextualize the reported performance.\n\n3) While the paper includes an ablation comparing 1.5B and 3B language decoders, it would be interesting to see whether the captioning performance continues to improve with larger models, such as 7B or beyond. Have the authors considered evaluating the proposed method with a more powerful LLM decoder to better characterize the scalability and upper-bound performance of the framework?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strength\n\n### Novel and well-motivated idea: \nThe core concept of using a frozen text-to-image diffusion model as a supervision signal to learn semantic-rich caption embeddings is original and intellectually engaging. The \"Vision-Language-Vision\" auto-encoding pipeline offers a fresh perspective on multimodal representation learning.\n\n### Cost-efficiency: \nThe method demonstrates impressive performance despite using only single-modal images and a small compute budget (under $1,000 USD), making it accessible and reproducible for a wide range of research groups.\n\n### Strong empirical results at small scale: \nEven with relatively small-scale data (6M–40M images) and models (1.5B–3B), the approach delivers competitive or superior results compared to state-of-the-art systems, validating the effectiveness of the design.\n\n### Scalability potential: \nThe observed scaling trends in both data size and model size suggest that the framework could further improve with larger-scale training. This indicates strong potential for future performance gains, making the method a promising foundation for more powerful captioning systems.\n\n\n# Weakness\n\n### Use of FID as an indirect and potentially confounded measure of caption quality:  \nThe paper primarily assesses caption quality via FID scores computed from images synthesized by a third-party text-to-image model (SDM-3.5). However, this indirect evaluation conflates the quality of the generated captions with the capabilities of the generation model, making it difficult to isolate the effectiveness of the proposed captioner. Furthermore, standard automatic captioning metrics such as BLEU, METEOR, CIDEr, and SPICE are not reported, which limits comparability to existing captioning approaches. To better isolate the contribution of the captioner, the authors are encouraged to supplement their evaluation with standard automatic captioning metrics (e.g., CIDEr, SPICE, METEOR) on established benchmarks such as MS-COCO or NoCaps. These metrics would offer a more direct and model-independent measure of semantic and linguistic quality, improving the comparability and credibility of the reported results.\n\n### Limited scope in scalability evaluation, especially in model scaling:  \nWhile the paper presents initial scalability experiments in terms of data (6M vs. 40M images) and model size (1.5B vs. 3B), the evaluation range remains limited—particularly on the model scaling side. Given that the proposed framework emphasizes cost-effective scalability, it would be beneficial to examine performance trends across a broader range of model sizes. In particular, extending the caption decoder to larger LLMs (e.g., 7B or beyond) could help reveal whether performance continues to improve with scale or begins to plateau. Although computational constraints are understandable, even an intermediate step toward a larger model would provide stronger empirical evidence for the framework’s scalability claims.\n\n### Limited diversity of pretrained model components:  \nThe proposed framework depends on a fixed set of pretrained modules—Stable Diffusion 2.1 as the frozen image decoder in Stage-1 and Qwen2.5 as the LLM decoder in Stage-2. While this setup is effective and cost-efficient, it leaves open the question of how well the method generalizes to more recent or stronger model architectures. For instance, replacing SD 2.1 with more capable T2I models such as FLUX, SANA, or Pixart-series may improve the quality of the learned semantic embeddings, and alternative LLMs like Mistral may offer gains in caption fluency or factual precision. Although the authors acknowledge this limitation for the image decoder, they do not provide empirical results with other models. If computational budget permits, evaluating the framework across a broader range of pretrained backbones would further strengthen the claim of modularity and demonstrate the robustness of the proposed approach.", "was_revised": true, "final_justification": "After reviewing the authors’ rebuttal, I have decided to maintain my original score of 5 (Accept). While some concerns remain partially addressed due to computational constraints, the core contributions of the paper are strong, and the authors have provided reasonable justifications and clarifications. My final assessment is based on the following points:\n- Use of FID as a proxy for caption quality\nThe authors clarified that FID was chosen intentionally to evaluate detailed captions by measuring how well a diffusion model can reconstruct an image from the caption. While I still believe that including standard text-based metrics (e.g., CIDEr, SPICE) would strengthen the evaluation, their justification for relying on FID in the high-detail captioning context is reasonable. The acknowledgment of the limitations and discussion of trade-offs reflect thoughtful consideration of the evaluation methodology.\n- Model scalability evaluation\nAlthough the experiments do not include models beyond 3B due to GPU limitations, the authors do present a clear scaling trend from 0.5B → 1.5B → 3B, showing consistent improvements. This partially addresses my concern and supports their claim of scalability, even if not exhaustively demonstrated. I appreciate the authors’ transparency about resource limitations and their commitment to making the scaling trend more explicit in the final version.\n- Modularity and diversity of pretrained components\nThe authors explained that using stronger diffusion models (e.g., FLUX, SD3.5) was infeasible within their current budget, but provided evidence that the current setup with SD2.1 is the limiting factor, not the captioner. This clarifies the rationale behind the fixed component choices and supports their claim of modularity under practical constraints.\n\nOverall, while a broader set of evaluations would have further strengthened the paper, the authors’ thoughtful rebuttal and empirical evidence reinforce my positive impression of the work. The proposed framework is well-motivated, effective under realistic constraints, and makes a meaningful contribution to the field. I recommend acceptance.", "rebuttal": "1. We report FID rather than text-only metrics such as CIDEr because our task is detailed captioning: a caption is good if it contains enough information for a diffusion model to recreate an image that is perceptually close to the original. Concretely, we feed each caption to Stable Diffusion 3.5-Medium, which accepts prompts longer than 77 tokens, generate an image with identical denoising steps, resolution, guidance scale, and sampler across all experiments, and then compute the Fréchet Inception Distance between the generated and source images; since every setting except the caption is fixed, lower FID directly implies a richer, more accurate description. By contrast, CIDEr, BLEU, and SPICE depend on ground-truth references whose average length on MS-COCO is only about ten words, so they systematically penalise the longer, fine-grained sentences produced by modern VLMs. While FID thus provides a fairer gauge of descriptive adequacy in our scenario, we agree that the community still lacks a universally accepted metric for high-detail captioning.\n2. Sorry for the confusion on the base model we use in our main experiments. We use Qwen2.5-3B as our LLM decoder in our training stage 2. Qwen 2.5-3B is the largest model we could afford given our GPU budget for fully fine-tuning.  But we do perform scalability experiments with Qwen2.5-0.5B, Qwen 2.5-1.5B, and Qwen2.5-3B. From the table here, it is obvious that with the increase of the LLM decoder size, the performance on the image captioning task improves. And according to the experiment here, we can infer that we can get better results with Qwen2.5 -7B or more powerful LLM. \n| Decoder    | GS=1 | GS=2 | GS=3 | GS=4 |\n|----------------------|---------------:|---------------:|---------------:|---------------:|\n| Qwen-2.5-0.5 B       | 14.70          | 9.37           | 11.26          | 12.45          |\n| Qwen-2.5-1.5 B       | 12.25          | 7.30           | 9.16           | 10.26          |\n| **Qwen-2.5-3 B**     | **11.47**      | **6.64**       | **8.56**       | **9.90**       |\n3. Given our computational constraints, it was infeasible to adopt heavier diffusion back-bones such as SD-3.5, or FLUX for Stage 1, so we relied on Stable Diffusion 2.1. As illustrated in Fig. 3 (second row), SD-2.1 effectively defines our performance ceiling: when we reconstruct images directly from our caption embeddings, minor detail mismatches still appear, indicating that the decoder—not the caption model—is the limiting factor. To narrow this gap in Stage 2 we unfreeze the VLV encoder and train it jointly with the auto-regressive decoder; although this choice modestly increases memory usage, it yields clearly more faithful captions (Table 4). We therefore conclude that, resources permitting, pairing our pipeline with a more powerful diffusion model would further boost caption fidelity, while the current results already represent the best achievable quality under SD-2.1.\n4. Thank you for pointing this out. Our intent in Lines 240–241 was not to claim that low absolute cost alone proves scalability, but rather to highlight that our pipeline already matches state-of-the-art performance despite running with modest-capacity diffusion and language decoders. The ablations in Table above show that when we increase only the LLM decoder from 0.5 B → 3 B parameters, caption quality (FID↓) improves monotonically. Thus, we expect better performance with a powerful diffusion decoder and LLM decoder. We will make this statement clearer in our manuscript once it is accepted."}, {"review_id": "76zj3D5vRR", "reviewer": "Reviewer_hQYb", "summary": "This paper introduces the Vision-Language-Vision Auto-Encoder, a two-stage framework for distilling cross-modal knowledge from pretrained text-to-image diffusion models into a lightweight image captioning system. In Stage 1, images are encoded into continuous “caption embeddings,” which a frozen diffusion decoder reconstructs back into images, effectively extracting semantic understanding without paired text labels. In Stage 2, these embeddings are fed into a frozen large language model that is lightly fine-tuned to generate natural-language captions. The authors demonstrate that, at a total training cost under $1,000, their method matches the caption quality of commercial systems like GPT-4o and Gemini 2.0 Flash on standard benchmarks.", "questions": "Please refer to the weaknesses.", "limitations": "Please refer to the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Achieves state-of-the-art captioning performance using only open-source models and modest compute budget, lowering the barrier for resource-constrained teams.\n\n2. Cleverly leverages a frozen diffusion decoder as a semantic bottleneck, enabling self-supervised extraction of rich cross-modal features without paired supervision.\n\n3. The continuous embeddings capture spatial layout and multi-item composition, enabling zero-shot reasoning about object relationships and 3D pose.\n\n4. Fully based on public checkpoints with detailed training recipes, facilitating community adoption and future development.\n\nWeaknesses:\n\nDespite the paper’s interesting contributions, I have several serious concerns that need clarification:\n\n1. The authors argue that pretrained diffusion models’ vision–language embeddings are highly valuable and therefore design a self-supervised pipeline to distill them. However, since CLIP’s text and image embeddings in these models are already well-aligned through extensive pretraining, why not simply extract the image embeddings directly? What concrete advantage does the proposed self-supervised scheme offer, if its main goal also seems to be alignment?\n\n2. The current work uses Stable Diffusion 2.1, which relies on CLIP for text embeddings, but SD 3.0+ shifts to a T5-based text encoder. Does this mean the entire system must be retrained from scratch for newer SD versions? Or could one simply extract embeddings to achieve seamless compatibility across different frameworks? This point deserves a thorough discussion.\n\n3. The authors employ DaViT as their vision encoder. Could alternative architectures (e.g., ViT variants, Swin, ConvNeXt) yield different or potentially better results within the same pipeline?\n\n4. In Figure 3, some Stage 1 reconstructions differ in fine details—for example, whether a cat is sticking its tongue out. Might such minor discrepancies degrade the quality of downstream captions?\n\n5. I’m curious whether the observed captioning improvements extend to more stylized or domain-specific imagery—such as posters, greeting cards, or other graphic art. Does the approach maintain its performance in these specialized scenarios?", "was_revised": true, "final_justification": "Thanks to the authors for the rebuttal, which solved most of my problems. It would be more meaningful if it could be further verified based on other base models. I choose to maintain my rating.", "rebuttal": "1. The previous works, like LLaVA have already done this. And in our **Table1**, we have proved that our pipeline is better than previous ones. Moreover, compared with our pipeline, the CLIP requires **high-quality text-image pairs**. While in our pipeline, we can learn language semantics representation **via images only** in our training stage 1 by establishing an information bottleneck. \n2. Retraining the entire system from scratch is unnecessary when moving to newer SD versions. In Stage 1, the VLV encoder is trained with an image-only, information-bottleneck objective and never encounters CLIP tokens. To accommodate a T5 text encoder, we simply introduce a lightweight linear/MLP adapter and fine-tune it. Concretely, we trained a two-layer MLP with a ReLU activation to map CLIP text embeddings to the T5 embedding space; after 2 000 steps on 512 k captions, the validation loss dropped to 0.003. Replacing the T5 encoder in SD 3.5 with the CLIP encoder from SD 2.1 plus this trained adapter still produced correct images. We use this pipeline for T2I tasks. Specifically, we generate hundreds of images with the corresponding captions and calculate the CLIP scores. We get the mean a CLIP score of **28.88**.\\\nBecause the text encoder in our pipeline remains frozen, functionally acting as a “large MLP layer”, adapting to any new text space only requires adding and briefly tuning a similarly small adapter.\n3. To endow our model with genuine image understanding—i.e., the ability to map raw pixels to rich, language-semantic representations—we build the VLV encoder on DaViT, following Florence-2[1]. As Tables 7 and 9 in Florence-2 show(we put the table below for reference), DaViT consistently surpasses alternative backbones (ViT variants, Swin, ConvNeXt) on detection and segmentation benchmarks, indicating superior capacity for high-level visual reasoning. Leveraging this stronger visual core allows the VLV encoder to comprehend scene layout, object attributes, and relational cues more effectively. Consequently, substituting DaViT with other backbones would be unlikely to improve results within the same pipeline.\n### Mask R-CNN / DINO Performance  \n\n| Backbone        | Pretrain        | AP_b | AP_m | DINO AP |\n|-----------------|-----------------|:--------------:|:--------------:|:-------:|\n| ViT-B           | MAE IN-1k       | 51.6 | 45.9 | 55.0 |\n| Swin-B          | Sup IN-1k       | 50.2 | –    | 53.4 |\n| Swin-B          | SimMIM          | 52.3 | –    | –    |\n| FocalAtt-B      | Sup IN-1k       | 49.0 | 43.7 | –    |\n| FocalNet-B      | Sup IN-1k       | 49.8 | 44.1 | 54.4 |\n| ConvNeXt v1-B   | Sup IN-1k       | 50.3 | 44.9 | 52.6 |\n| ConvNeXt v2-B   | Sup IN-1k       | 51.0 | 45.6 | –    |\n| ConvNeXt v2-B   | FCMAE           | 52.9 | 46.6 | –    |\n| **DaViT-B**     | *Florence-2*    | **53.6** | **46.4** | **59.2** |\n\n---\n\n### ADE20K Semantic Segmentation (UperNet)  \n\n| Backbone        | Pretrain        | mIoU | ms-mIoU |\n|-----------------|-----------------|:----:|:-------:|\n| ViT-B           | Sup IN-1k        | 47.4 | –    |\n| ViT-B           | MAE IN-1k        | 48.1 | –    |\n| ViT-B           | BEiT             | 53.6 | 54.1 |\n| ViT-B           | BEiTV2 IN-1k     | 53.1 | –    |\n| ViT-B           | BEiTV2 IN-22k    | 53.5 | –    |\n| Swin-B          | Sup IN-1k        | 48.1 | 49.7 |\n| Swin-B          | Sup IN-22k       | –    | 51.8 |\n| Swin-B          | SimMIM           | –    | 52.8 |\n| FocalAtt-B      | Sup IN-1k        | 49.0 | 50.5 |\n| FocalNet-B      | Sup IN-1k        | 50.5 | 51.4 |\n| ConvNeXt v1-B   | Sup IN-1k        | –    | 49.9 |\n| ConvNeXt v2-B   | Sup IN-1k        | – | 50.5   |\n| ConvNeXt v2-B   | FCMAE            | –    | 52.1 |\n| **DaViT-B**     | *Florence-2*     | **54.9** | **55.5** |\n\n4. We acknowledge that feeding our caption embeddings into Stable Diffusion 2.1 cannot reproduce fine-grained visual details—the generation quality is capped by SD 2.1 itself. To mitigate this ceiling, Stage 2 trains an auto-regressive caption decoder jointly with a trainable VLV encoder. Making the visual encoder update alongside the decoder allows it to refine its representations toward the specific descriptive cues the decoder needs, yielding noticeably richer captions. The ablation in section 4.3 (Table 4) confirms the benefit: enabling VLV encoder updates yields the best performance among all other settings, despite the generative ceiling imposed by SD 2.1.\n\n5. Yes. We use **laion2B-en-aesthetic** for stage 1 training to learn the language-semantic representation. This dataset is a subset of LAION-5B, which indeed contains stylized images, such as birthday greeting cards or posters. However, we filter out those images due to our criteria shown in the supplementary materials, Section A, data processing. We also explain that our model right now is not good at OCR due to the ability of SD 2.1. But our model could be applied to specific scenarios, fine-tuned on a corresponding dataset.\n\nReference:\\\n[1] **Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks**"}, {"review_id": "6D41vu6Pk8", "reviewer": "Reviewer_pgRD", "summary": "this work proposes vision-language-vision autoencoders as a simple and cost-effective way to construct vision-language models by combining pretrained components including LLMs and text-to-image diffusion models. The training happens in two stages, where in the first one a frozen pretrained vision model is used to generate output tokens in the CLIP text encoder's space directly (ie the embedding layer after the tokenization), which are then used with a frozen t2i diffusion model to create an image. The connectors are trained with a simple self-supervised image-space reconstruction loss. In stage two, this model is used to train a simple captioner, ClipCap-style using image-text data.", "questions": "* Comparison to De-Diffusion on 40M images is missing. As a key comparison method that is both simpler and already public for a while, it would be important to comapre against this method.\n * what happens if you train the LLM with PEFT methods?\n* how are 3D boxes constructed from the caption embeddings? this is not well described", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "+ good idea of reusing pretrained components\n+ simple training recipe and design choices\n+ strong captioning performance\n\n- Just a small point: \"This approach learns language-semantic representations only using image-based training.\" might be confusing since you do rely on text-image supervised pretrained models. \n- It seems that De-Diffusion is quite similar, except that they train discrete text tokens instead of continous ones, which further allows them to yield interpretrable captions directly, while VLV requires another stage for this. \n- some experimental settings missing/unclear", "was_revised": true, "final_justification": "I've read the rebuttal and the other reviews. My concerns have been addressed and I see the valuable contribution in terms of open-code and strong model performance, despite the conceptual incremental novelty in terms of method compared to De-Diffusion. I've increased my score.", "rebuttal": "1. To clarify our contribution: the core representation-learning stage relies exclusively on unpaired images, enabling the model to distill **language-semantic features** directly from visual data. Captioned text–image pairs are introduced only in a subsequent alignment phase to map the already-learned visual latent space onto the textual domain; they do not provide supervision for learning the underlying representations themselves.\n2. De-Diffusion is **not open-sourced**, so a direct, fully controlled comparison is impossible. With their authors’ cooperation, we attempted to reproduce their pipeline; however, convergence proved unstable because their discrete-token approach relies on a Gumbel-Softmax relaxation whose temperature schedule is undocumented. In contrast, our method operates with continuous text embeddings, making the entire image-to-text pathway differentiable end-to-end and eliminating the need for potentially brittle discrete sampling. Moreover, we build on openly available pre-trained models and public datasets, ensuring that our codebase is reproducible for the research community and readily scalable in industrial settings. We believe these practical and methodological advantages provide a fair—and, for many applications, more deployable—alternative to De-Diffusion.\n3. We added some experimental settings as **Reviewer CiwQ** requested, we hope those clarifications could help you have a better understanding of our work.\n4. As requested, we have added a **LoRA-based baseline** to Stage 2. Specifically, we fine-tuned **Qwen-2.5 (3 B)** with **LoRA rank = 16** under exactly the same hyper-parameter settings used for the full-parameter baseline. After training, we generated captions for the same 30 K COCO validation images and assessed quality by prompting SD-3.5-medium and computing FID with guidance scale = 2. The LoRA variant reduces GPU memory consumption substantially (only the low-rank adapters are updated), but, as shown in Table here, this comes at a noticeable cost in caption quality, with FID increasing relative to full fine-tuning. These results confirm that while LoRA is attractive for resource-constrained scenarios, full fine-tuning remains preferable when maximum performance is required.\n| Method               | FID ↓ (GS = 2) | Peak GPU Memory (GB) per GPU |\n|----------------------|----------------|------------------------------|\n| LoRA (rank 16)       | 8.09           | 20 GB                        |\n| Full fine-tuning     | 6.64           | 47 GB                        |\n\n\n5. Thank you for pointing out the ambiguity regarding the 3D bounding boxes in Section 4.4 (“Emerging Properties”). To clarify: the 3D bounding boxes are not produced directly from the text embeddings. Instead, after generating images from the text embeddings, we prompt Gemini 2.0 Flash to extract their 3D bounding boxes. The “emerging” aspect we highlight refers to the fact that our training pipeline unexpectedly yields images whose spatial structure can be reliably recovered in 3D—even though the model was never explicitly trained for that task. We will update the manuscript accordingly to make this workflow clear and avoid further confusion."}, {"review_id": "BXOyzHeeuh", "reviewer": "Reviewer_CiwQ", "summary": "This paper proposes a simple and low cost two-stage approach to generating good captions by distilling knowledge from t2I models.\nThe first stage is SSL based on images, where the VLV is required to learn good text embedding to align with the frozen t2I and reconstruct the input image. Based on the pretrained weights, the second stage utilizes text-image pairs and LLMs to construct a stronger captioner.\nSome experimental results show  that this approach can achieve comparable captioning performance with GPT4o and Gemini 2.0 FLASH on some tasks.", "questions": "See the weakness part.", "limitations": "YES.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. The paper studies an interesting topic which generates good text embedding for captions.\n2. The image only pretraining design at stage 1 is novel, and I enjoy this idea. \n3. The method is efficient and achieves comparable performance as GPT4o on some scenarios.\n\nWeakness\n1. During the training of stage 1, the diffusion decoder is frozen, how do you generate the image from the embedding? How many steps do you use to perform the de-noising process?\n2. Section 4.2.1 and Table 1, how are the captions generated ?  What are the prompt templates for each VLM? How do you ensure a fair comparison, since VLMs are sensitive to the prompt template. \n3. What's the validation set of Table~6? How many images are used to calculate the score? How do you predict the 3D bounding box?\n4. Some detailed settings are missed. What's the configuration of Transformer encoder-decoder and MLP in VLV Encoder? \n5. The evaluation is a bit limited and not comprehensive. A good caption model should cover many popular scenarios such as OCR and so on.\n6. The component in the baseline is a bit old. It's more common to apply FLUX or SD3+. \n\nMinors.\n1. Line 198， For memory efficiency parameters are optimized using AdamW. Is there a direct link for this statement?\n2. Line 205,   which do you use, FP16 or BF16? The statement of FP16/BF16 is confusing.", "was_revised": true, "final_justification": "Thanks for the rebuttal. As most of my concerns have been addressed, I decide to raise my score.\nIt's better to include modern t2i models to make this paper stronger.", "rebuttal": "1. During Stage 1 of training, we follow the pipeline established by Stable Diffusion 2.1 base. Specifically, we first obtain conditioning features from the VLV encoder. These features are then passed through our trainable projection layer followed by the CLIP Text Encoder from SD 2.1. The resulting projected conditions, together with random noise, are input into the U-Net from SD 2.1 to predict the noise residuals. We employ a standard mean squared error (MSE) loss for supervision in this stage. Throughout training, we adopt the default 1000 denoising steps used in SD 2.1 for the noise prediction process. For inference (image generation), we follow the same conditioning pipeline and generate images using the SD 2.1 sampling procedure with 50 denoising steps.\n\n2. It is indeed true that the VLMs are sensitive to the prompt template. Thus, for fair comparison, we use the same prompts for VLMs. Specifically, for `LLaVA-v1.5-7B` and `Qwen-2.5-VL-7B`, we use the same prompts. For `Florence-2 Large`, it has a fixed prompt template; for fair comparison, we use `<MORE_DETAILED_CAPTION>`. Please refer to the table below for detailed caption sources.\n| **Model**            | **Prompt / Caption Source**                                                                 |\n|----------------------|----------------------------------------------------------------------------------------------|\n| Original             | Captions from MS-COCO Dataset                                                               |\n| Recap-7B             | Captions from [1] .               |\n| LLaVA-v1.5-7B        | `Please describe this image as detailed as possible.`                                       |\n| Qwen-2.5-VL-7B       | `Please describe this image as detailed as possible.`                                       |\n| Florence-2 Large     | `<MORE_DETAILED_CAPTION>`                                                                   |\n| VLV                  | No prompt                                                                                   |\n*Table: Prompt templates or caption sources used for each model.*\n\n3. For Table 6, we randomly select 200 images from *laion2B-en-aesthetic* as the validation set. And we prompt Gemini 2.0 Flash to predict the 3D bounding box for those images and then we visualize the boxes by ourselves. For Section 4.4, Emerging Properties,  the “emerging” aspect we highlight refers to the fact that our training pipeline unexpectedly yields images whose spatial structure can be reliably recovered in 3D—even though the model was never explicitly trained for that task.\n\n4. Configuration of Transformer encoder-decoder:\\\nThe model employs a symmetric, sequence-to-sequence Transformer with 12 encoder and 12 decoder layers, each with a 1024-dimensional hidden size. Every layer uses 16-head self-attention, followed by a feed-forward network that expands to 4096 dimensions and applies a GELU activation. Regularization mirrors Stable Diffusion defaults: a 0.1 dropout is applied to activations, attention, and residual connections. The tokenizer supports sequences up to 4 096 tokens and uses the standard identifiers BOS = 0, PAD = 1, and EOS = 2. All weights are initialized from a normal distribution with σ = 0.02, ensuring stable training. \\\nConfiguration of MLP in VLV Encoder:\\\nThe MLP in the VLV Encoder is implemented as a two-layer feedforward network with a GELU activation in between. It projects language features from a 1024-dimensional input to a configurable hidden dimension (hidden_dim) and then back to 1024 dimensions.\n\n5. We agree with the reviewer that handling OCR-related content is an important aspect for captioning models. As noted in our limitations section, our current model is not well-suited for OCR tasks. This limitation primarily arises from Stable Diffusion 2.1 itself. Stable Diffusion 2.1 is poor at generating images with plenty of words on those. Thus, we cannot distill the ability of capturing the OCR-related content from SD 2.1.\n\n6. It is true that SD2.1 is a bit old; due to the computational resources, it is hard for us to apply SD3.5 and FLUX. Applying state-of-the-art diffusion decoders is also our future work.\n\nMinors:\n1. We follow this setting from **De-Diffusion**[2] work and **Adafactor**[3].\n2. Sorry for the confusion, we perform our experiment with both BF16 and FP16, and we find BF16 is the better option.\n\nReference:\\\n[1] **What If We Recaption Billions of Web Images with LLaMA-3?**  \n[2] **De-Diffusion Makes Text a Strong Cross-Modal Interface**\\\n[3] **Adafactor: Adaptive learning rates with sublinear memory cost**"}]

Author rebuttal:

Meta-review: This paper introduces the Vision-Language-Vision (VLV) auto-encoder framework by combining pre-trained T2I diffusion model (DM) decoder and LLM. The paper proposes a two-stage pipeline where stage 1 is based on a distillation from DM (by decoding images from the intermediate feature), and stage 2 is for training a captioner from generated captions by LLM from the intermediate features. The proposed framework is efficient, showing training cost under 1,000 USD.

This paper has three positive borderlines and one positive recommendation. 

The reviewers agree that the proposed framework is novel (yw4J, pgRD, CiwQ), efficient (yw4J, CiwQ), and effective even compared with large-scale systems (yw4J, hQYb, pgRD, CiwQ)

As unresolved weaknesses of the paper, the reviewers pointed out the problems in the limited set of scales and families of pre-trained models (yw4J, hQYb, CiwQ) and the problem of using FID as an evaluation metric for caption quality (yw4J). Also, pgRD pointed out that the novelty of the proposed method compared to De-Diffusion is somewhat incremental.

Even though the weaknesses still remain, the AC thinks that the weaknesses of this paper do not outweigh its strengths. I strongly suggest that the authors add more discussions related to the concerns raised by the reviewer.

Overall, I recommend accept this paper.
Decision: Accept (poster)