Title: Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Auto-Encoder

Diffusion Model

Vision-Language-Vision
Abstract: Building state-of-the-art Vision-Language Models (VLMs) with strong captioning capabilities typically necessitates training on billions of high-quality image-text pairs, requiring millions of GPU hours. This paper introduces the Vision-Language-Vision **(VLV)** auto-encoder framework, which strategically leverages key pretrained components: a vision encoder, the decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large Language Model (LLM). Specifically, we establish an information bottleneck by regularizing the language representation space, achieved through freezing the pretrained T2I diffusion decoder. Our VLV pipeline effectively distills knowledge from the text-conditioned diffusion model using continuous embeddings, demonstrating comprehensive semantic understanding via high-quality reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the intermediate language representations into detailed descriptions, we construct a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and significantly reduces data requirements; by primarily utilizing single-modal images for training and maximizing the utility of existing pretrained models (image encoder, T2I diffusion model, and LLM), it circumvents the need for massive paired image-text datasets, keeping the total training expenditure under $1,000 USD.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple and low cost two-stage approach to generating good captions by distilling knowledge from t2I models.\nThe first stage is SSL based on images, where the VLV is required to learn good text embedding to align with the frozen t2I and reconstruct the input image. Based on the pretrained weights, the second stage utilizes text-image pairs and LLMs to construct a stronger captioner.\nSome experimental results show  that this approach can achieve comparable captioning performance with GPT4o and Gemini 2.0 FLASH on some tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper studies an interesting topic which generates good text embedding for captions.\n2. The image only pretraining design at stage 1 is novel, and I enjoy this idea. \n3. The method is efficient and achieves comparable performance as GPT4o on some scenarios.\n\nWeakness\n1. During the training of stage 1, the diffusion decoder is frozen, how do you generate the image from the embedding? How many steps do you use to perform the de-noising process?\n2. Section 4.2.1 and Table 1, how are the captions generated ?  What are the prompt templates for each VLM? How do you ensure a fair comparison, since VLMs are sensitive to the prompt template. \n3. What's the validation set of Table~6? How many images are used to calculate the score? How do you predict the 3D bounding box?\n4. Some detailed settings are missed. What's the configuration of Transformer encoder-decoder and MLP in VLV Encoder? \n5. The evaluation is a bit limited and not comprehensive. A good caption model should cover many popular scenarios such as OCR and so on.\n6. The component in the baseline is a bit old. It's more common to apply FLUX or SD3+. \n\nMinors.\n1. Line 198， For memory efficiency parameters are optimized using AdamW. Is there a direct link for this statement?\n2. Line 205,   which do you use, FP16 or BF16? The statement of FP16/BF16 is confusing.\n\nQUESTIONS:\nSee the weakness part.\n\nLIMITATIONS:\nYES.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

comments: SUMMARY:
this work proposes vision-language-vision autoencoders as a simple and cost-effective way to construct vision-language models by combining pretrained components including LLMs and text-to-image diffusion models. The training happens in two stages, where in the first one a frozen pretrained vision model is used to generate output tokens in the CLIP text encoder's space directly (ie the embedding layer after the tokenization), which are then used with a frozen t2i diffusion model to create an image. The connectors are trained with a simple self-supervised image-space reconstruction loss. In stage two, this model is used to train a simple captioner, ClipCap-style using image-text data.

STRENGTHS AND WEAKNESSES:
+ good idea of reusing pretrained components
+ simple training recipe and design choices
+ strong captioning performance

- Just a small point: "This approach learns language-semantic representations only using image-based training." might be confusing since you do rely on text-image supervised pretrained models. 
- It seems that De-Diffusion is quite similar, except that they train discrete text tokens instead of continous ones, which further allows them to yield interpretrable captions directly, while VLV requires another stage for this. 
- some experimental settings missing/unclear

QUESTIONS:
* Comparison to De-Diffusion on 40M images is missing. As a key comparison method that is both simpler and already public for a while, it would be important to comapre against this method.
 * what happens if you train the LLM with PEFT methods?
* how are 3D boxes constructed from the caption embeddings? this is not well described

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Vision-Language-Vision Auto-Encoder, a two-stage framework for distilling cross-modal knowledge from pretrained text-to-image diffusion models into a lightweight image captioning system. In Stage 1, images are encoded into continuous “caption embeddings,” which a frozen diffusion decoder reconstructs back into images, effectively extracting semantic understanding without paired text labels. In Stage 2, these embeddings are fed into a frozen large language model that is lightly fine-tuned to generate natural-language captions. The authors demonstrate that, at a total training cost under $1,000, their method matches the caption quality of commercial systems like GPT-4o and Gemini 2.0 Flash on standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Achieves state-of-the-art captioning performance using only open-source models and modest compute budget, lowering the barrier for resource-constrained teams.\n\n2. Cleverly leverages a frozen diffusion decoder as a semantic bottleneck, enabling self-supervised extraction of rich cross-modal features without paired supervision.\n\n3. The continuous embeddings capture spatial layout and multi-item composition, enabling zero-shot reasoning about object relationships and 3D pose.\n\n4. Fully based on public checkpoints with detailed training recipes, facilitating community adoption and future development.\n\nWeaknesses:\n\nDespite the paper’s interesting contributions, I have several serious concerns that need clarification:\n\n1. The authors argue that pretrained diffusion models’ vision–language embeddings are highly valuable and therefore design a self-supervised pipeline to distill them. However, since CLIP’s text and image embeddings in these models are already well-aligned through extensive pretraining, why not simply extract the image embeddings directly? What concrete advantage does the proposed self-supervised scheme offer, if its main goal also seems to be alignment?\n\n2. The current work uses Stable Diffusion 2.1, which relies on CLIP for text embeddings, but SD 3.0+ shifts to a T5-based text encoder. Does this mean the entire system must be retrained from scratch for newer SD versions? Or could one simply extract embeddings to achieve seamless compatibility across different frameworks? This point deserves a thorough discussion.\n\n3. The authors employ DaViT as their vision encoder. Could alternative architectures (e.g., ViT variants, Swin, ConvNeXt) yield different or potentially better results within the same pipeline?\n\n4. In Figure 3, some Stage 1 reconstructions differ in fine details—for example, whether a cat is sticking its tongue out. Might such minor discrepancies degrade the quality of downstream captions?\n\n5. I’m curious whether the observed captioning improvements extend to more stylized or domain-specific imagery—such as posters, greeting cards, or other graphic art. Does the approach maintain its performance in these specialized scenarios?\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Vision-Language-Vision (VLV) auto-encoder, a novel and efficient framework for leveraging pretrained text-to-image diffusion models to learn rich multimodal representations. The proposed two-stage pipeline first distills semantic information from images using a frozen diffusion decoder, then decodes the intermediate continuous embeddings into natural language captions with a pretrained large language model. Notably, the approach requires only single-modal image data and remains cost-efficient—under $1,000 USD for training—while achieving captioning quality comparable to state-of-the-art commercial models such as GPT-4o and Gemini 2.0 Flash. The authors further demonstrate that the learned representations exhibit strong spatial consistency and compositional generalization, underscoring the effectiveness of their distillation strategy.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n### Novel and well-motivated idea: \nThe core concept of using a frozen text-to-image diffusion model as a supervision signal to learn semantic-rich caption embeddings is original and intellectually engaging. The \"Vision-Language-Vision\" auto-encoding pipeline offers a fresh perspective on multimodal representation learning.\n\n### Cost-efficiency: \nThe method demonstrates impressive performance despite using only single-modal images and a small compute budget (under $1,000 USD), making it accessible and reproducible for a wide range of research groups.\n\n### Strong empirical results at small scale: \nEven with relatively small-scale data (6M–40M images) and models (1.5B–3B), the approach delivers competitive or superior results compared to state-of-the-art systems, validating the effectiveness of the design.\n\n### Scalability potential: \nThe observed scaling trends in both data size and model size suggest that the framework could further improve with larger-scale training. This indicates strong potential for future performance gains, making the method a promising foundation for more powerful captioning systems.\n\n\n# Weakness\n\n### Use of FID as an indirect and potentially confounded measure of caption quality:  \nThe paper primarily assesses caption quality via FID scores computed from images synthesized by a third-party text-to-image model (SDM-3.5). However, this indirect evaluation conflates the quality of the generated captions with the capabilities of the generation model, making it difficult to isolate the effectiveness of the proposed captioner. Furthermore, standard automatic captioning metrics such as BLEU, METEOR, CIDEr, and SPICE are not reported, which limits comparability to existing captioning approaches. To better isolate the contribution of the captioner, the authors are encouraged to supplement their evaluation with standard automatic captioning metrics (e.g., CIDEr, SPICE, METEOR) on established benchmarks such as MS-COCO or NoCaps. These metrics would offer a more direct and model-independent measure of semantic and linguistic quality, improving the comparability and credibility of the reported results.\n\n### Limited scope in scalability evaluation, especially in model scaling:  \nWhile the paper presents initial scalability experiments in terms of data (6M vs. 40M images) and model size (1.5B vs. 3B), the evaluation range remains limited—particularly on the model scaling side. Given that the proposed framework emphasizes cost-effective scalability, it would be beneficial to examine performance trends across a broader range of model sizes. In particular, extending the caption decoder to larger LLMs (e.g., 7B or beyond) could help reveal whether performance continues to improve with scale or begins to plateau. Although computational constraints are understandable, even an intermediate step toward a larger model would provide stronger empirical evidence for the framework’s scalability claims.\n\n### Limited diversity of pretrained model components:  \nThe proposed framework depends on a fixed set of pretrained modules—Stable Diffusion 2.1 as the frozen image decoder in Stage-1 and Qwen2.5 as the LLM decoder in Stage-2. While this setup is effective and cost-efficient, it leaves open the question of how well the method generalizes to more recent or stronger model architectures. For instance, replacing SD 2.1 with more capable T2I models such as FLUX, SANA, or Pixart-series may improve the quality of the learned semantic embeddings, and alternative LLMs like Mistral may offer gains in caption fluency or factual precision. Although the authors acknowledge this limitation for the image decoder, they do not provide empirical results with other models. If computational budget permits, evaluating the framework across a broader range of pretrained backbones would further strengthen the claim of modularity and demonstrate the robustness of the proposed approach.\n\nQUESTIONS:\n1) The authors state in lines 204–241 that “Although VLV is not the top scorer in every setting, it reaches comparable [performance] while training at lower cost, underscoring its scalability.” It would be helpful to clarify how cost-efficiency alone substantiates the claim of scalability. Typically, scalability refers to a model’s ability to handle increasing data or model size while maintaining or improving performance—how does lower training cost directly support this notion?\n\n2) In Table 1 and Table 2, it is unclear which version of the Qwen2.5 language model was used (e.g., 1.5B, 3B, or larger such as 7B). While line 202 mentions Qwen2.5, it does not specify which model size corresponds to the key experimental results. Clarifying this would help contextualize the reported performance.\n\n3) While the paper includes an ablation comparing 1.5B and 3B language decoders, it would be interesting to see whether the captioning performance continues to improve with larger models, such as 7B or beyond. Have the authors considered evaluating the proposed method with a more powerful LLM decoder to better characterize the scalability and upper-bound performance of the framework?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}