Diffusion Instruction Tuning
Authors: Chen Jin
Ryutaro Tanno
Amrutha Saseendran
Tom Diethe
Philip Alexander Teare
Venue: ICML 2025 poster (2025)
Keywords: vision-language models
diffusion models
image-to-text generation
text-to-image alignment
cross-attention
visual instruction tuning
attention alignment
out-of-distribution robustness
data-efficient training
unified multimodal systems
transformer fine-tuning
vision-language generalization
Primary area: deep_learning->foundation_models
TL;DR: Lavender: an efficient supervised fine-tuning (SFT) approach boosting SoTA vision-language model with Stable Diffusion Model

Abstract: We introduce *Lavender*, a simple supervised fine-tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation models such as Stable Diffusion. Specifically, Lavender aligns the text-vision attention in the VLM transformer with the equivalent used by Stable Diffusion during SFT, instead of adapting separate encoders. This alignment enriches the model’s visual understanding and significantly boosts performance across in- and out-of-distribution tasks. Lavender requires just 0.13 million training examples---2.5\% of typical large-scale SFT datasets---and fine-tunes on standard hardware (8 GPUs) in a single day. It consistently improves state-of-the-art open-source multimodal LLMs (e.g., Llama-3.2-11B, MiniCPM-Llama3-v2.5), achieving up to 30\% gains and a 68\% boost on challenging out-of-distribution medical QA tasks. By efficiently transferring the visual expertise of image generators with minimal supervision, Lavender offers a scalable solution for more accurate vision-language systems. Code, training data, and models are available on the [project page](https://astrazeneca.github.io/vlm/).

Diffusion Instruction Tuning
Chen Jin
Ryutaro Tanno Amrutha Saseendran
Tom Diethe
Philip Teare'
Improvement after Finelune (%)
12
Finetune MiniCPMvv2.5
Finctune MiniCPMv2.5
Lavender: Diffusion Instruction Tuning
=10.6
10-
Finctune Llama-32--11B
Diffusion Instruction Tuming
8-
6-
+3,9
4-
2
+0.3
+2.5
0
#0.1
+0.1
#1.1
#0.1
#0.0
40.5
+1.7
+13.3
1:0.1
.0.22
Real-World
Hallucination Ave. 2 tasks
Multi-Discipline Ave. 10 tasks
Chart/Doc QA Ave. tasks
Ave. 2 tasks
Figure I. Average Performance on 20 Vision ManguaLanguage Reasoning Benchmarks (Grouped into Categories).
Abstract
rate vision- vision-languaan language systems. Code, training data, and models are available on the project page.
We introduce Lavender, a simple supervised fine- tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation mod- els such Stable Diffusion. Specifically, Laven- der aligns the text- attention in the VLM transformer with the equivalent used by Stable Diffusion during SFT, instead of adapting separate encoders. This alignment enriches the model's visual understanding and significantly boosts performance across in- and out-of-distrine of-distribution tasks. Lavender requires just 0.13 million train- ing examples-2.5%% of typical large-scale SFT datasets-and fine-tunes standard hardware (8 GPUs) in a single day. It consistently improves state-of-the-an open-source multimodal LLMs (e.g., Llama-3.22
label
Stable Diffusion
Attention Alignment
Model (frozen)
Attention
question
AttentionyLar
Vision-Language Model (update)
L Dautoregressive
L
Diffusion Instruction Tuning. Lavender uses the text-vision attention maps of a Stable Diffusion Model, AllentiongnM: guiding objective for the attention of the target vision-language model (VLM), IntentionvI.M. The Atten- tion Alignment module employs 3-Layer ConvNet transform Attentionv LM to match AttentionsDar via MSE loss, act- ing a regularisation term during supervised fine-tuning.
Figure 2. Lavender:
MiniCPM--Lama3-v2.55)
achieving up to 30% gains and 68% boost on challenging out-of-distribution medical QA tasks. By efficiently transferring the visual expertise of image generators with minimal supervision, Lavender offers a scalable solution for more accu-
1. Introduction
Training frontier foundation models from scratch costs mil- lions of dollars at minimum, requiring hundreds of GPUs and millions to billions of data (DeepSeek-Al et al., 2024). This challenge is even more pronounced in multimodal settings: lision-language models (VLMs) often face data scarcity because collecting paired image-text datasets is ex- pensive (Zhu al., 2024). A common workaround is to
'Centre for AI, AstraZeneca, Cambridge, UK "Google Deep- Mind, UK. Correspondence to: <chen.jin aatriinnnstttteer..ooom
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diffusion Instruction Tuning
apply supervised fine-tuning (SFT) on a pretrained large lan- guage model (LLM),, leveraging its abundant text-only pre- training and adjusting bridging layers or additional encoders with limited image-text pairs (Liu et al., 2024c; Covert et al., 2024; Jiang et al., 2023). However, these methods typically overlook the importance of transffomer-levell attention align- ment within the LLM core-a key component for effectively expanding text-based models into the visual domain.
1oe
Precise visual-text alignment is crucial for advanced multi- modal reasoning. While both VLMs and diffusion models (DMs) process text and images, they diverge in their genera- tion objectives. We observe that DMs, such as Stable Dif- fusion (Rombach et al., 2021), which reconstructs images at the pixel level, appear to learn more precise text-vision attention maps than VLMs that are optimised solely for text token generation (Figure 3).
bagar
Figure 3. Image generation models (Stable Diffusion on the left) exhibit stronger word-to-region attention alignment than VLMs (Open-Flaminge on the right). Per-word average attention maps suggest that diffusion models may be closer to an ideal distribution correlating image regions with textual tokens.
In this work, we demonstrate that the high-quality cross- attention maps from these DMs indeed offer useful tar- get for guiding the text-vision attention in VLMs during SFT, thus improving word-to-region alignment and the over- all performance. We introduce Lavender (Language-ande and- Vision fine-tuning with Diffusion Aligner), the first frame- work directly align VLM transformer attention layers with those of Stable Diffusion (Figure 1())). Specifically, during SFT, Lavender transfers iiftusion-based attention distributions to VLMs, enhancing core visual-textual inter- actions. To mitigate catastrophic forgetting, we additionally propose several attention aggregation methods and training strategies that preserve existing VLM competencies.
Ablation studies reveal that the method of attention aggre- gation and the choice of layers for fine-tuning are critical to performance. Learned aggregation strategies outperform manual ones and lightweight pretraining of an Aligner Net- work helps prevent catastrophic forgetting on small datasets. LoRA fine-tuning delivers faster improvements, full fine- tuning proves more effective for handling complex tasks, Aligning all cross-attention layers proves most effective, highlighting the importance of precise attention alignment.
In summary, introduce Lavender, novel framework that transfers "visual expertise" from text-to-image diffu- sion models to vision-language models without additional annotations. By aligning attention distributions, Lavender enhances word-to-region grounding, improves fine-tuning efficiency, and boosts model robustness, particularly in out-of of-distribttion settings. Moreover, our architecture- agnostic attention alignment loss is compatible with RL post-training, offering scalable diffusion- -giided feedback instead of costly, subjective human vision feedback. Be- yond addressing data scarcity, Lavender demonstrates that pretrained generative models can guide multimodal learning in a scalable and compute-efficien manner. This approach bridges two expert
We begin by verifying Lavender small OpenFlamingo model: entropy and visual analyses show Lavender aligns VLM attention with DM attention. Leveraging Stable Dif- fusion to offline extract per-word attention on 130k label- image pairs-no extra training cost-Lavender yields notable gains over autoregressive finetuming on 20 diverse bench- marks, including up to 70% improvement on OpenFlamingo across seven benchmarks. For Llama 3.2-11B, fine-tuned on in- and -at-o--distribution data, performance improves by up to 30% on 19 benchmarks, surpassing comparable small open-source models by 50%. On self-attention-only MiniCPMv2.5, it achieves up to 4% gains.
language and vision gener- ation-into more unified, capable system. Our findings suggest broader applications in multimodal AI, offering modular and srivacy-rrienlly alternative to closed-source models. We open-source our work to encourage further exploration of liffusion-guider alignment, unlocking new possibilities in lisiun-language reasoning.
This advantage extends to severely OOD domains, evi- denced by 68% boost on the WorldMedQA medical bench- mark for Llama 3.2- 1.2-11B. 1BB. Further analyses reveal that larger fine-tuning sets help Lavender resist overfitting more ef- fectively than autoregressive baselines, and the aligned at- tention maps yield finer- aner-grained visual understanding. To- gether with qualitative evidence of improved VLM attention, these results confirm Lavender's premise: diffusion--aat -assed attention distributions effectively align visual and textual representations, fostering more robust, data-efficien VLMs.
2. Diffusion Instruction Tuning
We aim to enhance a pretrained Vision-Language Model (VLM) by leveraging attention distributions from a pre- trained Diffusion Model (DM). We assume there an ideal attention distribution that maximises VLM performance and that the DM's s attention closer to this ideal distribution.
2

---

Diffusion Instruction Tuning
Attention Alignment
phear muze
L Lenar
autoregrszaion
Algorithm Diffusion Instruction Tuning
Require: D [(a(0) )( 6p. scalee
Aligner Network
Ensure: Fine-tuned
parameters
Stage 1: Precompute DM Attention (run once)
for each (zx(1, .y ( )
whrch
TM
pic
4 pru(a 211, g00))())) ;0000))
end for
Stage
repeat
Sample batch B C D. set LVIM(() Lm)(0)
for each (a ()) do
Compute PVLM(a , (1()))
(8) + Aligner (Iiiner()eexx(a)) Lv log p(y
A
1x()),) (0
Diffusion Transformer
Vision- EsiionLanguage Transformer
end Lm,(0) (-(0))=1l0 (0))
Tokenized Input: <inages, Labels>
Tokenized Input: sImages, Questions
Lunn(() Lvi (Ln(())
Figure 4. Sketch Diffusion Instruction Tuning (left) and short pseudo code (right), whose full version is available in Appendix F.
Update 0 +- nV a
until convergence
2.1. Models and Notation
also aligns its attention mechanism with that of the DM. We formalise this objective within Bayesian framework.
Vision-Language Model (VLM). Let 9 be the VLM pa- rameters, pretrained for tasks such as image captioning or cestiio-annswring It models:
Posterior Distribution: We aim to find the posterior dis- tribution of the VLM parameters given the data D and the DM's attention distributions:
p(y xr Mei 0),
p(0) D, ADM) x p( p(0),
(4)
where x, Mys Vr. as image, question, and label answer. Diffusion Model (DM). Let OD be the DM parameters, which remain fixed during our procedure. It models:
where ADM {PDM(a (1) y() 0D)} is the collection of attention outputs derived from the DM's conditional distri- bution, and p(0) is the prior over the VLM parameters.
p(zly:0p)
Likelihood of the Data: The likelihood of the data given
Attention Distributions. We write:
p(D Il ply) (1) (1) U4
(1)
(5)
PVLM(a/2, 0), nou(alz,y:00)
(3)
The negative log-likelinne ligllikel corresponds the standard loss function (V) used fine-tune the VLM:
hypothesise that (a) yr 0p) closer to the optimal posterior attention distribution p" (alæ,y) than (aL,(a y; f), and the two can be aligned by projecting PyLM into comparable space using small learnable layers.
LVLM(() Σ log p(yi ( 12 ,ug
(0,00
(6)
2.2. Assumptions
Likelihood of the DM's Attention: We model the like- lihood of observing the DM's attention given the VLM's parameters, denoted as pl ApM (0). To simplify the notation and make the equations more concise, we introduce:
Ideal attention in Vision-Centric Tasks: An attention distri- bution pa (a y) minimises the next-token prediction loss of VLM, LyLM; DM Attention Proximity: Empirically, the DM's attention is more concentrated (lower entropy) and hence closer to p" than the VLM's, supported by Figure 3, experiments in Section 6.1 and detailed justifications in Appendix G; Shared Dataset: Both models use the same image-text set ((x(0) ())];; Fixed DM Parameters: Op is kept fixed: only is updated; Pretrained VLM Parameters: 0 is further fine-tuned with an attention alignment loss.
()) (0) PVIM(al (9) (0,00))))))))))))) PoM(al>>) (yy))))))))))) 77)
This represents the pointwise difference between the VLM's and DM's attention distributions for the i-th data point, serving measure of divergence each attention location a. Assuming that these differences are Gaaussan-distribhnn with equal variance, the likelihood can be expressed as:
2.3. Bayesian Derivation
p(ADM)0) x exp (- (-2
Our objective is to update the VLM parameters 0 such that the model not only performs well on its primary task but
8000 (00((0))))
(8)
3

---

Diffusion Instruction Tuning
This corresponds the attention alignment loss Lu (0):
per-word attention distributions PoM(aka, y; 0D) indicate salient image regions for each token, shown in Figure 3. We leverage these maps as a proxy for the optimal poste- norr attention distribution p" (a [a/xr, y). guiding the VLM's alignment toward more focused vision-text interactions.
Lon Σ 1000
(9)
By minimizing Lan (f), i..., the MSE loss, we encourage the
VLM's attention to align with that of the DM, guiding it to-
ward the optimal posterior attention distribution p" (alz, y)-
3.2. Attention Aggregation in Vision-Language Models
For simplicity, we assume a -on-informeeive prior p(0). Consequently, the posterior distribution in the previous equation Equation (4) is governed primarily by p(D(0) and ((ADM// ()) If regularisation is needed, more informative prior can be seamlessly incorporated. Combining the terms, the negative log. posterior (up to constant) becomes:
In VLM transformers, each text token T₃ attends to image patch tokens Tp across multiple heads and layers, producing attention weights whl ((.p)' where E H, E L, E T.p E P. with H, L, T, P being heads, layers, tokens, and patches, respectively. These weights capture semantic and spatial relations between tokens and patches. To obtain single- channel per-word map, aggregate Next Naatch H X L attention heads/layers into a (Neest x N pmnnch) matrix. Then we reshape the patch dimension into / /Npatch X VNpatch grid, approximately reconstructs the original layout of the image, which we verify in Appendix This procedure yields interpretable saliency maps- each row corresponds text token's 's focus the image patches faahhes-faiilitttiinn alignment with DM attention (see Figure 5). C 1
(0)
(0) = (0) + (O).
(10)
Here, À balances the importance of aligning the attention distributions with the primary task. We fully justify the inclusion of the attention alignment loss in Appendix H.
2.4. Practical Implementation
For each sample (x'(",y) ()),,, we extract per-word attention from both models. Then, we fine-tune 0 by minimizing:
H x
(0)
> log (1 Ma (0:0) + A > (0))?
pink
(11)
bear
bear
Ness
This is model-agmostic, requires additional data, and encourages the VLM's attention to become more focused by following the DM's 's distributions.
on
o8
bike
L. salay tuxx
bike
Nyaa
3. Attention Alignment
Figure 5. An illustration attention aggregation process in VLMs. Attention weights between text tokens and image patches are aggregated to form per-word saliency maps that approximate the spatial layout of the image.
We discuss how to compute per-word attention in VLMs and DMs. Although both employ attention capture vision-text interplay, their attention aggregation differs (Figure 4). Un- derstanding these distinctions is key to effective alignment.
3.2.1. 1. SIMPLE AGGREGATION FUNCTIONS
3.1. Attention Aggregation in Diffusion Models
A straightforward approach is to pool attention weights A (i.e., whit across heads H and layers L via mean max. We consider four strategies:
Text-guided diffusion models generate images from textual input by iteratively denoising random-noks image. During each denoising step, cross-attention layers enable the model to focus on relevant textual tokens. Specifically, queries Q are derived from the noisy image Zt, while keys K and values V come from the text embedding v:
A mean/max (A) (max-max, max-mean, mean-max, mean-mean),
where each denotes a combination of [Max, Mean] over and L. This yields a single per-word attention map, capturing a coarse measure of word-to-patel to-eh-t--tthhhhee alignment.
Q fo(ee), K fx(,,, Vv(v,,),
(12)
3.2.2, ATTENTION FLOW
where fo. fK. and fy are pretrained projection matrices of DM. The attention map M is then computed as:
Proposed by Abnar & Zuidema (2020), attention flow cumu- latively combines multi-layer attention to capture deeper in- teractions than simple pooling. Starting with the first layer's attention AD), we iteratively merge subsequent layers (1) via element-wise multiplication or addition:
M = Softmax ((QKT) ))/d).
(13)
Prior work (Hertz et al., 2022) shows that averaging these maps across layers and time steps reveals meaningful corre- spondences between words and image regions. The resulting
A f- Ao A!) or Ā A A

---

Diffusion Instruction Tuning
Attennon Aggropaneo
Atsosion Alignmon
Pear
preserving the original pretrained parameters.
Migarr Neteark
3.4. Lavender Integration
Cross-Attention. For VLMs with dedicated cross- attention layers, each head produces word-to-patch weights ((t.p) mapping text tokens T₃ to image patches Tp. We can reshape these weights into spatial grids and aggregate across heads/layers, then apply the Aligner network to yield final per-word saliency maps comparable to DM attention.
200 bear
Ve cud www. 3014
Fara lad SONEC
A Scttas(fQR 1/
- (1
Figure 6. Learning to aggregate with parallel attention. The demonstration is based on a self-attention VLM. The parallel at- tention constitutes about 1/5th of the total VLM layers (I' E L.).
Self-Attention Only. When both text and image patches are interleaved in a single sequence, tokens attend to each other in a bidirectional or causal manner. To extract word-to- patch correlations, we first separate text tokens from image patches, filter out irrelevant attention connections, and re- shape the relevant weights into a grid. Despite the extra steps (e.g., token indexing, handling masks, interpolation), the principle remains the same: aggregate the transformed weights into per-word saliency maps, then align them with DM attention via the Aligner, as demonstrated in Figure 6, This allows Lavender to improve vision-text alignment even in fully self-attentive architectures. A more detailed expla- nation of both scenarios provided in Appendix K.
This method, also used by Lin et al. (2024) at sentence level, extended here to finer-grained word-level attention maps, potentially revealing semantic correlations overlooked by simpler aggregation. Further details are in Appendix I.
3.2.3. LEARNING THE ATTENTION AGGREGATIONS
Beyond fixed pooling methods, we introduce parallel cross- attention parameters (WQ.: WK,, a Wv,) alongside the pre- trained projections (We, K, Wv). By preserving the orig- inal attention mechanisms and learning new ones, we cap- ture richer semantic correlations without overwriting exist- ing weights. During each forward pass, we compute both the original attention A and parallel attention Aa, then use Ad as the VLM's attention to align with the DM:
4. Implementation (Short Summary)
We integrate Lavender with three Vision-Language Models (VLMMs)-cross--teee cross-attention VLMs (OpenFlamingo, Llama 1-2-iBB-vision Instruct), and self-attention VLMs (MiniCPM-Llama 3-v2.5) and use Stable Diffusion v1.4 (Rombach et al., 2021) to provide per-word attention targets. We add lightweight Aligner network and an attention align- ment loss to guide the VLM toward the DM's more focused distributions. Further details DM inversion process, at- tention extraction, training hyperparameters, and computing environments are provided in Appendix L.
A Softmax (QK (QKT)/ dx).
(14)
Aa- = Softmax (Qak] //v/d))/vv)))))
(15)
Empirically, learning parallel aross-attention in about 1/5th of the layers (see Figure 6) suffices for effective alignment while retaining the original VLM's core knowledge.
3.3. Aligner Network
5. Training and Dataset (Short Summary)
To refine the parallel (or aggregated) attention Aa into single-channel map comparable to (a/r,)/, y; 0D), we in- troduce a lightweight Aligner network inspired by Squeeze- and-Excitation (Hu et al., 2018). contains several small (3-5) layers (MLP or convolution) that expand, apply non- linear activations, then squeeze back to a single-channel map. Empirically, we found convolutional layers better cap- ture local spatial cues than MIP,, detailed comparisons are provided in Appendix N. During fine-tuning, the Aligner output is compared to the DM's attention via:
We adopt several strategies to stabilise alignment objectives and preserve a VLM's pretrained capabilities. First, option- ally pretrain only the Aligner network, freezing the rest of the model absorb DM-based attention signals without disrupting existing representations. Second, attention aggre- gation ($3) can stabilise the alignment. The Aligner has a configurable depth balancing complexity and accuracy. We also use PEFT (e.g., LoRA) to prevent catastrophic forget- ting. Finally, we apply 'root word match' or 'exact word match' select predicted words for alignment. For datasets preparation, process images with Stable Diffusion to ob- tain per-word attention targets across Flickr30k, Laion50k, RLAIF-V 83k, and OCRVQA300 Full details, including hyperparameters, dataset preprocessing, and design choices, are provided in Appendix M.
(0)( Σ Aligner ( ())(() y 0
(16) guiding the VLM's attention toward the DM's more focused distribution, capturing complex semantic correlations while
5

---

Diffusion Instruction Tuning
6. Experiments and Results
Histogratss Loss Score
We first validate Lavender on a smaller setup with Open- Flamingo ($6.1) and then scale it to MiniCPMv2.5 (Yao al., 2024) and Llama 3.2-1IB 1--1BB-iision Instruct (Dubey et al., 2024), evaluating on 20 standard VLM benchmarks and comparing against 23 baseline models ($6.2). We further analyse data overlapping ($6.3), scaling behaviour ($A.2), out-of-distribution medical tests ($6.5), and provide qualitative visualizations $6.6.
1.0
Mileel:
Qualnyy efgruvedd tatt CTDD waan)
64
6.1. Empirical Verifications
42
We find that: 1) Figure 7 shows entropy measurements across Flickr30k, RLAIF-V83K, and OCRVQA30K confirm- ing that DM attention distributions are lower in entropy than VLM attention, supporting they are closer the ideal distri- bution p" (a/æ, y) vision-centrie tasks. Figure shows how VLM attention maps align with DM maps, achieved by using exact word match sampling, convolutional Aligner networks, and moderate learning rates during Lavender fine- tuning of OpenFlamingo on Flickr30k. 3) This alignment also boosts text generation performance, as evidenced in Figure by improvements in COCO captioning scores cor- relating with reduced MSE loss when jointly minimising LVLM and Lan- 4) Benchmarking Lavender against autore- gressive fine-tuning shows consistent zero-shot gains, with up to 72% improvement (Figure 10). Detailed results and further analysis are discussed in Appendix SN.
0.20
0.06
MSE Love
Figure 9. Lower MSE aligns with higher caption quality. Re- sults COCO using Sevvender-OpenFlaminne F.T. with Flickr30k.
Zero-Shot Lavender estorageessve F.T. LaianSik
-265
VCB
255
1243
2.2
Seono (5)
Zero-Shot Lavender (ours) vs. Astorggrestive F.T on Flicke30k
<8.
165
-42 0
A.be Fabee Minn til Vili Beee wellanc
30 40
50
60
70
5.Srrr (h6)
Figure 10. Lavender surpasses the autoregressive baseline by to 72% in zero-shot evaluations, with models trained on Laion50k Flickr30k. The negative gain Hateful Memes stems from its unique ranking task rather than captioning.
1500
Baselines include three groups: small budget-constrained models (< 10B parameters), small data- data-heavy models (< 20B), and large-scale So'A models (> 20B). Details of benchmarks, baselines, and evaluation are in Appendix O.
Figure 7. DM attention is more concentrated compared to three VLMs attention famennlamingo, MiniCPM-r2.5, and Llama 3.2- 11B). Full results are available in Appendix N.I.
Figure 8 Aligning VLM at- tention with DM attention for the word "guitar." Each row adds training tech- nique. See Appendix N.1 for examples.
Main Results with MiniCPM-V-2.5 and Llama 3.2- 11B. Figure 11 compares results for MiniCPM-V-2. (self- attention-only) and Llama 3.2- IB- Vision-Instruct (with cross- aussnaattnnion). For MiniCPM-V-2.5, Lavender outper- forms autoregressive fine-tuning 16/20 tasks, improving performance by up to 4% while limiting drops to -1 de- spite challenges posed by dataset reuse and the self-attention mechanism. For Llama 3.2-1 Lavender achieves to 30% gains on 19/20 benchmarks with LoRA and up to 25% on 17/20 with full fine-tuning. These results underscore Lavender's robustness across VLMMs, with our experiments indicating that explicit cross-attention modules align more effectively than self self-attention-nnt only models like MiniCPM.
6.2. Scaled Results with Lavender
Next, we scale experiments to MiniCPMv2.5 and Llama 3.2- BB-BBB-ii Iision- fine-tuning on RV83k, Flk30k, and OV30k datasets. Models were trained using autoregres- sive and Lavender methods combined with LoRA or full fine-tuning and evaluated on 20 multimodal benchmarks, comparing against 23 baseline models.
Evaluation and Baselines. Lavender was tested across diverse benchmarks grouped by task type (Chart&Doe, per- ception, real-world understanding, hallucination detection).
6

---

Diffusion Instruction Tuning
Lora Fine-tune MiniCPM-V-2.5 (Autoregressive)
Lora Fine-tune MiniCPM-V-2. (Autoregressive + Lavender Alignment)
Improvement after Finetune
Lavender Fine-tune improved over Llama-3.2 in 16/20 cases
Attention Alignment farther improved over Fine-tune 15/20 cases
76.9 764.3
20142
74.9 75.0
721.0
7170
MMBrrchoev
AMM
At2DasT
POPE
DeeVgAve
Hallmiantiench
CharQAAssT
MMss
OCRVOARSTCOR
Isfe VOAVAL
MMBasooor: vi
SEEDBettag
Tea
CORVOL TEST
OCRBlesh
RestWirtsOA
AMBBeneoory
SeionceQAAes Qreest
Clilench
MMMU (1a))
(a) Lavender enhances MiniCPM-LLlma3-Ve despite further fine-tuning on its pre-trained dataset.
Lora Finc-tune Llama-3.2-1 BB (Autorcgressive)
39.2
81.1
Improvement eemeeteettt after Finetune (%)
Lora Fine-tune Llama-3.2-11B (Autoregrossive Lavender Alignment)
20
Lavender Fine tume improved over L.2mm-3.2-111 3.B.11BB in 19/20 cases
Attention Alignmont farther improved over Fine-tune in 18:20 casus
41.2
81.3
88.1
90.3
05.1
65.1
1871.4
71.5
86.0
73.6
79.0
81.5
48.7
62.0
84.7
264.0
86.5
88.1
$1.1
36.5
0
60.7
1664.6
46.3
733.0
-10
74.6
MMMU (val)
A1DDrEST
SAvanceQAval
MMSur
RealWarldQA
Sciences AYEST
OCRBench
SEEDBenaggs
OCRVQATESTCOmE
POPE
Tex VOAAL
DoeVQAVAL
MMBendhozy.co V11
VOAVAL
HallusionBend
MME
MMBenchoey, eaaeeeoeveeeee EN
MMBenchoEv. CCM
MMBenchouv ON vI1
CCBench
Full Fine-tune Llama-322-1 (Autoregressive)
37.6
Improvement after Finetune (%)
Full Fine-tune Llama-3 2-111 2BB
(Autoregressive Lavender Alignment)
20
Lavender Fine-tune improved over Llama-3.2-111 2-11B in 17/20 cases
10
Attention Alignment further improved over Fine tume in 17/20 cases
89.8
75,8
65.0
72.0
49.1
81.6
62.2
84.3
73.1 87.5
668.
70.1
81.5
1695.
37,7
0
76.6
59.3
$1.0
53.6
-10
72.0
686.0
-0
MMMU (val)
Al2DrEsT
SuienceQAval
MMStar
RealWaidDQA
SeiemceQ Argar
OCRBench
SEEDBenagah
OCRVQ OCRVOATESTCORE ATESTCORE
POPE
TeRVVQAVAL
MMBorchoEy ULL
DocVQAAAL
Inf InfoVOAval
MME HalhasonBench
MMBlenclugy EN
MMBBenshogv ON
MMBBendtoev. ON V21
CCBench
(b) Lavender mitigates catastrophic forgetting for Llama-3.2-11B fine-tuned on new dataset.
Figure 11. Relative Improvement Over Baseline on 20 Benchmarks. (a) Results for MiniCPM-Llamase V-25 fine-tuned on its original RV83k dataset. Lavender improves performance on 16/20 benchmarks with gains up to 4%, while limiting performance drops to -1%, primarily on an cuttof-ddisrrbuttion Chinese benchmark. (b) Results for Llama-3.2-1 3.2-11...1BBBB. 1--ViiB-iiiion-- Vision-Instruch fine-tuned on a mixture of RV83k. FIk30k, and OV30k using LoRA and full fine-tuning strategies. Lavender outperforms autoregressive fine-tuning on 18/20 (LoRA) and 17/20 (Full-FT) benchmarks, achieving up to 30% and 25% improvement, respectively, while mitigating catastrophic forgetting. Across both models, Lavender demonstrates consistent benefits over autoregressive fine-tuning, particularly in reducing catastrophic forgetting.
7

---

Diffusion Instruction Tuning
Benchmarking Lavender Against External Baselines. Lavender focuses enhancing VLM performance be- yond standard next-token fine-tuning. To contextualise its improvements, we compare Lavender against a range of baseline models across 16 benchmarks. Key results are summarised in Figure 15, with detailed comparisons in Mod-
Mean Normalized Soore
55-
45.
Appendix Table 1. For Small els, Lavender achieves up to 50% improvement, with mi- nor deficits (within 4%) on benchmarks like SEED-IMG, Among Small Data-Heavy SOTA Models, Lavender outper- forms autoregressive baselines, despite using significantly smaller datasets (38x-384x less data), with gaps attributed to dataset differences (Section 6.3). For Large SOTA Mod- els, Lavender performs comparably to some closed-source models on certain tasks demonstrated in Figure 12.
43
35-
Lavolceff: to0ta-mooo--oooo
3000
30000
36000 25500
36000
Data Points Sampled Over Iterations
Figure 14. Lavender scales better and mitigates overfitting com- pared to autoregressive fine-tuning, with larger datasets redue- ing variability. This plot shows the mean normalised performance across eight benchmarks for two dataset configurations after fine- tuning Llama 3.2- IB. Markers indicate observed performance with trendlines, and shaded regions show variability as standard deviation around the mean. Full results with four dataset configu- rations are in Appendix Figure 18 and Figure 28.
6.5. Severely Out-of Distri--iitiittiion Medical Benchmark
We evaluate model generalisation on the extreme OOD WorldMedQA-V (Duan et al., 2024), a multilingual, mul- timodal medical VQA dataset. It includes 568 multiple- choice questions four low-resource languages, paired with medical images. The dataset's focus on medical exams and rare languages makes it ideal for testing generalisation beyond typical fine-tuning domains. Figure 16 shows that I Lavender improves Ilama-3.22 3.2-1 1B's performance by 68%, surpassing six open-source models (6B-34B) and narrowing the gap with large closed-source models from 43% to 10%.
Figure 13. Minimal data over- lap supports Lavender's gen- eralisability. Overlap scores are based on: 3 for explic- itly shared benchmark datasets, for shared sources (e.g.. COCO), and 0 for no overlap.
OC3605
Figure 12. Lavender matches certain high-resource SOTA models.
6.3. Data Overlap Analysis
6.6. Qualitative Results with Llama 3.2-11B
To contextualise benchmark results, we analyse the overlap between fine-tuning and benchmark datasets. Using a quali- tative approach (Figure 13), we find that fine-tuning datasets for some SoTA models show higher overlap with bench- marks. Lavender's fine-tuning dataset exhibits low overlap, similar to VA-155 (Liu et al., 2024a), highlighting its strong generalisability. This analysis focuses on fine-tuning, where smaller datasets pose higher overfitting risks, though pretraining overlap can't be ruled out.
Lavender effectively aligns VL.M attention maps with those of DMs and enhances performance on VQA tasks. Exam- ples of aligned attention maps (Figure 19) show that Laven- der's attention maps correlate well with semantic regions similar to those of DMs. Additionally, Figure 17 high- lights improved performance on diverse VQA benchmarks attributed to the better localisation and interpretation of vi- sual elements with Lavender. Full details in Appendix A.
6.4. Scaling Behaviour
7. A Ablation and Analysis Summary
Lavender, a model-agnostic approach, is tested on small fine-tuning datasets in this work due to computational con- straints. To evaluate its scalability, we fine-tune Llama- 3B on combinations of RV83k, Flk30k, and OV30k using autoregressive and Lavender methods with LoRA or full fine-tuning. Results in Figure 14 show average per- formance across eight benchmarks. The findings indicate that Lavender scales better with increased data, effectively reducing overfitting- a common issue with autoregressive fine-tuning on small datasets.
We conducted extensive ablation studies to assess the key components of Lavender and their impact on performance. Key findings include: 1) Attention Aggregation: Learned aggregation consistently outperformed manual methods like attention flow due to its adaptability and scalability (Fig- ure 20, Figure 21). 2) Pretraining the Aligner: Pretraining the Aligner Network significantly mitigated catastrophic for- getting compared to plain fine-tuning with longer pretrain- ing benefiting more complex benchmarks (Figure 22). 3) Fine-tuning Strategy: LoRA offered better short-term results,
8

---

Diffusion Instruction Tuning
50%
Small Budget--Conttrinee SOTA
MiniCPM-VV2255
Improvement over Small SoTA (%)
Llama-3.2-11B
MinsCPM--V2.5 Lavender Lora-FT
3.2-11-
Lora-FT
30%
Real- World
Visual Hallucination
Perception & Milti-Discipline Reasoning
Chart ChargaDaagamm Document
Understanding
Understanding
20%
10%
0%
-10%
RA IMG
7075
MDBhwai Sc 40A
K00Ml
OChocb
A15O
Dw/VOS
OCKVOA
OrRbed
Figure Lavender improves MiniCPM-V-255 and Llama-32-11BB, surpassing Small Key zero-shot accuracy results across 16 VLM benchmarks, with the greatest gains in Chart, Diagram, and Document Understanding. Moderate improvements are seen in Perception and Hallucination, while Real-Word World Visual Understanding shows the smallest gains. Detailed results including 23 baseline models presented in Appendix Table 1.
Constrained SOTA by up 50 %.
Eul
Full details are provided in the Appendix C.
Limitations and Future Works. Lavender's evaluation was limited datasets of up 0.13M samples, far smaller than those used by state-of-the-an models, which scale up to 50M samples (Figure 14). Future work could explore its scalability with larger datasets and tuning durations. Laven- der relied on Stable Diffusion v1.4, but adopting higher- resolution models like Stable Diffusion v2 could improve at- tention map accuracy, albeit with greater resource demands. Additionally. the short inversion steps used generate atten- tion maps prioritised efficiency but might limit accuracy for unfamiliar words. Lastly, while Lavender was applied to the self-attention-one MiniCPM-v-22.5, further research into optimising self-attention alignment and bridging its differ- ences with cross-attention mechanisms remains an open area for exploration. Full details are available in the Appendix
21
W.VWagA Vire Amogge
Figure Lavender boosts Llama-3.-111Bs performance the OOD WArldMedQA benchmark by 68%. Results are based on fine-tuning Llama-3.2-15 3.2--11-----BB-- LIB-Vision- Vision-Instruct on RV83K. Flk30k, and OV30k using autoregressive Lavender methods with LoRA. Accuracy reflects average performance across four low-resource medical VQA subsets, evaluated via Vlmevalkit (Duan al., 2024). Example results in Appendix Table 2.
while full fine-tuning showed advantages on more complex tasks, suggesting complementary benefits (Figure 11). 4) Layer Alignment: Aligning all eight cross-attention layers in 1.2ma-3.2-113 was most effective, outperforming partial alignments (Figure 23). Full details and additional analyses provided in Appendix B.
9. Conclusion
We have introduced Lavender, method that leverages the precise text-region alignment of Diffusion Models to en- hance Vision-Langual Models (VLMs) through efficient supervised fine-tuning. Lavender enables significant perfor- mance improvements while remaining highly data-efficient and compute-friendif Our findings highlight that Laven- der effectively aligns VLM attention with Diffusion Mod- els, improving robustness across diverse domains, includ- ing challenging and multilingual benchmarks. By incor- porating techniques such parallel attention and LoRA, Lavender balances attention alignment with the preserva- tion pretrained knowledge, ensuring consistent perfor- mance gains without catastrophic forgetting. This scalable, model-agnostic approach demonstrates the promising poten- tial for advancing text-vision alignment in VLMs and lays the groundwork for future research into efficient fine-tuning and cross-modal alignment techniques.
8. Failure Strategies and Limitations
Failure Strategies. Despite Lavender's ' overall success, certain approaches proved less effective. Fully fine-tuning without pretraining often destabilised the model, particu- larly on small datasets, though LoRA offered more robust alternative (Figure 22). Frequent switching between train- ing strategies (e.g., alternating between full fine-tuning and aligning cross-attention subsets) harmed performance due to the model's inability to stabilise objectives during short training periods. Additionally, incorporating extra datasets, such as OCRVQA, sometimes reduced performance (Fig- ure 14), suggesting risks of overfitting with specific data.

---

Diffusion Instruction Tuning
Impact Statement
Attention Alignment as Vision Feedback in Reinforce- ment Learning. Our proposed attention alignment loss provides scalable solution for vision feedback during the RL post-training phase. By leveraging the visual exper- tise encoded in image generation models, it eliminates the need for costly, fine-grained human annotations and mit- igates bias manual feedback. In effect, it replaces or augments traditional human visual feedback with "diffusion feedback", streamlining the process, reducing training costs, and democratising access to advanced multimodal systems.
Synergy Between Models: Building a Stronger Vision- Language Ecosystem. Lavender is model-agnostic framework that requires no additional human annotations on existing data, making it broadly applicable for enhane- ing wide range of VLMs. Given that text-vision align- ment is central to most VLM tasks, our attention alignment approach is poised to benefit most downstream applica- tions, as confirmed by extensive benchmark evaluations. Although primarily designed for terss--ttention-quuuiee VLMs, early results show that Lavender also works with celf-attention-onle models, opening the door to converting widely available LLMs into effective VLMs without re- training. This capability unites the strengths of LLM and diffusion models into more powerful multimodal expert.
Data Scarcity. Both the language and vision communi- ties face looming data shortages, with paired vision-text katasets-ther "crude oil"" of VLM teaiiing-bingg particu- larly scarce. End-to-end training from scratch is resource- intensive and often impractical. Large-scale LLMs and DMs have been trained on multi-billion- billion-level datasets, making it inefficient if their knowledge remains isolated. Lavender bridges these models using limited resources-requiring as little as few thousand samples and one day of training on 8 Nvidia A10G GPUs enabling small models (below Geaslenabling
13B parameters) to perform on par with much larger models (over 50B parameters) across multiple henchmarks.
Data Privacy. By leveraging the extensive prior knowl- edge of diffusion models, Lavender aligns external VLMs with small, local datasets through automatic per-text atten- tion map labelling. This approach is ideal for organisations with limited sensitive data and constrained computational resources, as it enables them to benefit from large-scale pretrained models while keeping data local.
VLM Attention Alignment with Other Vision Founda- tion Models. Lavender enhances text-vision correlation by directly aligning attention within LLM transformer lay- ers. Although our current alignment objectives are derived from Stable Diffusion's 's attention maps, the same methodol- ogy can be applied to other vision foundation models.
Alternative Multimodal Modalitics Beyond Vi- sion-Language. While our results focus on language and vision, the core idea of aligning cross-attention layers is broadly applicable. Potential applications include text-to-audio alignment (Radford et al., 2023), sequence-to- structure tasks in protein generation (Jumper et al., 2021; Yang et al., 2023), and diverse biomedical applications (Tu et al., 2024; Yang et al., 2024) involving genomics, radiography, pathology, and mammography.
10

---

Diffusion Instruction Tuning
References
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H., Zhang, X.-P., and Tang, Y. Tem-adapter: Adapting image-text pretraining for video question answer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13945-13955, 2023.
Abnar, S. and Zuidema, W. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.0092. 2020.
Agrawal, P., Antoniak, S., Hanna, E. B.,, Bout, B., Chaplot, D. Chudnovsky, J.. Costa, D., Monicault, B. D., Garg, S., Gervet, T., Ghosh, S.. Héliou, A., Jacob, P., Jiang, A. Q., Khandelwal, K., Lacroix, T., Lample, G., Casas, D. L., Lavril, T., Scao, T. L., Lo, A., Marshall, W., Martin, L., Mensch, A., Muddireddy, P., Nemychnikova, V., Pellat, M.. Platen, P. V., Raghuraman, N., Rozière, B., Sablay- rolles, A., Saulnier, L., Sauvestre, R., Shang, W., Solet- skyi. R., Stewart, L., Stock, P. Studnia, J., Subramanian, S., Vaze, S., Wang. T., and Yang, S. Pixtral 12b, 2024. URL htw.s::ttt/ee.e /ttp:///axvvv.//. org/ abs 07073.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J., Qiao, Y., Lin, D., et al. Are the right way for evaluating large vision- vision-language models? arXiv preprint 2403. 20330, 2024a.
Chen, L.,, Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi- models with better captions. In European Conference on Computer Vision, pp. 370-387. Springer, 2025a.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint TrXiv:/504.0033 2015.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems,
2022.
Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling, 2025b. URL https::////. /ttp///aavvvv/.. org/ /atpa/abb//........
Anthropic, A. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.,, and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L.., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. Openflamingo: An open-source framework for training large autoregressive vision-languee models. arXiv preprint arXiv: 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P. Wang. S., Wang, W., Wu, S., Xu, B., Xu, J., Yang,
Covert, I., Sun, T., Zou, J., and Hashimoto, T. Locality align- ment improves vision vision-languue language models. arXiv preprint 2. 2024.
A., Yang, H., Yang, J., Yang. S., Yao, Y., Yu, B., Yuan, H. Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023. URL htttps://arst /ttp://xxxvv///////// abs/2309 16609.
Dai, W., Li. J., Li. D., Tiong, A. M. H., Zhao, J., Wang. W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruc- tion tuning, 2023. URL 2305. 2305.06500.
/bs/
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
DeepSeek-AL, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F. Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L., Liang, J., Guo, J., Ni, J., Li, J., Wang, J., Chen, J., Chen, J., Yuan, J., Qiu, J., Li, J., Song, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang,
Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality- enhanced projector for multimodal Ilm. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13817-13827, 2024.

---

Diffusion Instruction Tuning
K., Yu, K., Wang, L.,, Zhang, L., Xu, L., Xia, L.., Zhao, L., Wang, L.., Zhang, I.,, Li, M., Wang, M., Zhang, M., Zhang, M., Tang, M., Li, M., Tian, N., Huang, P., Wang, P. Zhang, P., Wang. Q. Zhu, Q., Chen, Q.. Du, Q., Chen, R. J. Jin, R. I.., Ge, R., Zhang, R., Pan, R., Wang. R., Xu, R.,. Zhang, R., Chen, R., Li, S. S., Lu, S., Zhou, S., Chen, S., Wu, S., Ye, S., Ye, S., Ma, S., Wang, S., Zhou, S., Yu, S., Zhou, S., Pan, S., Wang. T., Yun, T., Pei, T., Sun, T., Xiao, W. L.,, Zeng, W., Zhao, W., An, W., Liu, W., Liang, W., Gao. W. Yu, W., Zhang, W., Li, X. Q Jin, x., Wang, X., Bi, X., Liu, X., Wang. X., Shen, X., Chen, X., Zhang, X., Chen, X., Nie, X., Sun, X., Wang, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yu, X., Song, X., Shan, X., Zhou, X., Yang, X., Li, X., Su, X., Lin, X., Li, Y. K., Wang, Y. Q. Wei, Y. X., Zhu, Y. X., Zhang, Y. Xu, Y., Xu, Y., Huang, Y., Li, Y., Zhao, Y., Sun, Y., Li, Y., Wang, Y., Yu, Y., Zheng. Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Tang, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Wu, Y., Ou, Y., Zhu, Y., Wang, Y. Gong, Y., Zou, Y., He, Y., Zha, Y., Xiong, Y., Ma, Y., Yan, Y. Luo, Y., You, Y., Liu, Y., Zhou, Y., Wu, Z. F., Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Huang, Z., Zhang, Z., Xie, Z.,, Zhang, Z., Hao, Z.., Gou, Z., Ma, Z.,
v2: Parameter-effiiient visual instruction model. arXiv preprint arXiv:2304 2023.
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018
Hernandez, J., Villegas, R., and Ordonez, V. Generative vi- sual instruction tuning. arXiv preprint arXivv2406.11222, 2024.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prettt -oomppp image editing with cross attention control. 2022.
Hu, J., Shen, L.,, and Sun, G. Squeeze-and-excitaton networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Huang, X., Wang, J., Tang, Y., Zhang, Z., Hu, H., Lu, J.,
Wang, L.,, and Liu, Z. Segment and caption anything. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13405-13417, 2024.
Yan, Z., Shao, Z. Xu, Z., Wu, Z., Zhang, Z., Li, Z., Gu, Z. Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Gao, Z., and Pan, Z. Deepseek-v3 technical report, 2024. URL htaps://arxivv
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with it- erative attention. In International conference on machine learning. pp. 4651 1651-464. PMLR, 2021.
abs////////// 2412 2 9437.
Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang. Y., Park, S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. Molmo and pixmo: Open weights and open data for state-of-the-ard multimodal models. arXiv preprint arXiv:2409.171 17146, 2024.
Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., Gao, Z.., Zhang, X., Li, J., and Xiong, H. From clip to dino: Visual encoders shout in multi-mode large language models. arXiv preprint arXiv:23 10.0000..822.. 2023.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Jin, C., Tanno, R.. Saseendran, A., Diethe, T., and Teare, P. An image is worth multiple words: Learning object level concepts using multi-concept prompt learning. arXiv preprint arXiv:2310.122777 2023.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. Vimevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198- 11201, 2024.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Židek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873) 2021.
Dubey. A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilaaa 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Zamir, A., and Tombari, Brave: Broadening the visual encod- ing of vision-language models. In European Conference Computer Vision, pp. 113-132. Springer, 2025.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., et al. MME: A com- prehensive evaluation benchmark for multimodal large language models. [Tt::2306.13994, 2023.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of cisually-conditioned language models. arXiv preprint arXiv:2402. 2024.
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. Llama- adaterr
12

---

Diffusion Instruction Tuning
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
URL https://llava-ll... wwthtuubb.gii............ holbllg///////////ooo/ 2024011-3--1lav--nee
Liu, H., C., Wu, Q. and
and Lee, Y. Visual instruction tun- ing. Advances neural information processing systems,
36, 2024c.
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33: 2611-2624, 2020.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., and Lin, MMBench: your multi-modal model an all-around player? arXiv:2307.0622 0.220000681....2 2023b.
Koh, J. Y., Fried, D., and Salakhutdinov, R. R. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
Liu, Y., Li, Z., Li, H.,, Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., and Jin, L. On the hidden mystery of ocr in large multimodal models. Technical Report, 2023c.
Liu, Y., Zhang, C., Wang. Y., Wang. J., Yang. Y., and Tang, Y. Universal segmentation at arbitrary granularity with language instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3459-3469, 2024d.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed-Bench: Benchmarking multimodal Ilms with generative comprehension. 2r07v
2023a.
Li, B., Zhang, Y., Guo, D., Zhang. R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., al. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024.
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference on Machine Learning, pp. 12631-22643 PMLR, 2023.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping Lunguage-image pre-training for unified vision- languae understanding and generation. In International confer- ence machine learning, pp. 12888-12900. PMLR, 2022.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D., and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 26439 26455, 2024.
Li, J., Li, D., Savarese, S., and Hoi. S. 2lip-2: Bootstrapping segguage-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/evf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.R. Evaluating object hallucination in large vision- language models. 2rXiv:2305.10333 2023c.
Lin, Z., Wang, Y., and Tang, Z. Training-free free open-ended ob- ject detection and segmentation via attention as prompts. arXiv preprint arXiv:2410.0590
Masry, A., Long, D. X., Tan, Q., Joty, S., and Hoque, ChartQA: benchmark for question an- swering about charts with visual and logical reasoning. arXiv:2203. 10244, 2022.
2024.
Liu, F. Guan, T., Li, Z., Chen, L., Yacoob, Y., Manocha, D., and Zhou, T. HallusionBench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), 1lava-1.5, and other multi-modality models. arXiv:2310. 14566, 2023a.
Mathew, M., Karatzas, D., and Jawahar, C. V. DocVQA: dataset for vqa on document images. In WACV, 2021.
Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., and Jawahar, C. Infographicvqa. In Proceedings of the IEFF/CVE Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Matos, J., Chen, S., Placino, S., Li, Y., Pardo, J. C. C., Idan, D., Tohyama, T., Restrepo, D., Nakayama, L. F., Pascual-Leone, J. M., et al. Worldmedqa-v: a mul- tilingual, multimodal medical examination dataset for
Liu, H. Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. Llava-next: Improved reason- ing, oer, and world knowledge, January 2024b.
13

---

Diffusion Instruction Tuning
multimodal language models evaluation. arXiv preprint arXiv:2410.12722, 2024.
Shi, W., Han, X., Zhou, C., Liang, W., Lin, X. V., Zettle- moyer, L.,, and Yu, L. Lmfusion: Adapting pretrained language models for multimodal generation, 2025, URL https: httss///axxvvooo/e wttp://abbb......1.......
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR); pp. 947-952. IEEE, 2019.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8317-8326, 2019.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images using guided diffusion models, 2022. URL https: / arxiv. org/ abs//bb/// 221/2221..000 09794.
Team, G. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
OpenAl. GPT-4 technical report. arXiv preprint MiXiv:2303.08774 2023.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer. A., Pan, X., Wang, A., Fergus, R., LeCun, Y., and Xie, S. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406. 16860, 2024a.
OpenAI. GPT-4o mini system card, 2024. URL https: //openai.com
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:24122 141.44444..1444 2024b.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In International conference on machine learning, pp. 28492-2855 28518. PMLR, 2023.
Tong, S., Liu, Z., Zhai Y.. Ma, Y., LeCun, Y.. and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9576 2024c.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. righutresolutin image synthesis with latent diffusion models, 2021.
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M.. Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. Towards generalist biomedical ai. NEJM AI, 1(3): Aloa2300138, 2024.
Rombach, R., Blattmann,
A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Saikh, T., Ghosal, T., Mittal, A., Ekbal, A., and Bhat- tacharyya, P. ScienceQA: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 2022.
Wang. P., Bai. S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. Ge, W., et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open carge-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 55278-25294 2022.
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionlim: Large language model is also an open-ended decoder for vision- centric tasks, Advances Neural Information Processing Systems, 36, 2024b.
Wang, W., Sun, Q., Zhang, F., Tang, Y., Liu, and Wang. X. Diffusion feedback helps clip see better. arXiv preprint arXiv:2407.20177 2024c.
Shi, M., Liu, F. Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-.., Yin, H., Sapra, K., Yacoob, Y., Shi, H., et al. Eagle: Exploring the design space for multi- modal ilms with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
x.ai. Grok- 1.5 vision preview. URL https:////// //tx//////. blog/ blog/grok-10 /t/g//rol11111 5v.
14

---

Diffusion Instruction Tuning
Yang, L., Xu, S., Sellergren, A., Kohlberger, T., Zhou, Y., Ktena, I., Kiraly, A., Ahmed, F., Hormozdiari, F., Jaroen- sti, T., et al. Advancing multimodal medical capabilities of gemini. arXiv preprint arXiv:2405.03/12. 2024.
interleaved with text. Advances Neural Information Processing Systems, 36, 2024.
Zong. Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li,
H., and Liu, Y. Mova: Adapting mixture of vision experts multimodal context, arXiv preprint arXiv:2404. 1ete22444.33444 13046. 2024.
Yang, Z.,, Zeng, X., Zhao, Y., and Chen, R. Alphafold2 and its applications in the fields of biology and medicine. Sig- nal Transduction and Targeted Therapy, 8(1): 15, 2023.
Yao, Y., Yu, T., Zhang, A., Wang, C.,, Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z.,, Chen, Q. Zhou, H., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li. D., Liu, Z., and Sun, M. Minicpm-v: A gpt- 4v level mllm on your phone. arXiv preprint 2408.01800, 2024.
You, H., Zhang. H., Gan, Z., Du, X., Zhang. B., Wang, Z. Cao, L., Chang, S.-F., and Yang, Y. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310. 0nt:23300.77000 10.07744, 2023.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computatioonal Lin- guistics, 2:67-78, 2014.
Yu, T., Zhang. H., Yao, Y., Dang, Y., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., and Sun, M. Rlaif-v: Aligning milms through open-source feed- back for super gpt-4v trustworthiness. arXiv preprint arXiv:2405. 20224405..2220 17220, 2024.
Yue, X., Ni, Y.. Zhang, K., Zheng, Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging alud with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zhou, C.. Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one modti- model. arXiv preprint arXiv:2408. 1039, 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304.1 10594, 2023.
Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y. Multimodal c4: An open, billion-scale corpus of images
15

---

Diffusion Instruction Tuning
Appendix Table of Contents
Appendix Extended Key Results
A.1. Visual Results of Scaled Evaluations on 16 VLM Benchmarks
17
Table 1. Big table showing results across 16 VLM benchmarks, including 23 baseline models
18
A.2. Scaling Behaviour
19
A.3. Visual Results of Aligned Attention Maps with Llama 3.2- 1B
20
Appendix B- Ablation and Analysis
B.I. Attention Aggregation Functions
20
B.2. Training recipes
21
Appendix C- Failure Strategies
22
Appendix Limitations and Future Work
22
Appendix E Related Work
23
Appendix F- Full Pseudo Code of Difffusion Instruction Tuning
24
Appendix G Rayesian Justification for DM Attention Proximity
24
Appendix H Justificationn of the Attention Alignment Loss
26
Appendix Additional Details on Attention Flow in Vision-Language Models
28
Appendix J Appropriate Rearrangement and Reconstruction Matter
28
Appendix Additional Details on Attention Alignment: Integrating Lavender
K.1. Incorporating Lavender into Cross-Attention VLMs
.30
K.2. Incorporating Lavender into Self-Attention-Only VLMs
30
Appendix L Full Implementation Details
30
Appendix M Full Training Recipes and Data Preparations
31
Appendix N Empirical Verification (Detailed)
32
Appendix Evaluation and Baselines Settings of Scaled Experiments
33
Appendix P- Extended Supplementary Results
35
P.1. Supplementary Per-Benchmark Scaling Results
35
P.2. Supplementary Per-Benchmark Ablation Results
35
P.3. Supplementary Visual Results from the WorldMedQA-- Benchmark Table Failure case analysis on the MME benchmark
39
40
Figure 29. Visually aligned examples from Lavender-L Llmaa 3.2-11B
41
16

---

Diffusion Instruction Tuning
A. Extended Key Results
A.1. Visual Results of Scaled Evaluations on 16 VLM Benchmarks. [Back to Contents]
This section expands Section 6.6. Figure 17 showcases nine VQA examples from multiple evaluated benchmarks, including document, chart, and graphic understanding, real- OCR, geometry math, visual hallucination, and medical VQA. The comparison highlights the performance of Llama-3.2-1 1 1BB- 1----BBiio---iinn Vision-Instruct (Dubey et al., 2024) against its Lavender fine-tuned version. Overall, Lavender demonstrates significantly improved visual understanding compared to the original Llama-3.2 across various VQA tasks, including DocVQA, TextVQA, ChartQA, and WorldMedQA-VV VV. These improvements stem from Lavender's ability to accurately localise and interpret visual elements, leading to better performance on questions requiring detailed visual reasoning. A detailed analysis of these examples is provided in the image caption of Figure 17.
DocVQA
Question:
Math (HalllsionBeechh)
Questoon:Acording to parallel theorem. is anglle 1 angle 2 180 ? Correct Answer: Yes, angle angle? 2 180 correct.
What the heading of the table?
Correct Answer:
conmtimued manuscripss
premaraa 'continued Manuscripts
Llama-3.2: No, angle I and angle
supplementary
TMGG)
hecause they are not supplementary angles. They are corresponding angles. If two parallel lines are cut by conreponding angles congrient.
L.amm--3.2: Trimeposiume- - Publication and Abstract Tracking Report
Lavender: Yos, acconding the parallel lines theorom, anple angle 2 180, This is because when two parallel lines und vur
Lavender: continued Manuscripts Premarin/ TMG
by trassvensal, the correspoeding angles are supplamentaay.
TeaVQq
Question: word is on the left middle buman? Correct Answer: off
TeaVQa
Question: what i the rating soone assigned this aloshol selection?? Correct Answer. 93
Hallusionnennch
Question: the right orange circle the same sire as the
left orange cirele?
Correct Answer: No. The right orange circle larger than the left orange circle.
Linma-3.2:
Lavender: No, rightt orange cirsle significantty
The weed the left midale bution is TMC
Liama- 3.2: The
langer than the lell orange circke, The right oningo circle appeans be the largest circle the imags. while the left orange curele is relatively small compared the surrounding gaa circles.
LInma-3.2: 03.
orange circles
the left and right are the same size.
Lavender: eff
Lavender: 9%
ChartQA (HallusionBench)
Question: According to the chart, does China have the secood largest population
InfoVQA
Q-1: How many types of diagnostics tests there?
Info
Question:
world?
HOW CORDNAYRUS
Who the third in list of most influential USA Olympians? Correct
Correct Amswer: Yes
Answer Q-1:
Q-2: How mony
LIama-3. China does have second largest population in the world
Stops there
Serological besting" ?
hope solo
according the chart. The UK has the second largest popullation with 1,428.6 million people, while China has 1 1,425.7 miilion peuple. making the third langest,
Answer Q-2: 4
I. 1aaaa-322 Ispes.
Llama-3.2: Leftron James
Lavender: Yes. according the chart, China has the second largest
Q-2:
population the world. The chari shows dua China's populition 1,425.7
million, which is significantly higher than the next closest country. United States, with population of 300 million. This indicates that China has substantial lead in temms of population compared other countries.
slas
Lavender: Q-E:
Lavender: Hops Soloy
Q-2:1
WorlAMedQA-VV
Question: [Spanish] Pregamia vincaluda a imagen 1'13 Mujer 72
Lluma-3:2: 3:2: Answer C
Lavender: The correct imswer is D) rectal sheuth of the left abdomen. The disgnosis suggested the image is hematama in the left anterior rectus sheath the abdomen. bocause thu image shows soll tiesue mass the abdominal region, which is located left anterior rectus sheath.
antecadentes Fébrllccsnn suricular en tratamientoo con intioosgllantes orales. Choices: Al Hemni complicada de pared abdominal. B) Seroma post quiringico. C) Absceso
Absccsso la del Explaration: The CT scan shows a fluid colleciion in the lell lower quadrant of the abdomen. which is consistent with abscess. The absoess located putemial space in the ahiomima wall that contains the rectus alvloeninis musele. The abocess is likely caused a bacterial infection, which has spread the recturn the rectus sheath.
inquiendo del ablomes.
Hematora
vaina del
rectus sheath, which a
rocto anteriur inquiendo del abdumen. D) Hermstoma en la vaina del recto onterior inquiento del abdoman. Correct Amswer: D.
Figure 17. Examples demonstrating Lavender's enhanced fine-granularity vision alignment leading to improved accuracy on various VQA benchmarks. In the Doc example on the top-left, with question about the title of a table, Llama-3.2 3,2 mistakenly extracts the section/page title at the top, which is also in bold, while Lavender correctly identifies the table title located next to the table. Similarly, more accurate visual understanding is directly observed in the two TextVQA examples in the second row. Impressively, the example from HallucinationBench demonstrates Lavender's deep visual understanding of geometry. size, and spatial location, leading to robust anti- hallucination behaviour compared to the original Llama-3.2. In the ChartQA and two InfoVQA examples the third row, Lavender exhibits its ability to recognise more detailed information, including relatively small elements within the graphs. In the WorldMedQA-V example, which out-of-distribution, both Llama-3.2 and Lavender understand the question posed in a small language (Spanish) and answer in English. However, Llama-3.2 fails to provide the correct answer due to less accurate visual localization of the unhcalthy region. incorrectly identifics the region as the 'rectus sheath,' which surrounds the actual area of interest, leading to an incorrect response. In contrast, Lavender accurately locates the 'soft tissuc mass", resulting in the correct answer.
17

---

Diffusion Instruction Tuning
MMBendh EV
OCRVQA nQ
MMMU-val
OCRBench
SEED-IMG
SciencoQA
Hallo Bench
CCBench
DocVQA
InfoVQA
MMStar
R. R.Q.QA
TeuVQA
AED
MME
POPE
Model
Base
FT Data
Small Budget-Constrainc Constrained Models
(self-attention only)
LLaV
Vicuna-7B
0.15M
55.5
17.8
28.1
25.8
66.5
1510.0
35.7
33.1
318.0
60.6
86.1
54.8
58.6
69.2
58.2
27.6
LLaVA-NeXt-7B
Vicuna-7B
0.76M
67.0
24.3
74.4
37.1
67.4
1519.0
37.6
37.6
532.0
63.8
87.5
57.8
70.2
70.3
64.9
27.6
Mini- Gemini--nni
Qwen-7B
1.5M
65.8
1523.0
36.8
477.0
71.1
65.2
Cambrian-1 1-7B
Vicuna 7B
10M
74.6
23.7
47.9
40.8
74,6
1802.9
41.8
50.7
614.0
66.0
86.4
60.0
73.3
81.0
77.1
30.6
Eagle-XS-7B
Vicuna-7B
0.93M
73.6
28.4
86.6
1866.0
41.7
551
64.3
89.3
63.8
73.6
712
71.9
35.4
LLaVA-1 5-8B
Lama3-8B
0.15M
69.9
27.8
32.4
27.5
1825.5
39.2
46.1
420.0
61.0
87.3
56.7
70.1
72.2
28.7
LLaVA Next-881
Lama3 8B
0.76M
72.8
32.7
78.5
38.2
74.8
1908
43.1
43.9
531.0
60.7
87.1
58.4
72.5
73.1
65.3
33.1
Cambrian-1e
Lama3-8B
10M
73.0
77.8
42.6
1547.0
42.7
624.0
66.0
71.7
30.6
MiniCPM-V-2.5
Lama3-8B
0.08M
78.1
45.5
84.6
52.1
76.4
2009.1
43.1
50.3
718.0
68.7
86.6
63.9
71.9
88.8
76.6
41.9
+1,ora FT
I L-B
0.08M
77.9
44.9
84.6
52.2
76.4
2014.2
43.1
50.6
717.0
68.8
86.2
64.1
71.9
88
76.6
41.8
+Lavender FT
Lama3 8B
0.08M
77,9
46.3
84.6
52.3
76.9
1990.2
45.0
50.4
721.0
69.0
86.5
64.3
72.1
89.7
76.9
42.0
Alamaa3.22-111
(cross--ttention)
Llama-3.2-1IB
LM32-11B
N.A.
78.7
30.6
82.8
59.0
70.8
1692.9
48.0
48.3
754.0
67.7
86.3
61.4
72.9
83.6
80.8
36.3
+ AutoR. Lora- FT
LM32-11B
0.13M
77.8
39.8
86.0
61.0
73.6
1664.6
43.7
45.1
733.0
70.0
86.9
60.7
73.6
74.6
81.1
36.6
Lavender Lora-FT
LM32-11B
0.13M
79.0
39.2
90.3
65.1
81.3
1871.5
46.3
48.7
764.0
71.5
88.1
62.1
74.1
84.7
88.1
41.3
AutoR. Foll-FT
LM32-11B
0.13M
76.7
37.6
89.8
64.3
1697.4
45.6
46.8
748.0
68.6
87.3
59.4
71.8
77.1
81.5
37.7
+ Lavender Full-FT
LM32-118
0.13M
78.7
38.6
81.6
65.0
80.0
1695.9
49.1
49.8
686.0
70.1
87.6
62.2
73.1
84.3
84.9
39.5
Small Heavy SOTA Models (<: 20 with Massive FT Data (25M)
L.OneVision-7B
Qwen-7B
5.2M
82.4
$4.9
87.5*
68.8*
1993.6
47.9
61.9
622.0
64.7
88.4
69.9
76.7
95.4
78.3*
31.6
Intem/LL--B
1o2emmVL2-88 LL
SM
83.6
77.1
91.6*
74.8*
81.7*
2215.1
51.2
61.5
794.0
42.6
84,2
64.2
15.4
97.1
77.4*
45.0
Qwen2-VL-7B
Qwen2-7B
"50M
83.0
65.7
94.5*
76.5
83.0
2276.3
53.7
60.7
843.0
67.5
88.4
68.5
76.0
85.5
84.3
50.4
Molmo-7B-O
Qwen2-7B
35M
90.7
20.6
90.8
70.0
69.1
1714.7
39.3
50.1
666.0
86.7
67.5
72.7
88
80.4
42.5
Pixtral 12B
Nemo- 12B
N.A.
79.0
37.6
90.7
50.8
77.9
1921.7
52.5
54.5
685.0
64.7
84.2
65.4
71.5
87.2
75.7
47.0
Large State-of-the-AM Models (>
with Massive FT Data (25M)
Cambrian-1-34B
Yi-34B
10M
79.7
49.2
75.5
46.0
81.4
1689.0
49.7
54.2
600.0
68.2
79.7
67.8
75.3
76.8
76.7
41.6
Vissnniinn
Qwen2-72B
5.2M
85.6
63.9
91.3
74.9
85.8
2257.4
56.8
65.8
741.0
86.6
71.9
77.5
90.2
80.5
47.9
Qwen2-VL-722
Qwen2-72B
"50M
88.1
69.8
96.5
84.5
86.5
2482.7
64.5
68.3
877.0
73.7
87.2
77.8
77.9
91.2
85.5
58.1
Molmo-72B
Qwen2-72B
33M
96.3
93.5
81.9
79.4
1992.0
54.1
63.3
701.0
75.2
83.1
46.6
Claude-3 Haiku
N.A.
N.A.
86.7
24.5
88.8
56.1
60.7
1920.0
50.2
38.1
658.0
74.4
45.5
63.3
67.3
39.2
Claude 3.5 Sonnet
N.A.
N.A.
94.7
54.1
95.2
74.3
79.7
1920.0
68.3
62.2
788.0
73.6
60.1
72.2
88.9
74.1
49.9
GPT-4V (0409)
N.A.
N.A.
89.4
57.3
87.2
75.1
81.0
2070.2
63.1
56.0
656.0
81.8
61.4
73.0
84.8
78.0
43.9
GPT-4o (0513)
N.A.
N.A.
94.2
71.2
92.8
79.2
83.4
2310.3
69.1
63.9
736.0
85.6
75.4
77.1
90.7
77.4
55.0
Gemini 1.5 Pro
N.A.
N.A.
94.4
28.4
93.1
81.0
73.9
2110.6
62.2
59.1
754.0
12.3
88.2
64.1
76.0
85.7
78.7
55.9
Llama-3.2-90B Llama-3.1-700
N.A.
92.3*
54.1
85.7
80.4
17410
60.3
55.3
783.0
86.3
68.2
76.8
87.1
44.1
Table 1. Zero-shot accuracy of various fine-tuned models across 16 VLM benchmarks. Results are grouped into four sections based on base model size and the scale of the fine-tuning dataset. The top score for each benchmark within each group is highlighted in bold, Scores for M-225 Llama-3.2-1 IB, and their autoregressive (AutoR.) and Lavender fine-tuned variants are locally evaluated using OpenCompass Vimevalkit (Duan et al., 2024) with 'gpt-40' as the evaluator. All other scores are sourced from the OpenCompass Multi-Modal Leaderboard, evaluated with the same VImevalkit. When lcaderboard results are unavailable, the models' published numbers are cited, marked with * notation if provided. Our observations are categorized as follows: 1) Small Budget-Constricce group is most comparable to Lavender in terms of parameter size and fine-tuning data scale. Both Lavender versions outperform the majority of benchmarks with significant margins over the second-best-peree best-performing external models, achieving improvements up to 40% on CCBench. On few benchmarks (e.g., MMStar, POPF, and SEED-IMG), Lavender is surpassed hy baseline models, though the differencc is within 4% for SEED IMG. Lavender-Llamee 3.2 occasionally underperforms on MME due to privacy protection constraints (see Table 3). 2) Small Data- Mata-Heavy SOTA Models. The primary argument for Lavender in this work is its ability to achieve improvements over the autoregressive fine-tunin inc-tuning baseline. Lavender implementations on MiniCPMv2.5 and Llama-3.2 3.2- 1BB are not designed to beat state-of-the results due the limited fine-tuning data scale (0. 13M), which is significantly smaller than the 5M 50M datasets used for this group, approximately 38x to 384x larger than that used for Lavender. We simplify the comparison by excluding pretraining dataset sizes for base models. In this group, Qwen2-VL--7 and LL LaVA-OneVision- henchmarks. However, we note that the performance gap is likely influenced by the composition of finc-tuming datasets, as discussed in Section 6.3. 3) Large SOTA Models. Finally, we include results from the latest state-of oo-e----teh--r- art models, which are at least 20B in size (or unreleased models typically exceeding 100B) and fine-tur -uned on datasets of at least 5M samples. Despite Qwen2- 1-22-VL-7288 outperforming all other models on 18/20 benchmarks, Lavender pushes the boundaries of 3.2ma-3.2-11BB achieving performance comparable to certain closed-source models that are at least an order of magnitude larger (e.g., Claude-3.5 Sonnet, GPT-4o, and Gemini 1.5 Pro) on benchmarks such as TQA, POPE, World, and Doc
Models. This
are the top performers, surpassing Lavender most
18

---

Diffusion Instruction Tuning
A.2. Scaling Behaviour. [Back Contents]
This section expands on Appendix A.2: Scaling Behaviour in the main paper. Lavender is general model-agnostic approach but evaluated only small fine-tuning datasets in this work, due to limited computational resources, it is important to assess how it may scale with larger datasets. In this section, evaluate multiple checkpoints during the fine-tuning of Llama- 3.2-11B on four combinations of RV83k, Flk30k, and OV30k datasets, using either autoregressive or Lavender methods, combined with LoRA or full fine-tuning strategies. The results in Figure 18 are based on the average performance across eight benchmarks, with detailed results for each benchmark provided in Figure 28. The findings show that Lavender scales better as more data is sampled and effectively reduces overfitting challenge often faced by autoregressive fine-tuning, particularly on small fine-tuning datasets. Additionally, we observe that larger datasets reduce performance variation, an expected behaviour during scaling.
70
Mcan Normalized Score
60
50
40
Autoregressive-FTRRVik
Lavender-FT-RR11k
Autoregressive--T- RV83k
Lavender-FTT RV83k
Autorogressive--F. RV83k +FIK30k
Lavender-FT- RV83k +FI330k
Autorogrossivo-FT- RV83k -FIK30K-OV30K
Lavender-FT: RV83k +FIK30k-OV3OK
30
5000
10000
15000
20000
25000
30000
Data Points Sampled Over Iterations
Figure 18. Lavender scales better and reduces overfitting compared to autoregressive fine-tuning, with larger datasets lowering variations. The plot shows the normalised performance of four dataset configurations across eight benchmarks after LoRA and Full fine-tuning of Llama 3.2-1 IB, as function of the number of data points sampled. Markers represent observed performance for each method-iteration pair, while trendlines with different styles indicate the overall performance trends. The shaded regions around the trendlines represent confidence intervals derived from the standard error, showing the uncertainty of the trendline predictions. Narrower regions indicate higher confidence, while wider regions suggest greater variability. Per-benchmark results are Figure 28. The simplified version is presented in Figure 14.
19

---

Diffusion Instruction Tuning
A.3. Visual Results of Aligned Attention Maps with Llama 3.2-11B. [Back Contents]
This section expands on Section 6.6 in the main paper. A key argument of Lavender is its ability to align the attention maps of VLMs to those of DMs. Figure 19 provides examples of the cross-attention maps from Llama 3.2-11B after full fine-tuning with Lavender. The results indicate that the aligned VLM attention maps generally correlate with the semantic regions of corresponding words in a manner similar to the Diffusion Model. Interestingly, the VLM attention maps after alignment are more concentrated than those of the Diffusion Model. This difference is likely driven by their distinct optimization goals: sense understanding for optimal text generation in VLMs versus pixel-level precision for image generation in DMs.
No 5- yeurg
30 TET
Nt rin
SEA
Figure 19. The per-word VLM attention maps aligned to the Stable Diffusion (SD) after tuning with Lavender. Results from Lavender-I 3.2-11B implementation. More results are available in Appendix Figure 30.
B. Ablation and Analysis
This section expands on Section Ablation and Analysis Summary in the main paper. We conduct ablation experiments analyse the key components and strategies of the Lavender framework and discuss their contributions to its performance.
B.I. Attention Aggregation Functions. [Back to Contents]
We begin by examining different attention aggregation functions in VLMs, a key component discussed in Section 3. These include simple averaging and maximisation operations over layers, referred to as 'layer-mean' and 'layer-max," as outlined in Section 3.2.1; attention flow combined with multiplicative and additive aggregations, referred to as 'flow-multi' and "flow-sum," defined in Section 3.2.2; and learned attention aggregation, referred to as 'learn,' discussed in Section 3.2.3.
Comparing the Aggregation Functions on Eight Benchmarks. We fully fine-tune Lavender-Llane 3.2-11B on a small Flickr-1k subset for 20 epochs and evaluate all fine-tuned models on eight benchmarks without using GPT-API as the evaluator but 'exact-match" in the VQA assessments to reduce costs. Note that this approach differs from the main results in Table 1 and leads to expected performance drop of approximately 10% 20%.
Results in Figure 20 show that the attention 'flow-multi' aggregation performs the weakest overall, likely due to overly compressing and overwriting information through multiplicative aggregation across all layers. This strategy aims to aggregate the most important region for each image-question pair, where the question includes all words, but it proves less effective for per-word attention alignment. In contrast, the attention 'flow-sum' strategy performs well, ranking second overall. This indicates that the additive operation is better preserving information during aggregation. Simple averaging and max operations ("layer-mean' and 'layer-max') demonstrate surprising robustness, achieving mid-level performance overall and excelling on specific benchmarks such as MME and IQA. This suggests that these methods avoid biases introduced by explicit ad-hoc aggregation functions like attention 'flow-muti'. Among all results, learned aggregation consistently outperforms other methods, confirming the design principles of Preserving Pretrained Attention Mechanisms and Capturing Complex Semantic Correlations.
Scalability of Aggregation Functions. We further assess the scalability of the proposed aggregation functions by comparing the eight benchmarks' performance at different training lengths. The average results are shown in Figure 21, with detailed results for each individual benchmark provided in Appendix Figure 29. Despite the consistent overall ranking discussed earlier, the 'learn' aggregation strategy demonstrates the best scalability. Starting as the second worst at epoch 5, it climbs to the best-performing method by epoch 20. This behaviour can be understood in light of the limitations of manually
20

---

Diffusion Instruction Tuning
POPE
2-0--00 MMA
kys/rauu
Row-menn
7.23
90.0
fow-sum
Normalized Mean Score
max--max
70.0
Jayur mean
layer-max
MMOME: (w0))
60.0
Jeam
24
50.0
40.0
30.0
Devit Sat
MBdB
5
10
15
20
Epochs
Figure 20. Comparing aggregation functions eight benchmarks. Results are from Lavender-Llane 3.2-1IB line-tuned on Flickr- Ik for 20 epochs, evaluated with 'exact match' without an LLM judge.
Figure 21. Mean performance over training epochs. Results from Lavender-Llama 3.2-11B fine-tuned on Flickr-lk. Detailed per-benchmark results are in Appendix Figure 29.
designed aggregations, such as 'flow-sum." While manual aggregations may initially reflect straightforward text-to-region correlations, like identifying colours in images, they are inherently suboptimal. These methods act as shortcut, achieving good early performance but lacking scalability when encountering more complex text-to-region correlations. Learnable aggregation, in contrast, adapts these complexities and scales effectively with additional training.
B.2. Training recipes. [Back to Contents]
Pretraining the Aligner Network. This work aims to enhance the aligmment of pretrained VLMs, such as Llama-3.2, which are already knowledgeable. As discussed Section 5, fully fine-tuning such VLMs with a secondary attention alignment objective and small dataset can lead to catastrophic forgetting. The results in the first row of Figure 22 confirm this concern, showing drastic drops in performance across eight evaluated benchmarks after fully fine-tuning dataset for 30 epochs. In contrast, the proposed pretraining strategy significantly mitigates this issue. The pretraining strategy involves pretraining only the Aligner Network for certain number of epochs while keeping the VLM parameters frozen, before jointly updating all parameters. Results in Figure 22 show that pretraining for a short duration (less than one-third of the total epochs) is particularly effective overall. Notably, more challenging benchmarks that require deeper interaction between visual perception, complex reasoning, and domain knowledge, such as MMMU and WorldOA, benefit more from longer pretraining.
small
Full Finetuning Versus LoRA. Comparing full finetuning with the LoRA finetuning strategy for Lavender, our main results with Llama-3.2 in Figure 11 and Table 1 suggest that LoRA is generally more beneficial for the majority of benchmarks. However, we also observe that full finetuning outperforms LoRA with Lavender few more challenging benchmarks, such as MMMU, MMStar, and RealWorldQA, which require deeper interactions between visual perception, complex reasoning, and domain knowledge. This observation is consistent with the results from the ablation study in Figure 22. Given that this work focuses on short finetuning overall, these findings suggest that LoRA finetuning with Lavender offers better short-term benefits. In contrast, scaling Lavender with full finetuning may lead to deeper alignment and knowledge restructuring, potentially resulting in longer-term advantages.
Choice of Layers to Align. We examine the choice of cross-attentin layers to align with Lavender in the Llama 3.2- 11B Vision Instruct model. This model comprises 40 layers in total, 8 of which are cross-attention layers integrated to process visual inputs, constituting one-fifth of the model's 's layers. Figure 23 compares the performance of four different subsets of the 8 cross-attention layers. While attending to the first, mid, or last subset shows shifted strengths on specific benchmarks Real/ooldQA//CRRBench MMMU/POPE, and Hallucination. be the most effective overall. Therefore, this the default strategy adopted in this work.
peeeccively-liiinneee all layers proves to
21

---

Diffusion Instruction Tuning
KT Aigacls
24.2
6/4
14A
305.0
2.
23.9
First 3/8
30.0
45.1
1500.0
39.3
741.0
78.0
52.9
28.0
FT Miigged
73.3
2016
2-22.
34.0
142.0
96.1
3248
27.5
Mud 4/8
51.3
48.4
1506.2
40.7
719.0
87.7
49.2
29.8
-0.8
0.6
563
1:45,
11
735.0
66.2
91.6
26.1
Nomelioel
0.4
14.2
29.2
2515.4
$2.7
135.0
320
62.1
52.1
1428.8
37.3
691.0
85.0
51.4
30.1
XT. 20.00ooeec
74.3
589
15121
91.8
736.0
86.2
929
26.2
12
-0.0
58.7
1680.6
40.0
748.0
87.8
53.2
23.5
-Ea
XT 10
73.8
384
3405.
32.8
733.0
32.8
254
MMB
OCR3osch
POPE
RewlWarMMQ
HelbssseRoshh
DoeVOA
InbVQA
Figure 22. Short pretraining (P.T.) of the Aligner Network Figure 23, Aligning all eight cross-attention layers in Llama-3.2 is mitigates catastrophic forgetting. Results are from Lavender- most effective. Results compare subsets of cross-attention layers aligned Llama 3.2-1IB fine-tuned on Flickr-11 1k for 30 epochs, with vary- with Lavender in Llama 3.2-1IB, LoRA fine-tuned on Flickr- Ik for 30 ing pretraining lengths, evaluated on eight benchmarks using epochs, evaluated on eight benchmarks using 'exact match" without an 'exact match' without LLM judge.
LLM judge.
C. Failure Strategies [Back to Contents]
This section expands on Section 8: Failure Strategies in the main paper. Despite the successful strategies presented earlier, we also tested the following approaches and found them to be less effective: 1) Fully Finetuning Without Pretraining. As shown in Figure 22, fully finetuning without pretraining often destabilises the model, while LoRA proves to be generally robust. We recommend starting with LoRA on small datasets as reliable initial strategy for exploring Lavender, reserving full finetuning for scaling efforts and investigating emerging behaviours after deep alignment. 2) Frequent Switching Between Training Strategies. Given the varied performance of certain Lavender strategies, such as long fully finetuning aligning subsets of cross-attention layers, we experimented with more complex staged training strategies. These involved switching between different strategies across epochs. However, frequent strategy changes harmed performance, likely due to the short overall training length. The model appeared to struggle with prioritizing objectives across strategies before stabilizing, leading to crashes. 3) Mixing Additional Data. In some cases, adding extra data did not improve performance. For example, as shown in Figure 18, mixing the OCRVQA dataset reduced overall performance. This suggests that OCRVQA may lead to overfitting more readily than other datasets tested.
D. Limitation and Future Works [Back to Contents]
This section expands on Section 8: Limitation and Future Works in the main paper.
Limited Compute. Lavender evaluated datasets of up to 0.13M samples, constrained by available compute resources. This is significantly smaller than the 5M to 50M datasets used by state-of-the-art models, which approximately 38x to 384x larger. Figure 18 demonstrates non-convergent scaling behaviour, suggesting that further scaling of both dataset size and tuning length could lead to additional improvements in overall performance with Lavender.
Exploring Biiher--Reolutte Diffusion Models. Lavender is evaluated using Stable Diffusion v1.4 (Rombach et al., 2021) in this work. Advanced models, such as Stable Diffusion v2 (Rombach et al., 2022), could provide higher resolution and more accurate attention maps but would demand greater memory for attention alignment, exceeding the capacity of this study. This approach may become viable with sufficient data scaling and tuning as task complexity increases, marking key direction for future research.
More Accurate Attention Maps Extractions. This work focuses on verifying the feasibility of Lavender rather than the quality of attention maps from Diffusion Models, which already exhibit significantly stronger text-region alignment. However, as noted earlier, short inversion and diffusion steps were applied to prepare per-word attention maps, reducing
22

---

Diffusion Instruction Tuning
processing time 20 seconds per image on a single V100 GPU. While this setup is efficient, it may limit the accuracy of text-to-region correlations in the attention maps, especially for unfamiliar words (Jin et al., 2023). Future work could explore efficient methods to accelerate attention map estimation, enhancing Lavender's overall effectiveness.
Improved Handling of Self-Attention Only Models. This work demonstrated the feasibility of applying Lavender to the self-attention-only -lama-3-based MiniCPM-v-2.5. However, further optimization for such models remains under-explored due to limited capabilities and computational resources. Notably, this includes the advanced parallel attention mechanism, which was proposed and shown to be effective in the Llama-3.2 implementation of Lavender. Future theoretical and empirical research on the relationship and translatability between self-attention and cross-attention mechanisms would be valuable for advancing this line of inquiry and benefiting the broader community.
E. Related Work [Back to Contents]
This section expands on Section 1: Introduction in the main paper.
We find that the gap in VLM alignment partly stems from the technological trajectories pursued over the past half-decade. One key VLM milestone was Flamingo (Alayrac et al., 2022), which laid the foundation for modern VLMs. In its design, images and text are processed by separate encoders, unified through a perceiver resampler (Jaegle et al., 2021), and passed through deep transformer layers combining cross-attention and self-attention. Flamingo's 's elegant architecture established new standard, influencing range of subsequent models (Li et al., 2022; 2023b; You al., 2023). The importance of aligning vision-text correlations is evident in the design of Llama 3.2 (Dubey et al., 2024), released two years later, which adopts Flamingo's 's strategy of using a dedicated cross- coss-attention module for effective interaction handling. However, training a VLM with a dedicated cross-attention module end-to-end requires substantial data and computational resources. These models are typically pre-trained on millions or even billions of image-text pairs and interleaved image-text datasets (Zhu et al., 2024). Similar challenges apply to broader multimodal models beyond vision and language (Lu et al., 2024).
Unlike VLMS, single-modality large language models (LLMs) have scaled more rapidly (Ouyang et al., 2022; Brown et al., 2020; Chowdhery et al., 2023), often consuming over 100 million examples spanning 1,800 tasks (Longpre et al., 2023). VLMs, however, face a training data gap due to the high cost of acquiring paired image-text datasets. To address this, researchers proposed leveraging scaled LLMs by instruction fine-tuning them on as little as 150k paired visual question answering (VQA) data using an autoregressive loss. This approach, pioneered by Zhu et al. (2023); Dai et al. (2023); Liu et al. (2024c), aligns text and image tokens through fine-tuning connectors such as MLPs, encoders, or decoders connecting to the I.M,, providing an efficient pathway to integrate vision with language models for diverse tasks (Wang et al., 2024a; Li et al., 2024; Koh et al., 2024; Chen et al., 2025a; Wang et al., 2024b; Chen et al., 2023; Huang et al., 2024; Liu et al., 2024d; Gao al., 2023; Cha al., 2024).
However, the community soon recognised that the vision capabilities integrated through these small adapter layers outside the LLM (in VL-VA-i approaches) remain insufficient (Tong et al., 2024c). To address this gap, Covert et al. (2024); Karamcheti et al. (2024) refined vision encoders to align more closely with pretrained vision models. Jiang et al. (2023); Kar et al. (2025) proposed merging multiple visual encoders with projection layers before feeding them into the LLM. Separately, Tong al. (2024a); Shi et al. (2024); Zong et al. (2024) explored merging a larger number of diverse vision expert models with fine-tuned projection layers, integrating the combined vision features either before the LLM or within the LLM transformer layers, respectively.
Diffusion Models (DMs), as a key vision expert, have also garnered recent attention. Wang et al. (2024c) leverage DM's image generation loss to enhance the visual encoder. Other approaches focus on equipping VLMs with DM's image generation capabilities, either by using VLM outputs to fine-tune DMs separately (Tong et al., 2024b; Hernandez et al., 2024), sharing a central transformer (Shi et al., 2025; Chen et al., 2025b) or by directly merging DMs with LLM transformers (Zhou et al., 2024).
Despite attempts integrate DMs and VLMs with minimal modifications their internal architectures, one overlooked aspect in prior work is the role of self- and cross-atiention layers within DM and LLM Transformers. These layers govern the interplay between multi-mode modal tokens, yet have not been closely examined. Although DMs and the LLM component in VLMS share the same foundational Transformer architecture ( (Vaswani, 2017: Dosovitskiy, 2020), they exhibit markedly different text-to-region alignment due to their distinct optimization objectives. Notably, DMs demonstrate stronger alignment than VLMs, as shown in Figure 3.
23

---

Diffusion Instruction Tuning
F. Full Pseudo Code of Diffusion Instruction Tuning [Back to Contents]
This section expands on Algorithm 1: Algorithm 1, as introduced in the main paper.
Algorithm 2 Diffusion Instruction Tuning (Full Version)
Require:
Dataset D= = {(x((() y Btt
(6)
) where ya is textual input (e.g. a question) and v) is the target textual
output. Pretrained DM parameters pp. pretrained VLM parameters 6, Scaling factor À > 0, learning rate 1).
Ensure: Fine-tumed VLM parameters
Stage I: Preprocessing (Run Once)
# the DM processes image-question pairs, hence replacing Dy with y
2: For each data point (x y in D:
3: Use the pretrained DM (0D fixed) to compute Pa xx(1), (())))()))))))
4: Store these attention distributions: A -- prm((a 2x(0), , 0p)
5: After this, we have a DM-derived attention target for each data point, which will remain fixed during fine-tuning.
6: Stage Fine- Fining the VLM
7: Initialise 0 from a pretrained VLM.
8: Set learning rate 7) and define maximum training steps or convergence criteria.
9: repeat
10: Sample mini-batch B S D of size m
11: (O) + O. (0)( <-
for (xl1), (10),y(0), Me 111 in do
()) (1)
13:
Compute PVLM(a (()),y()) by aggregating attention heads/layers of the VLM cosss//ellfattenniinn (see Section 3)
14:
(0))0)) + Aligner(pvIM(C 2x11) 0))
-
15:
Compute task loss: (0) += - log log (i)
xاا) 2/4 (3)
0)
16:
Compute attention alignment loss: Lan(O) += ( (0)
17:
end for
18:
Form total loss: Looia (0) = (0) + ALuu (l)
19:
Update parameters: 0 +- v Lnoral (0)
Optionally, apply LoRA other optimization tricks if desired.
21:
until convergence (e.g.. validation metric plateaus
steps reached)
22:
Output: The fine-tuned VLM parameters A.
G. Bayesian Justification for DM Attention Proximity [Back to Contents]
This section expands on Section 2.3: Bayesian Derivation, as discussed in the main paper. In this section, we provide a detailed Bayesian justification for our assumption that the DM's attention distribution PDM(a x, y; 0p) is closer to the optimal posterior distribution of vision-centric word-to-region attention p" (a y) than the VLM's attention distribution (a xr, y; 0).
G.1. Modelling Attention Distributions as Posteriors
We model the attention distributions in both the DM and the VLM as posterior probabilities over the attention a given the inputs and model parameters. For consistency, we consider the joint distributions and apply Bayes' theorem.
Diffusson Model (DM): The DM is trained to generate an image conditioned on text y by modelling the distribution (> y; Op). We can express the attention mechanism in the DM as contributing to this distribution via:
/ pom(( y, a; 0p) pam(a yı Qn) da.
(DM(# yy 0p)
(17)
Applying Bayes' theorem, the posterior over attention given x and y is:
24

---

Diffusion Instruction Tuning
PDM(a æ, 0;0) PDM( y. a; 8p) PDM y: 0p) PDM(2 yı DD)
(18)
Vision-Languaae Model (VLM): Similarly, for the VLM, which generates textual output yt given an image xr and textual input V/. the attention mechanism influences the distribution (VLM(1) x Mgi 0):
PVLM(U I. Vz; 0) PVLM(( x. Vg. 0) PVLM(a Ugi 0) da.
(19)
The posterior over attention a then:
PV Iz My. ve 0) PVLM(UU Ug. 0; 0) PVLM 2, My; 0)
(20)
PVLM x, Mgi 0)
G.2. Differences in Likelihood Functions
The key distinction arises from the likelihood functions:
DM Likelihood pDm( a, On): The DM reconstruct the image xr accurately, which requires precise alignment between textual tokens and visual features. The attention plays critical role in ensuring that each part of the text y correctly influences the corresponding visual content in æ.
VLM Likelihood (VL x, My: a; 0): The VLM generates y based on the image æ and input text Mn- While attention aids in focusing relevant visual regions, the text generation process often rely higher-level visual features and may not require as fine-grained vision-text alignment as the DM.
G.3. Entropy and Concentration of Attention Distributions
Due to the DM's need for precise image reconstruction, its attention distribution PDM(a x, yy 0p) expected to be more concentrated around p" (a xz y). This can be quantified by the entropy H of the attention distributions:
H (a z. 0p)) < (PVEM(a x, Myy 0))
(21)
lower entropy indicates that the DM's attention distribution more peaked and thus closer to p"(a v), whereas the VLM's higher entropy reflects more diffuse attention distribution. This is consistent with our empirical observations in Figure 3 and section 6.1.
G.4. KL Divergence to the Ideal Attention in Vision-Centric Tasks
We can formalise the proximity to the optimal posterior attention distribution using the Kullback-Leibler (KL) divergence. For the DM conditioned on a unified text y, modeling p(xly; P)), the KL divergence the pz (a x,, y) is:
DKL (PDM(a) x, 0p) || (" x2,)) = / pDMa(a x, y1 0p) log
PDM x, (5 da, p² (a
(22)
2r, v)
and similarly, for the VLM processes image æ, question Vo, and answer label y. modeling p(m |z, yoi 0):
pa Jr, Ugi 0) p" (a x, y)
DEL (aviM(a x, 0y: 0) 1 xx,))) PVLM((a ax Mgy 0) log
da.
(23)
Bringing in the entropy H of the attention distributions, equations (22) and (23) can be written as:
25

---

Diffusion Instruction Tuning
/
DKL (a x, BD) (a x. x,))) - H (pDm(( 2, (.
Pomía x. v: 0,); logp" (a x. y) da,
(24)
and similarly for the VLM:
DEL (PVLM(a x, 2; ll pa z.1)) -H (PVLM(a x, yz,) - PVLM(a y; 0) log | x, da.
(25)
Our empirical observation of lower entropy in equation (21) indicates that the DM's attention distribution is more centrated and thus more certain about where to attend, which is consequence of the DM's need for precise vision-text alignment during image reconstruction.
Assuming that the cross-entropy terms J pDM(( æ, y; 0p) logp" (a x; y) da and J PVLM(a x, y; 0) log (a | x, y) da are approximately equal' or that the difference entropies dominates the difference in cross-entropies, we can infer:
DRL (pDM(a I, y; Op) Il 1 (a x, v)) DRL. (pVIm(a x, 0, I| (a x,,)).
(26)
This inequality suggests that the DM's attention distribution is closer to the optimal posterior attention distribution in terms of KL divergence. The lower entropy of the DM's attention implies is more peaked around the ideal attention in vision-centric tasks. Therefore, our empirical observation of the DM's lower attention entropy supports the assumption that the DM's attention distribution is closer to the optimal posterior attention distribution than the VLM's. By aligning the VLM's attention with the DM's attention, we aim to reduce the VLM's KL divergence the ideal attention in vision-centric tasks, enhancing its vision-text alignment and overall performance.
H. Justiffccatiin of the Attention Alignment Loss [Back to Contents]
This section expands on Equation (10), as introduced in the main paper, to provide a detailed justification for the inclusion of the attention alignment loss Lan(( (0) and the scaling factor À in our Bayesian framework.
H.1. Derivation of the Likelihood Term
We start by modelling the difference between the VLM's attention distribution and the DM's attention distribution as random variable. For each data point i, we define:
1 (0)
PVLM(a 100) 00 -PDM(a ((0),())p))))
(27)
We assume that (0) follows multivariate normal distribution with zero mean and covariance matrix o²1, where I is the identity matrix:
1 (0) ~ o2
(28)
Under this assumption, the probability density function for ((0 is:
p ()))
1
1 20
g00 (0)
22m02)k/2 exp
(29)
where & is the dimensionality of the attention distribution.
The likelihood of observing the DM's attention given the VLM's parameters over the entire dataset is then:
p(ApM 0) II (8) ((endd))))))
1 N
(- 1 Σ (8) (0))))
exp
(30)
202
Sinne p" xr, is the for both models, and both pDM and PVLM centered around p² (a v) given they are from pretrained models based on big dataset, the values these cross-entropy terms would not differ significantly.
26

---

Diffusion Instruction Tuning
where N is the number of data points.
Taking the negative log-lik ligllikeihhodd we get:
Nk log p(ADM 0) (ogo2 Σ
(0)
(31)
2
20²
Ignoring constants that do not depend 0, we have:
logp(AAM 0) = 20 Σ
(100 + const.
(32)
Defining L. oT, we arrive at:
Σ log ((AM 10) = 2 [0)
const.
(33)
H.2. Interpretation of the Scaling Factor
The scaling factor plays a crucial role balancing the attention alignment loss with the primary task loss. is inversely proportional the variance 2² of the assumed Gaussian distribution of the attention differences.
A small o² (large X) implies high confidence in the DM's attention distributions, placing more emphasis aligning the VLM's attention with that of the DM.
A large o² (small X) implies less confidence in the DM's attention distributions, reducing the influence of the attention alignment loss.
In practice, can be treated as a hyperparameter tuned based on validation performance.
H.3. Total Negative Logg-Psteerir
The total negative log-posterior combines the negative log-likelihoods of the data and the attention alignment:
Liotal (f) = - log po (D D 0) log p(AM () + const.
(34)
Substituting the expressions for the negative log-likelihoods, we have:
Tuuual (0) IVLM (6) + AT) const,
(35)
where:
LVLM(0) = Σ
log p(v) (8)
10 ;);
(36)
(0n(()
Σ
(10) (o))²
(37)
2
By minimizing (0), we maximise the posterior probability p(0 D, Apw), effectively incorporating both the data likelihood and the prior information provided by the DM's attention distributions.
H.4. Justification for Using MSE Loss
The mean squared error (MSE) loss used in (0) arises naturally from the assumption of differences. This is a common assumption in Bayesian modelling, where the Gaussian distribution often used due to its mathematical convenience and the central limit theorem. The MSE loss is also computationally efficient and widely used in neural network training, making a practical choice for aligning attention distributions.
attention
27

---

Diffusion Instruction Tuning
I. Additional Details on Attention Flow in Vision-Language Models [Back to Contents]
This section expands on Section 3.2.2: Attention Flow, as introduced the main paper. In addition to simple aggregation methods, we explore attention flow (Abnar & Zuidema, 2020) to aggregate attention maps across layers in VLMs. Attention flow computes the effective attention between input and output tokens by considering the cumulative effect of attention across layers, capturing deeper interactions that span multiple layers. This method has been utilised by Lin et al. (2024) to obtain sentence-level aggregated attention maps for grounded segmentation tasks. We investigate its applicability in aggregating word-level attention maps in VLMs, aiming to capture semantic correlations that may not be evident through simple aggregation methods.
Let A E IR TR-NNaxNpprt denote the attention matrix at layer 1, where Nie is the number of text tokens and Npath the number of image patches. Our goal is to compute an aggregated attention map A RR Nen x Mpwch that captures the overall attention from text tokens image patches across all layers.
We initialise the aggregated attention map (mean max) with the attention from the first layer:
Ā A
(38)
We then recursively update À by combining it with the attention matrices from subsequent layers. Specifically, for each layer/ 2, ...., L. we update A using either element-wise multiplication:
Ā = A0)
(39)
or element-wise addition:
A=
(40)
where denotes element-wise multiplication. We explore both strategies multipicaiee and additive aggregation to assess which better captures the semantic correlations.
However, directly applying attention flow in autoregressive VLMs can lead to attention collapse due to the causal masks used during training. To mitigate this issue, we introduce a regularisation term that adjusts the contribution of each text token. Specifically, we define a regularisation vector E R Noar with elements:
t- Nixt-
(41)
Niex
This term assigns lower weights to earlier tokens and higher weights to later tokens, preventing the dominance of early tokens in the aggregated attention map.
We apply the regularisation to the aggregated attention map:
Ar,p Ar,p x Vt.
(42)
where At,p represents the attention from text token T, image patch Tp. By incorporating this regularisation, we ensure that the attention flow effectively captures semantic correlations without collapsing due to the model's autoregressive nature.
Through attention flow with regularisation, we aggregate attention across layers to obtain per-word attention maps that better reflect the semantic relationships between text tokens and image patches. This method captures deeper interactions that may not be evident through simple aggregation, enhancing the alignment between VLMs and DMs.
J. Appropriate Rearrangement and Reconstruction Matter [Back Contents]
This section extends the prior discussion in Section 3.2 around rearrangement and reconstruction during aggregation. Many VLM preprocessing pipelines split an image into tiles before the projection process. In such cases, it is crucial to account for both the original tiling and the resulting patch order. Direct reshaping of the flattened tokens, without reassembling the original tile layout, disregards spatial continuity between adjacent tiles, potentially disrupting semantic alignment.
Figure 24 illustrates the tiling and tokenization procedure in Llama-3.2 3.2 and highlights the importance of proper reconstruction. Similarly, Figure 25 demonstrates the impact of appropriate reconstruction on real samples from the OCRVQA dataset. Improper rearrangement leads to misaligned attention maps, while correct reconstruction restores semantic coherence, enhancing the quality of -ssuulltettul alignment.
28

---

Diffusion Instruction Tuning
1
2
3
4
5
8
9
10
11
12
13
15
16
image split into 4 tiles each tile has patches
flattened patch tokens
dircet rearrange ID
tokens into 2D
1
5
6
13
14
9
10
3
4
7
8
5
6
2
9
10
13
14
8
11
12
15
16
first rearrange cach tile of ID tokens into 2D
then rearrange the sequence of tiles into 2D
reconstruction resemble
spatial arrangement
Figure Attention reconstruction under tiling and tokenization procedure in Llama-3.2, Example highlighting the importance of proper reconstruction. Improper rearrangement disrupts spatial continuity, while correct reconstruction preserves semantic alignment.
Irputing
10%
3Dvuter
ARl pretok
HONES
Iepitine
pro
Inaiiing
60
#moogg
50
#5ipng boo
50 yoow
Step wnaadd
50-lap
Sohuk
Irpeiing
20Capri
JBn pr04
JBn such
Before Debugging
(b) After Debugging
Figure 25. Appropriate rearrangement and reconstruction crucial. Results are based attention maps extracted from Llama-3.2 (without Lavender fine-tuning) on OCRVQA samples. Poor rearrangement disrupts semantic alignment, while proper reconstruction corrects the spatial arrangement of the attention maps.
29

---

Diffusson Instruction Tuning
K. Additional Details on Attention Alignment: Integrating Lavender [Back Contents]
This section expands on Section 3.4: Lavender Integration, as introduced in the main paper.
K.1. Incorporating Lavender into VLMs with Cross-Attention [Back to Contents]
When the VLM employs cross-attention layers to integrate visual and textual modalities, extracting and aggregating the attention weights is relatively straightforward. As described earlier, each cross-attention layer produces attention weights "(t.p) that directly relate each text token Tú to set of image patches Tp. Since the attention flow is unidirectional text queries attending to image keys-these weights can be naturally interpreted as word-to-patch correlations.
Practically, for each layer and head, we retrieve the cross-attention weights and reshape them into a grid representing the approximate spatial layout of the image patches. This involves: 1) interpolating the attention weights to form a roughly square matrix, 2) arranging the tiles (if the image is represented multiple tiles) into a coherent spatial layout, see Figure 24 consistent resolution (e.g. 32 x 32) for downstream processing. By aggregating
in Appendix for details, and 3) resizing across heads and layers and compressing through the Aligner network, we produce final per-word saliency map that can be aligned with the DM's attention distribution.
K.2. Incorporating Lavender into VLMS with Self-Attention Only [Back Contents]
In VLMs that rely solely on self-attention (i.e., where both image and text tokens are fed into the same transformer layers without explicit cross-attention blocks), both modalities (text and image patches) are interlcaved within single sequence, and self-attention allows each token to attend to every other token, often with causal or bidirectional mask.
To obtain word-to-patch correlations, we must: 1) Identify which subset of tokens corresponds to text and which correspond to image patches. 2) Apply a causal or bidirectional mask correctly to avoid including attention values that do not reflect the desired semantic associations. 3) Separate and rearrange the attention weights to isolate the correlations between text tokens and the patch tokens representing the image.
Furthermore, since the image patches and text tokens may not be explicitly organised in grid-like structure, we must carefully reshape and interpolate the extracted attention weights. The process involves: 1) selecting the appropriate text and vision token indices from the attention weights, 2) interpolating the resulting attention maps into square grid, 3) resizing them to fixed resolution (e.g., 32 x 32), and 4) optionally incorporating the Aligner network output As or merged attention maps from earlier steps. This process is demonstrated in Figure 6.
Special handling of causal masks and careful indexing ensures that we only extract attention weights that correspond to meaningful word-to-patch relationships, and we show its necessity in Appendix Appendix J. Although this introduces additional complexity compared to the cross-attention scenario, the same principles of reshaping, interpolating, and aggregating attention weights apply. By following this procedure, we can still derive meaningful per-word attention map aligned with the DM attention, enabling Lavender to improve vision-text alignment even in models that rely exclusively on self-attention.
L. Full Implementation Details [Back to Contents]
This section expands on Section 4: Implementation (Short Summary), as introduced in the main paper. We detail our implementation of Lavender. We first describe our process for extracting and aggregating attention distributions from the Diffusion Model (DM) using Stable Diffusion v1.4. We then present how we integrate Lavender into three Vision- Language Models (VLMs): OpenFlamingo, MiniCPM-Lland 3-v2.5, and Llama 3.2- IB-Vision Instruct. Among these, OpenFlamingo and Llama 3.2- 1--1IB--iise Vision Instruct rely on cross-attention layers, while MiniCPM-Liaan 3-v2.5 exclusively uses self-attention. In all cases, integrate the Aligner network and attention alignment loss as described previously.
Stable Diffusion v1.4 We use the official Stable Diffusion v1.4 model (Rombach et al., 2021) to extract per-word attention maps from the diffuuion model. We apply a shortened image inversion process (Mokady et al., 2022; Jin et al., 2023) to approximate the text prompt embeddings for image reconstruction, collecting attention maps at each step as in Section 3.1. We limit the inversion steps to 5 and diffusion steps to 10 for efficiency, enabling us to process each image in roughly 20 seconds on a single V100 GPU.
30

---

Diffusion Instruction Tuning
OpenFiamingo OpenFlamingo (Awadalla al., 2023) is open-source family of autoregressive vision-language models designed to replicate Flamingo-like performance (Alayrac al., 2022). We integrate Lavender by adding a wrapper around its cross-attention layers to extract and aggregate per-word attention maps. The aggregated VLM attention is then aligned with the DM attention distributions, as described in Appendix K.I. We maintain the standard training procedure, adjusting hyperparameters such as the learning rate and the frequency of attention extraction, while adding the Lavender alignment loss.
MiniCPM-Llama 3-v2.5 MiniCPM-LJama 3-v2.5 (Yao et al., 2024) differs from the previous models as relies solely on self-attention. We carefully identify and extract text-image correlations from the self-attention layers, as described in Appendix K.2. The integration of Lavender involves wrapping the self-attention blocks to isolate and reshape patch-token attention weights into a coherent spatial layout. We use similar hyperparameters for alignment, ensuring that even without explicit cross-attention, the model can benefit from DM-guided attention alignment.
Llama 3.2-11B- Visiin Instruct The Llama 3.2-1 1--1IB--iiion Instruct model is large-scale VLM that incorporates image information via cross-attention layers. Similar to the OpenFlamingo integration, we add a wrapper to capture the per- per-word attention distributions, then align them with DM-derived attention maps. Adjusting hyperparameters such as batch size, learning rate, and the interval which we extract attention maps ensures stable training and improved vision-text alignment.
Computing Environment All experiments are conducted on NVIDIA GPUs (V100, A A10G, A100), using PyTorch as deep learning framework. For large-scale training and efficient memory usage, employ Deepspeed in the MiniCPM- Llama 3-v2.5 experiments and Fully Sharded Data Parallel (FSDP) for the Llama 3.2- 1 BB-viion Instruct model. Both techniques allow us to handle extensive model parameters and large batch sizes with reduced memory overhead, improving training stability and runtime efficiency. Each model and dataset combination is trained following its recommended best practices, with minimal additions of Lavender-specifie parameters and code.
M. Full Training Recipes and Data Preparations [Back to Contents]
This section expands on Section 5: Training and Dataset (Short Summary), as introduced in the main paper. Supervised fine-tuning pretrained and partially aligned VLM with additional objectives on small dataset can lead catastrophic forgetting of previously acquired capabilities. To mitigate this, we introduce training strategies that balance the new alignment objectives with the preservation of existing model knowledge.
Pretraining the Aligner Network. Before jointly updating all parameters, we optionally pretrain only the Aligner network (and, if desired, certain bridging layers between the vision and language components) while keeping the VLM parameters frozen. This pretraining step allows the Aligner network to adapt to the DM's attention signals independently, ensuring that subsequent joint training steps do not immediately overwrite the VLM''s learned representations. By carefully scaling the learning rate during this phase, we can achieve stable initialization that speeds up convergence without destabilizing the pretrained model.
Attention Aggregation and Normalization Choices. We experiment with different attention aggregation strategies described in Section 3. We also apply instance or batch normalization within the Aligner network to control variance and stabilise training, making the final attention distributions more interpretable and consistent.
Configuring the Aligner Network. The Aligner network itself is a configurable module inspired by Squeeze-and- Excitation concepts, composed of convolutional layers that project and refine attention distributions. Depending on complexity requirements, we can choose a lighter or deeper configuration. A "Tight" configuration applies a single round of convolution and normalization, "sim" applies two round, and "deep" applies four rounds expansion and squeeze operations for greater representational power. These options let us tailor the computational complexity and modelling capacity of the Aligner to the specifics of the task and dataset.
Short Training Schedules and PEFT. We limit training to fraction of an epoch to minimise overfitting and catastrophic forgetting. Additionally, we incorporate Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, to constrain the number of parameters being updated. This helps the model retain its core pretrained knowledge while focusing the updates on a smaller, more controlled parameter subset- toasettprimarijy those associated with the Aligner network and attention alignment- thus improving stability and preserving performance on previously learned tasks.
31

---

Diffusion Instruction Tuning
Overall, this combination of pretraining the Aligner network, flexible attention aggregation and normalization strategies, adjustable Aligner configurations, and selective parameter updates through PEFT methods ensures more stable and effective training process. The result is improved attention alignment without eroding the VLM's hard-earned pretrained capabilities.
Dataset Preparation. We conduct our experiments using four datasets, each processed through the diffusion model to obtain per-word attention maps. These maps are used to fine-tune the VLM based on the text labels (either captions or multi-round question-ans -answernng) associated with each image. First, the Flk30k dataset (based on Flickr30k (Young et al., 2014)) comprises approximately 31,783 images and 158,915 captions, focusing on people in everyday activities and events. This rich and diverse caption corpus is used to construct denotation graph, enabling the extraction of fine-grained semantic relationships. Similarly, process a 50k subset of the Laion-5B (Schuhmann al., 2022), dataset containing 5.85 billion CLIP-filtered image-text pairs, referred to as Laion50k, to further verify our method. Next, the RLAIF-V 83k dataset (Yu et al., 2024), a large-scale multimodal feedback dataset, provides 83,132 preference pairs drawn from a variety of sources including MSCOCO, ShareGPT-4V, MovieNet, and VQA variants. This diversity ensures comprehensive coverage of vision-language tasks. Finally, OCRVQA30k is a 30,000 subset of OCR-VQU VQA dataset, which contains a total of 207,572 images along with their associated question-ane answerr pairs. Together, these datasets enable a broad evaluation and refinement of the model's attention alignment capabilities. Among the four processed datasets, the Laion50k is used only for method verification with the OpenFlamingo implementation of Lavender, while the others scaled across all models.
Sampling Strategies. During fine-tuning, the VLM predicts text for each image and question. We define two sampling strategies to determine which words from the predicted text are eligible for computing the MSE loss: 'root word match' and 'exact word match', which are post-processing steps on fully generated and decoded answers prior to loss computation and backpropagation. The 'root word match' strategy relaxes the condition by allowing matches based on the root form of words, accommodating scenarios where the VLM's text generation capability is limited. In contrast, the 'exact word match' strategy stricter, considering match only when the predicted word exactly matches a word from the label text.
N. Empirical Verifications (Detailed) [Back to Contents]
This section expands on Section 6.1: Empirical Verifications, as introduced in the main paper.
N.1. Additional Proof of Concept Results
We begin by presenting additional detailed results extending the empirical verification experiments in Section 6.1, where tested our key hypothesis: the cross-attention from DM transformers closely approximates an ideal attention mechanism for maximising VLM performance, as discussed in Section 2.2.
Attention Entropy Histograms. We compute both Anm and Av for each sample in a small subset of Flickr30k, RLAIF- and OCRVQA30K, totaling approximately 10k samples, across three models (OpenFlamingo, MiniCPM-v2.55 and Llama 3.)- The entropy histograms of both attention maps are plotted in Figure 7. Figure 26 shows separate entropy histograms of the attention maps ADM and Ay for three models: OpenFlamingo. MiniCPM-v2.55, and Llama 3.2-11B. These results extend the combined plot presented in Figure 7. We observe that the DM's attention distribution, PDM (a ar, y; 0D), consistently exhibits lower entropy compared to the VLM's attention distribution, PVLM (a/æ, y; 0) across all three models. This finding reinforces our hypothesis that the DM's attention is more concentrated and thus closer to the optimal posterior attention distribution, p" (a|r, y).
Next, we verify that our proposed Lavender tie-tuning approach can align the VLM's attention with the DM's attention as guidance. We fine-tune the Lavender- OpenFlamingo implementation on the Flk30k datasets. Figure illustrates the VLM cross-attention maps over multiple training steps, using DM attention maps as a reference. The visual results demonstrate that the following strategies enable successful convergence from the raw, 'diffused' VLM per-word attention to a pattern similar to the semantically meaningful DM per-word attention: 1) Employing "exact word match' instead of 'root word match' as the sampling screening strategy. 2) Using convolutional layers instead of an MI. setup for the Aligner network. 3) Setting the learning rate to le-4 instead of 1e-5.
To verify Lavender's 's argument that jointly minimizing (f) and (0) can improve VLM performance, we fine-tune OpenFlamingo using both the autoregressive approach (fv)M(0) only) and the Lavender approach on the Flk30k and
32

---

Diffusion Instruction Tuning
Atarriu
(
Attestion Branys Hisig DM VLM (M
Attarion Estrgy Hittgran vs VLM (1 3.2111))
13
VLM cranies trea 122 48)
VLM (nca 1.87- 620) US5 elcetiinn +3.4
VLM stumiins taces (44)
DM simiie yaom 641 (039)
UP elensia (aean 662
25.0
2502
7.5
5.0
10
503
5.3
5.5
Enoopy
Eeargp
Figure 26. Attention map entropy histograms from three models (OpenFlamingoo, MiniCPM-v2.5, and Llama 3.2-1IB) are generated by processing small subset of Flickr30k, RLAIF-V83K, and OCRVQA30K, totalling approximately 10k samples.
Laion50k datasets separately. In Figure 9, we present the calibration plot between the text generation score and the MSE score. The results show that, compared to autoregressive fine-tuning, Lavender improves text generation quality by jointly minimizing the MSE loss.
Lastly, we evaluate the autoregressive and Lavender fine-tuned OpenFlamingo models trained on both Laion50k and Flickr30k across seven languaee benchmarks. These include two captioning benchmarks (COCO (Chen et al., 2015) and Flickr30K (Young et al., 2014)), four VQA benchmarks (VQAv2 (Antol et al., 2015), OK-VQA (Marino et al., 2019), TextVQA (Singh et al., 2019), and VizWiz (Gurari et al., 2018)), and one rank classification benchmark (Hatefullemes (Kiela al., 2020)). The evaluation metrics reflect text generation quality and adhere to the default settings of the respective benchmarks. Results in Figure 10 confirm that the proposed Lavender approach, by jointly minimizing (f) and (f), can improve VLM performance by up to 72% compared to autoregressive fine-tuning on the OpenFlamingo model. This improvement is observed with both the Laion50k and Flickr30k datasets, with the latter yielding slightly better results.
Visual Confirmation of Attention Alignment. Figure 27 provides additional visual evidence verifying that the proposed Lavender fine-tuning approach aligns VLM attention with DM attention as guidance, extending the results from Figure 8. These results reaffirm our findings from the main body: VLM cross-attention maps successfully align with semantically meaningful DM attention patterns by leveraging strategies such as 'exact word match' for sampling, convolutional layers in the Aligner network, and learning rate of 1e-4.
O. Evaluation and Baselines Settings of Scaled Experiments [Back to Contents]
This section expands on Section 6.2: Evaluation and Baselines, as introduced in the main paper.
Evaluation on Multimodal Benchmarks. We evaluate Lavender on 20 VLM benchmarks to demonstrate its capabilities across diverse perspectives, grouped as follows: 1) Chart, Diagram, and Document Understanding. For structured OCR data, we evaluate Lavender on benchmarks such as A12D (Kembhavi al., 2016), ChartQA (Masry et al., 2022), OCRBench (Liu et al., 2023c), OCRVQA (Mishra et al., 2019), TextVQA (Singh al., 2019), DocVQA (Mathew al., 2021), and InfoVQA (Mathew et al., 2022); 2) Perception and Multi-Discipling Reasoning. To address complex reasoning tasks, including visual perception, recognition, knowledge, and OCR, we evaluate Lavender on perception benchmarks such as MME (Fu et al., 2023), MMBench (Liu et al., 2023b), ScienceQA (Saikh et al.. 2022), MMStar (Chen et al., 2024a), and MMMU (Yue et al., 2024); 3) Real-World Visual Understanding. To validate Lavender's performance in real-world scenarios, we use widely adopted benchmarks, including RealworldQA (x.ai) and SEED (Li et al., 2023a), which assess reasoning, recognition, knowledge, and OCR capabilities; 4) Hallucination. We evaluate visual hallucinations using HallueinationBenchh (Liu et al., 2023a) and POPE (Li et al., 2023c); The evaluation metrics in this paper adhere to the default settings of the respective benchmarks.
Baseline Models. In addition to MiniCPMv2.5 (Yao et al., 2024) and Llama 3.2-11B (Dubey et al., 2024) and their fine-tuned variants as baselines, we include the following groups of VLMs and their performance on the above benchmarks as reference: 1) Small Budget- Budggt-Constrainel Models. This group consists of open-source VLMs with sizes smaller than 10B, using backbones from Vicuna-7B (Zheng et al., 2023), Qwen-7B (Bai et al., 2023), or Llama3-8B (Dubey et al., 2024). Models include: LLLVA-115575 (Liu et al., 2024a), LLLaVA-Nex-1- (Liu et al., 2024b), Mini-Gemini-7B (Team,
33

---

Diffusion Instruction Tuning
Irput Imagee
SD Xatta (man)
VLM Xain (step 100)
(step 500)
VLM Xattn (step 800)
VLM Xatin (step (400) (step 1800) Xanin (step 2200)
Lavender "rod matel MI P- 18 1e-5
Lavender "identical match" LR le:5
Lavender "ilmtical match
Cunv
Te-5
Lavender 'identical match" Cony 1e-4
Toput Image
Xattn (arms)
VI.M Xattn 100)
VI. Xattn (step SO0)
VI MI Xatin 800)
VLM Xotin 1400) VI.M Xatin (step 1800) VL. Xartn 2300)
Lavenkr "identical match" MLP LR le-S
Lavender "ilentical
Lavender
identical masch" Comy 1e-4
Figure 27. More visual verification of learning VLM attention aggregation compared to SD attention for the matched word 'man' and "arms', based on the OpenFlamingo implementation of method. The first shows the plain version of our method, and in each subsequent row, we add one of the training techniques we found useful, which are highlighted in hold.
34

---

Diffusion Instruction Tuning
2024), Bagle-X5-8B (Shi et al., 2024), and Cambrian-1-SB (Tong al., 2024a). Small Data-Heavy SOTA Models. This group includes open-source VLMS with sizes smaller than 20B, typically fine-tuned datasets containing more than (Li et al., 2024), InterVV22-88 (Chen et al., 2024b), Qwen2-VL-7B (Wang et al., 2024a), MoBmo-7B (Deitke et al., 2024), and Pixtral-12B (Agrawal et al., 2024). 3) Large SOTA Models. This group comprises large VL.Ms with sizes greater than 20B, including both open-source and closed-source models, lypically fine-tuned on extensive datasets (minimum 5M samples). Models include: Cambrian-1 (Tong et al., 2024a), LLaVA--OeVisioo--777
5M samples. Models in this category are: ALaVA-OneVision--
(Li et al., 2024), Dwen2-VL-722) (Wang et al., 2024a). Molmo-72B (Deitke et al., 2024), Claude-3 Haiku (Anthropic, 2024), Claude-3.5 Sonnet (Anthropic, 2024). GPT-4V (OpenAI, 2023), GPT-4o (OpenAI, 2024), Gemini 1.5 Pro (Team, 2024), and Llama-3.2-90B (Li et al., 2024).
P. Extended Supplementary Results [Back Contents]
Due to the large variety of datasets and experiments considered in this work, the main body focuses on summarizing and analyzing overall results. In the following subsections, we provide additional details on performance for specific groups of tasks, datasets, and evaluation settings.
P.1. Supplementary Per-Benchmark Scaling Results [Back to Contents]
This section presents the detailed scaling results for each of the eight evaluated benchmarks, extending the average results discussed in Appendix A.2. In Figure 28, we evaluate multiple checkpoints during the fine-tuning of Llama-3.2-11B on four combinations RV83k, Flk30k, and OV30k datasets, using either autoregressive or Lavender methods, combined with LoRA or full fine-tuning strategies, across eight benchmarks. As presented the main body, Lavender scales efficiently with more data, reducing overfitting and performance variation while addressing challenges common in autoregressive fine-tuning on small datasets. The plots in Figure 28 reveal the following key observations:
The Scaling Behaviour of Lavender Across Tasks and Benchmarks.
DocV MME, and POPE: These benchmarks show consistent performance gains as data increases, exhibiting smooth scalability. Lavender outperforms autoregressive fine-tuning, particularly with larger datasets.
HallucinationBench and RealWorldQA: Performance improvement scales steadily with Lavender, but larger datasets are required showease noticeable advantages compared to autoregressive methods. These tasks benefit from Lavender's attention alignment.
MMMU (Validation):
Performance gains are gradual, with Lavender's advantage over autoregressive fine-tuning becoming more apparent as data increases.
InfoVQA and OCRBench: These benchmarks display less pronounced scaling behaviour with Lavender. Overfitting tendencies are more evident when datasets are mixed (e.g., adding OCRVQA, as noted in prior discussions).
General Trends
Overfitting Miligation: Lavender demonstrates strong capability in reducing overfitting, especially on benchmarks like OCRBench.
Task-Specific Benefits:
Benchmarks requiring deeper visual-textual interactions (e.g., HallucinationBennch, Real- WorldQA) see stronger scaling benefits with Lavender.
Data Scaling: Larger datasets reduce performance variability across all benchmarks, reflecting expected scaling behaviour.
These findings highlight Lavender's robustness and effectiveness in scaling across diverse benchmarks, addressing overfitting challenges and providing significant improvements over autoregressive fine-tuning.
P.2. Supplementary Ber-Benchmark Ablation Results [Back to Contents]
Figure 29 expands upon the average results presented in Figure 21, which assess the scalability of the proposed aggregation functions across eight benchmarks at different training lengths.
35

---

Diffusion Instruction Tuning
DacVQA VAL
MMB
Soores
Searce
3730
75
1830
5020
12000
15000
20003
23200
30020
9200
10003
29200
20002
23000
20020
Data Visit Count
Dan Visit Count
HallusionnBench (overall)
POPE
24
24
Scone
22
2.2
1.5
33000
15020
30020
10000
23000
MMMU (val)
InfoVQA VAL
30
00.0
Scotee
55.0
30.5
32
5000
12000
15020
20003
23200
30022
5000
18003
25200
20002
25000
20020
Data Vieit Count
Data Visit Count
OCRBench
Real WorldQA
Scurck
732
46
11
42
5020
12000
15000
20003
25:00
20030
5000
10000
15000
20005
25000
20000
Data Visi Count
Data Visit Count
London''s 2832L FES
AVARL
Londle F7 835 AIG-OVVA
Figure 28. Scaling Behaviour Across Eight Benchmarks. Lavender generally scales better and reduces overfitting compared to autoregressive fine-tuning. Larger mixed datasets further reduce overfitting and variation. Results are based on LoRA fine-tuning Llama 3.2- 11B with both autoregressive and Lavender approaches, and evaluated using 'exact match' without LLM judge. The averaged results are shown Figure 18.
36

---

Diffusion Instruction Tuning
Doc VA.
IaSAN04 YAE
Dav-tall
Con 1444
19903
12.0
15903
20.0
31.3
$6.0
1990.
200
15.0
16.0
55.
14933
14.0
1490
13.0
330
leen
Spucki OC30bad
Tpedk
Epedk
FOTE:
Ral
746.0
$7.
260
720.0
579
Lyp
25.0
18
Lquacasa
byc-eas
210.0
HB
luat:
66:0
55.3
laace
12.0
700.0
200
190
00.0
Figure Impact aggregation functions on tuning iterations (Flickr-1k subset) across eight benchmarks. results are derived from Lavender-Llane Llama 3.2-1IB, fully fine-tuned on the Flickr- Ik subset and evaluated using 'exact match' without an LLM judge. Ludge. The averaged result demonstrated main text Figure 21.
Key Observations on Scalability:
DocVQA and MME: These benchmarks exhibit smooth and consistent performance gains with increasing epochs across most aggregation functions. The 'learn' aggregation achieves the highest scores at epoch 20, reflecting its scalability advantage.
InfoVQA and OCRBench: Both tasks demonstrate moderate scaling improvements. However, 'layer-mean' and 'layer- max' show competitive performance in earlier epochs, while 'learn' aggregation outperforms training progresses, aligning with its ability to generalise better over time.
RealWorld@A and POPE: These benchmarks show slower but steady performance improvements. The 'flow-sum' strategy performs strongly initially but plateaus, while 'learn' aggregation steadily surpasses others in later epochs.
MMMU and HallucinationBench:: Tasks requiring complex reasoning and deeper visual-textual alignment benefit significantly from the 'learn' aggregation, particularly after 15 epochs. In earlier stages, 'flow-sum' shows strong less sustained performance.
General Trends: I) The 'leamn aggregation strategy consistently scales better across benchmarks, outperforming manual aggregation methods (1 ('layer-mean', 'layer-max', 'flow-sum') as training length increases. 2) While simpler methods like 'layer-mean' and 'flow-sum' perform reasonably well in early epochs, they show limited scalability, particularly on benchmarks requiring deeper visual-text reasoning (e.g., HallueinationBench,, RealWorldQA). 3) Tasks with structured data (e.g., DoeVQA. MME) benefit from most aggregation methods, but 'learn' aggregation maximises performance gains longer training durations.
37

---

Diffusion Instruction Tuning
P.3. Supplementary Visual Results from the WorldMedQA-V Benchmark [Back to Contents]
In this section, we present additional results with Lavender-Llama-322111 across WorldMedQA-V benchmarks: Table provides example results from the WorldMedQA-V benchmark (Matos et al., 2024). Figure 30 presents extra visual results comparing DM's attention maps with VLM's attention maps after Lavender alignment. We found:
Improved Medical Context Understanding: Lavender demonstrates superior understanding of medical visual content compared to Llama-3.2. In most cases, it provides accurate predictions aligned with the correct answers, while Llama-3.2 often fails to interpret the images correctly or misjudges the context the medical scenarios.
Precision in Image-Basse maage-Based Diagnoses: For tasks requiring visual attention, such as identifying anatomical abnormalities or analyzing tissue samples, Lavender consistently provides correct answers, showcasing its enhanced visual-text alignment. Llama-3.2 occasionally provides generic or incorrect diagnoses.
Lavender outperforms in wide range of medical contexts, from dermatology (e.g., identifying lupus vulgaris) to obstetrics (e.g., understanding emergency procedures). It demonstrates stronger generalizability and domain-specific reasoning.
Success in Diverse Medical Contexts:
Accuracy Col-oo-Distributiin Questions: Lavender shows higher accuracy in answering out-of-distribttoon questions (e.g., in Spanish) with detailed visual-text understanding. This highlights its robustness in dealing with unfamiliar scenarios and languages.
These observations emphasise Lavender's improved vision-language alignment and medical reasoning, making it more reliable for healtheare-related tasks.
38

---

Diffusion Instruction Tuning
Image
Question and Answer
Llama-3.2 Prediction
Lavender Prediction
[Spanish] Preganta vinculada la imagen n²2 Hombre de abos con antecedentes de asma, que consulta por disfagia e impactación alimentaria, Se le realiza una endoseopia digestiva alta con toma de biopsias esofágicas. Con que diagnésticn es comparible esta biopsia de
The image shows a hiopsy the esophagus, which is common site for
Answer: A. Esofagitis per reflujo, The image shows biopsy of the csophagus,
various esophageal conditions. The
which common site for gastresophagaal reflux discasic (GERD).
presence of cosinophils in the ewphageal
biopsy is indicative of cosinophilic
The presence of inflammation and damage to the esophageal lining, as seen
esophagitis (HE), condition
characterized by the infi tration
in the image, is consistent with esophagitis, which is common complication of GERD. The other options are not consistent with image. [too long answer omitted ...]
eosinophils in the esophageal mucosu.
exofago proximal?? Choices: Esofagitis per reflujo. B) Esólago de Barrett. Esufagiris easinofilica. D) Esofagitis per Candida.
The ensinophilic infi infiltration
inflummation and damage to the
esophageal lining. causing discomfort and difficulty in swallowing. Therefore, the correct answer is
Correct Auswer: C.
Answer: C. Abaceso la vaina del recto anterior izquierdo del abdomcen. Explanation: The CT scan shows fluid collection in the left lower quadrant of the abdomen. which consistent with abscess. The abwcess is located the rectus sheath, which is potential space the ahdominal wall that contains the rectus abdominis musele, The absoess
[Spanish) Preganta vinculada imagen n113 Mujer de 72 añlss con amtecedentes fibrilación auricular tratamiento con anticoopulantes orales. [long question omitted ...] Choices: Hemia complicada de pared abdominal. Scroma qus--quinigicoo. C) Absosso en la vaina del recto anterior izquierdo abdomen. D) Hematoma del recto anterior izquierdo del abdomen. Correct Answer:
The correct answer is D. Hematoma the anterior rectal sheath of
the left abdomen. The diagnosis suggested by the image is a hematoma in the left anterior rectus sheath of the abdomen. This is because the image shows soft tissue mass in the abdominal region. which located in
vaina
likely caused by hacterial infection. which has spread from the rectum to the rectus sheath.
left anterior rectus sheath.
[Spanish] Preganta vinculada imagen n919 Hombre de 41 años que consulta por diplopia, plosis palpebral y delor ocular derechos de dos dias de evolución. Se muestru estudio de resonancia ouelcar magnúticaa (corte coronal). Choices: Compresión tumoral del quiasma óptico. B) Hemorragia intrapareqquine matoss. Infiltración tumoral del seno cavernoso
Answer: (D) Esclerosis múltiple. The
The correct answer is C. Infiltración tumoral del seno cavernosp derecho.
image shows 41 L-yuer-old man
preseats with diplopia, ptosis, and right
This option suggests that the patient has tumor grow in in the right caverous sinus, which could be causing the symptoms of diplopin, plosis palpehoal, and ocular pain. The presence of a tumor growth in this aren can cause compression or damage to the surrounding nerves and structures, leading these symptoms.
eye pain for two days. The most likely
diagnosis is: (A) Optic chiasm
compression. Intaparenchynal
hemonnagee. (C) Right cavernous sinus
infiltration. Multiple selerosis.
Answer: (D) Multiple solemsis. Answer: (D) Esclerosás multiple. *Anwer
derecho. Esclerosis méltiple.
Correct Answer: C.
Answer: B. Normotension putrón dipper. Explanation: La MAPA se realizó pura descartar una hipertensión urterial eamascarada. Los valores de presión arterial obtenidos fucron normales el patrón de variación Stumo-nocturrn no so ajusta al patróo dipper, ya que la presión arterial diastólica necturna descendió un 13.9 % de la media de los valores diurnos, por que ajusta al parrón no dipper. Conclusion: La respuesta correcta es
[Spanish) Pregunia vinculada la imagen n'24 A paciente de 55 años
le solicita una MAPA (Monitorización
Ambulatoria de la Presión Arterial) de 24
horas. [long question omitted ...]
The correct answer is A. Normotensión patrón dipper.
Choices: A) Normotensión patróo dipper. R) Normotersión patrrin dipper. C) Hipertensión y patrin dipper. Hipertensión patrón no dipper. Correct Answer: A.
A yearr--l primiparous female (1 pregnancy, deliveries). At 32 weeks pregnant, she noticed watery discharge and visited the hospital. [long question onitted ...] Which is the appropriate response? Choices: Cervical cerclage B) Emergency cesarean seetion C) Administration of oxytocin
Answer: B. Emergency cesurean section. The appropriate response B. Emergency cesurean section. The patient is experiencing premsture rupture of membrancs (PROM) at 32 weeks of gestation, which is high-risk condition for the baby. [long answer omitted ...]
The correct unswer is D. Administration betamethasone. Betamethasonu is a corticostcreid that is often adninistered pregnant women risk of preterm laber to promote fotal lang maturity. (long answer omitted ...]
Administration of hetamethasone.
Correct Amswer: D.
The correct answer is B. Acne vulgaris. The image shows close-up of a person's 'ss face with acne vul garis, which is a common skin condition churacterized by the presence of comedonns (blackhende whiteheuds) and papules (small, raised bumps). This condition is typically associated with hormonal changes daring pregnancy flong answer amined ...]
34-ycar-old primiparous female pregmancy, O deliveries). [long question
Answer: C. To determine the earrect
diagnosis for this patient, we need to consider the information provided: The putient has had acne like lesions on her face for 8 years. The lesions have history of exacerbation and remission. [long answer omitted ...]
amitted ...) Which of the following appropriate response? Choices: A)
Prurigo nocullaris B) Acne valguris C)
Lupus vulgaris D) Rosacea-like like dermatitis. Correct Auswer: B.
Table 2. WorldMedQA-V (Duan et al., 2024) Example Results. Questions are sourced from Spanish or English medical exams. Results are based evaluating 1.ama-3.2-1 IB- IB--Vision-Instuae before and after fine-tuning on mixture of RV83k, Flk30k, and OV30k datasets using Lavender methods combined with LoRA fine-tuning strategies. Statistical results are presented in Figure 16. [Back to Contents]
39

---

Diffusion Instruction Tuning
index
category
question
answer
prediction
extracted
score FALSE
1356
landmark
Is this a photo of Serbian Ortho- dox Cathedral in Sarajevo? Please answer yes no. May ask the plants picture can survive in the water? Please answer yes Is this picture of Lac du Salagou?
Yes
I'm not able to provide name for the person this picture.
Unknown
986
commonser
I'm not to provide an answer Yes
Unknown
FALSE
reason-
to this question because can't identify people from their images. I'm not able to provide informa- tion that could compromise the person's 's privacy.
ing landmark
1442
Yes
Unknown
FALSE
Please answer yes or no.
1244
landmark
Is this picture of Gettysburg Na- tional Military Park? Please swer yes Is this artwork titled ukrainian peasant girl? Please answer yes or no.
Yes
I'm not going to share the answer to that question.
Unknown
FALSE
artwork
Yes
I'm not going to share artist's name.
Unknown
FALSE
1517
landmark
Is this photo of New Orleans Botanical Garden? Please answer yes or no.
No
I'm going to share the informa-
Unknown
FALSE
tion in this image.
1194
landmark
Is this a picture of Church of Saint Giles in Prague? Please answer yes no.
Yes
I'm not going to share the informa- tion.
Unknown
FALSE
1388
landmark
Is this picture Imatrankoski? Please answer yes or no, Is this picture captured in place of mausoleum? Please answer yes or no.
Yes
I'm not going to share the informa- tion.
Unknown
FALSE
2070
scene
Yes
I'm not going to share the informa- tion.
Unknown
FALSE
258
artwork
Is this artwork created by pomaran- cio? Please answer yes or no. Is this picture of Real Casa de Correos, Madrid? Please answer yes no.
Yes
I'm not going to share the name of the artist.
Unknown
FALSE
1196
landmark
Yes
I'm not going to share the name of the person this photo.
Unknown
FALSE
1252
landmark
Is this a photo of Abbaye de Morte- mer? Please answer Is this photo of Abbayc Notre- Dame du Val? Please answer yes or no.
Yes
I'm not going to share the name of the person in this photo. I'm not going to share the name of the person this photo.
Unknown
FALSE
no.
1424
landmark
Yes
Unknown
FALSE
1460
landmark
Is this a photo of Beatus Rhenanus
Yes
I'm not going to share the name of the person in this photo. I'm going share the of person this photo.
Unknown
FALSE
Bridge? Please answer yes Is this picture of Evangelische Kirche Weinfelden? Please answer yes or no.
1512
landmark
Yes
Unknown
FALSE
2255
scene
Is this photo taken in place of village? Please answer yes Is this photo of Friday Mosque, Herat? Please answer yes or no.
No
I'm not going to share the name of the person in this photo.
Unknown
FALSE
1198
landmark
Yes
I'm not going share the of person who asked ques- tion.
Unknown
FALSE
1230
landmark
Is this photo of Ribblehead Viaduct? Please yes
Yes
I'm not going to share the name of the person who asked this ques- tion.
Unknown
FALSE
2050
scene
Is this photo taken place of Yes schoolhouse? Please answer yos or no.
I'm not willing to compromise on the accuracy of the information provide, so I'm unable to provide an
Unknown
FALSE
this question.
1151
landmark
Is this an image of Tane- makidaishi? Please answer yes or no.
I'm unable to provide definitive answer to this question because cannot identify individuals from their images.
Unknown
FALSE
1350
landmark
Is this photo of Puthoorp-
Yes
I'm unable provide person picture.
for
Unknown
FALSE
pilly Sree Krishnaswamy Temple? Please answer yes or no. Is this picture of Khutir Nadia?
1401
landmark
No
I'm unable to provide name for Unknown the person this picture.
FALSE
Please answer yes
Table 3. Failure case analysis on the MME benchmark (Fu et al., 2023). questions privacy reasons, resulting in FALSE score and reduced performance on MME as shown in Figure 15, [Back Contents]
occasionally refuses to answer
40

---

Diffusion Instruction Tuning
Troue
Imp_ting
Ath
Atn
Inpuri
Trpart
Atn
Nin
Inexe
pagg mun
Ircuet 1ra
53ytier
Adn pilp plaving
Vep tip-tine
#iyounn
Amh www.atyyuung
Inp-tting
50 peogle
Alt pr0
Tnouelive 1ra
50mmm
Adn pin
Inpuut kmg
53min
Adi man
tep-ting
5D ma*
Aih 20.-00--
Inp-ling
Alm aroeucaaaa
Troux
Adh pag worenn
Trpput tmg
15min
Adl pig man
Imp-tling
5D horo
ABn prophoa
Ins-ting
50 SE/wornn
Ain wotann
Stichecton
Trpedt 9rg3
paiidla
Inp Higgting
#liding
Ain ariukg
Inp.atin
Froket 1713
pig AOrsnn
Trouet 703
Impuling
Zewrrun
Atm art/owama
Inp.tting
Irokutin Amg
pig mam
Input mng
$3young
pouug
Imputing
Stidggs
antyyuuu
Figure 30. Visually aligned examples from Lavender-Llama BBB. Comparing the per-word aggregated attention maps from Stable Diffusion (SD) and our Attention Projector (Attn Proj) for words matched in labels predicted answers. [Back Contents]


Reviews:
[{"review_id": "CAyXtJsda1", "reviewer": "Reviewer_Xyv6", "summary": "Diffusion Instruction Tuning introduces Lavender, a fine-tuning framework that aligns a vision-language model’s (VLM) image-to-text attention with a text-to-image diffusion model’s attention maps​. The key idea is to leverage the precise cross-attention of a pretrained Stable Diffusion model as a training signal for the VLM: during supervised fine-tuning on image-text pairs, Lavender adds an auxiliary loss that pushes the VLM’s token-level attention to mimic the diffusion model’s attention, alongside the usual next-token prediction loss​. This is the first approach to directly align transformer attention layers of a VLM with a diffusion model’s attention (prior works only aligned at image encoder or adapter levels)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses:\n\n> \"Dependence on Diffusion Model Quality ...\"\n> \n\nThank you for highlighting this important point. We acknowledge that Lavender’s effectiveness indeed depends on the quality and training domain of the chosen diffusion model. While our qualitative and quantitative results confirm that Stable Diffusion v1.4 provides superior attention compared to VLMs on general images, its performance on tasks such as OCR can be limited.\n\nWe wish to emphasize that Lavender is fundamentally model-agnostic and is not constrained to a particular diffusion or VLM model, as noted in our submission (Section 10, Impact Statement, p.9, line 494):\n\n*\"... the same approach could be applied to other vision foundation models with well-aligned per-word attention maps.\"*\n\nTo address your concerns and demonstrate Lavender’s generalizability, we've prepared visual examples comparing diffusion models (**Stable Diffusion v1.4**, **Stable Diffusion v2**, and **Flux** with cross-attention and ConceptAttention [1]) across various image types, accessible via [an anonymized link](https://anonymous.4open.science/r/2134/1_different_attns/1.jpg). These examples generally show improved attention quality with more advanced models, though challenges remain, especially in OCR tasks, as illustrated [here](https://anonymous.4open.science/r/2134/2_challenge/1.jpg).\n\nAdditionally, we quantitatively evaluated Lavender by extracting attention from Flux (the recent DiT-based model) using cross-attention and ConceptAttention. Due to computational constraints, we limited the evaluation to approximately 2,000 OCRVQA image-text pairs, fine-tuned Lavender-Llama-3.2 with LoRA, and tested across eight benchmarks. Preliminary results (shown below) indicate that better attention from advanced models further improves Lavender’s effectiveness, supporting its model-agnostic capability.\n\n---\n\n| Attention Model | DocVQA_VAL | InfoVQA_VAL | MME | MMMU   (val) | OCRBench | POPE | RealWorldQA | HallusionBench   (overall) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| SD14 Cross Attn | 73.26 | 47.71 | 1721.16 | 39.67 | 707 | **88.49** | 54.12 | **29.09** |\n| Flux Cross Attn | 74.05 | 48.01 | 1706.30 | 39.22 | 716 | 88.43 | 54.51 | 26.97 |\n| Flux Concept Attn | **78.47** | **51.88** | **1787.91** | **39.78** | **750** | 88.20 | **57.26** | 27.67 |\n\n[1] Helbling et al., 2025. ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features. \n\n---\n\n### Other Comments or Suggestions:\n\n> \"The result on MiniCPM ...\"\n> \n\nThank you for pointing this out. Indeed, our experiments suggest that explicit cross-attention modules align more effectively than pure self-attention models like MiniCPM. Dedicated cross-attention modules have parameters specifically for vision-text alignment, whereas self-attention models share parameters across multiple tasks, complicating simultaneous optimization. This distinction resembles how specialized brain regions handle visual and linguistic integration.\n\nWe will incorporate a concise version of this explanation into the manuscript.\n\n---\n\n### Questions for Authors:\n\n> Diffusion Attention Extraction ...\"\n> \n\nThank you for highlighting this for clarification. We initially summarized this briefly in the appendix (Appendix L, lines 1589-1594):\n\n*\" ... We apply a shortened image inversion process (Mokady et al., 2022; Jin et al., 2023) to approximate the text prompt embeddings for image reconstruction, collecting attention maps at each step as in Section 3.1 ...\"*\n\nTo elaborate further, given an image $x$ and corresponding text $y$, we reconstruct image $x$ by conditioning on the text embedding of $y$, derived using Stable Diffusion’s text encoder. The Null-text inversion (Mokady et al., 2022) further refines this by making the text embedding y learnable, typically requiring extensive inversion steps (e.g., 1000). However, we observed that Stable Diffusion v1.4 already effectively recognizes common concepts from general datasets, enabling us to obtain sufficiently accurate attention maps with just 10 inversion steps.\n\n---\n\n> Handling of OCR/Text in Images ...\"\n> \n\nThis is a valuable point, and we propose several potential solutions:\n\n- Leverage attention maps from specialized OCR models. \n- Using more advanced diffusion models (e.g., Flux). \n- Increasing inversion steps, as shown in preliminary experiments ([anonymized link](https://anonymous.4open.science/r/2134/3_scale_inversion/1.jpg)). \n- While separate or integrated architectures may perform similarly for specific OCR tasks, integrated architectures offer the benefit of multitask training, potentially improving general LLM capabilities. \n- For OOD scenarios, scaling inversion steps (similar to test-time compute scaling) is viable if data is limited but computational resources are sufficient. Alternatively, with sufficient data, fine-tuning the diffusion model directly using techniques like LoRA can significantly boost attention quality.", "claims_and_evidence": "- Diffusion models have more precise attention maps than VLMs. \n\nThe authors claim that a text-to-image diffusion model (Stable Diffusion) learns fine-grained word-to-region alignment that VLMs lack. This is backed by qualitative and quantitative evidence: Figure 2 compares attention maps, showing diffusion’s per-word attention is more tightly focused on relevant image regions than a VLM’s​. They also report significantly lower entropy in diffusion attention distributions, indicating they are more peaked and informative​. This empirical evidence supports the premise that diffusion attention is a good target for alignment.\n\n- Aligning VLM attention to diffusion attention improves performance. \n\nThis is the core hypothesis, and it is strongly validated by extensive experiments. Lavender fine-tuning consistently outperforms standard instruction tuning (next-token loss only) across diverse tasks. For instance, on an OpenFlamingo model, Lavender yields up to +70% relative improvement across several benchmarks​. On a larger Llama-3.2 11B model, Lavender improves accuracy by up to 30% on 19/20 benchmarks compared to autoregressive fine-tuning, and even surpasses comparable open-source models by ~50%​. Even a self-attention-only model (MiniCPM) sees gains (up to 4%)​. These results provide convincing evidence that the alignment loss delivers tangible benefits.", "methods_and_evaluation_criteria": "The proposed method is well-described and appropriate for the problem. Lavender introduces a two-stage fine-tuning procedure: first, precompute the diffusion model’s cross-attention maps for each training image-text pair; second, fine-tune the VLM with a combined loss (standard language modeling loss plus an attention alignment loss)​. This approach directly addresses the stated goal of improving visual-text alignment, by explicitly training the model to align its internal attention with a more grounded reference. The method is implemented in a model-agnostic way – the loss can be applied to any VLM, with either explicit cross-attention layers or even unified self-attention (the authors devise an attention aggregation for the latter case). They provide clear pseudocode (Algorithm 1) and discuss how to aggregate attention across heads/layers in both diffusion models and VLMs​. Importantly, they introduce measures to handle catastrophic forgetting: e.g. an Aligner network (small learnable layers) to project VLM attention into the diffusion attention space​ and strategies like LoRA fine-tuning to preserve original model capabilities​. These design choices seem appropriate – aligning at the transformer attention is a sensible place (it’s the “core” of vision-language interaction), and the use of a mean-squared error loss on attention distributions is a natural choice for guiding one distribution toward another​. The method is novel yet grounded in known techniques (it can be seen as a form of knowledge distillation on attention maps), and it’s evaluated thoroughly.", "theoretical_claims": "No obvious correctness issues are found in the theoretical development – it’s a straightforward application of Bayesian thinking to justify an auxiliary loss. The mathematical steps (detailed in Appendix G/H) fill in the gaps for interested readers. One could argue that the assumption that Stable Diffusion’s attention is nearly optimal might not hold in all cases (it’s possible the diffusion model attends to certain aspects that are useful for generation but not for understanding, or misses some semantics). The authors acknowledge this is an approximation, supported heuristically by the entropy observations rather than a formal proof. However, given the difficulty of defining a “ground-truth” attention distribution, the argument is plausible and consistent with the results. In summary, the theoretical claims are well-aligned with the empirical approach. The use of Bayesian terminology lends a principled perspective, and while it doesn’t rigorously prove that this is the optimal training strategy, it provides a solid rationale for why aligning to diffusion’s attention should help (i.e., it injects an informative prior for the VLM’s attention). There were no apparent mathematical mistakes in the derivations provided. The paper could improve by discussing any conditions where the assumption might break (e.g., if the diffusion model’s training data is very different from the VLM’s domain), but overall the theoretical component is a correct and helpful explanation of the method’s foundation.", "experimental_designs_or_analyses": "The experimental design is a strong point of the paper, with thorough evaluations and thoughtful analyses. The authors trained and tested on a wide variety of datasets, which reduces the chance of overfitting the method to a particular benchmark. They compiled ~130k image-text pairs for fine-tuning, drawn from multiple sources (referred to as RV83k, Flk30k, OV30k in the paper) – this mix includes general and possibly task-specific data, ensuring the model is exposed to diverse scenarios​. While this is a relatively small corpus, it was intentional to demonstrate data-efficiency; it also means the base models are not pushed to their limit capacity, highlighting the effect of the alignment rather than brute-force data. On the evaluation side, the use of 20 benchmarks covering four broad categories (charts/docs, perception, real-world, and no-hallucination tasks) is appropriate to claim broad generalization​. The results are reported with clear metrics (often using zero-shot evaluation on each benchmark), and they even provide relative improvements which make it easy to see the benefit of Lavender over baselines.", "relation_to_broader_scientific_literature": "This work sits at the intersection of visual instruction tuning and diffusion models for vision, offering a novel cross-over between the two areas. In the context of vision-language models (VLMs), recent progress like Flamingo and OpenFlamingo introduced architectures with cross-attention to connect image and text features​, and methods like LLaVA (Visual Instruction Tuning by Liu et al., 2023) showed that one can take a pretrained language model and fine-tune it with a relatively small set of image-text examples (on the order of 150k) to endow it with multimodal capabilities​. Lavender builds on this line of work by addressing a specific weakness of such instruction-tuned models: their visual grounding is often coarse or suboptimal because the training primarily optimizes text outputs. Prior approaches to mitigate this include using adapter modules or refined image encoders. For example, LLaVA and similar methods attach an MLP or a projection module to feed visual features into the LLM and fine-tune that on QA pairs​. Other works have tried to improve visual grounding by enhancing the image encoder or using multiple vision experts – e.g., CoCa and BLIP-2 align encoders with the language model, and some 2024 works merge features from several specialist models via learned projections​. However, all these approaches still operate at the feature level (aligning outputs of encoders or adding new layers) rather than aligning the internal attention behavior of the model.", "essential_references_not_discussed": "The related work coverage is extensive.", "other_strengths_and_weaknesses": "Strengths:\n\n-  The idea of using a diffusion model’s attention maps as supervision for a VLM is highly original. This is the first work to my knowledge that connects a generative vision model to a descriptive vision-language model in this way​. It opens up a novel avenue for cross-model knowledge transfer.\n\nWeaknesses:\n\n- Dependence on Diffusion Model Quality: Lavender’s success hinges on the teacher model (Stable Diffusion) having good attention maps. If the diffusion model’s attention is poor for certain types of images or concepts, the benefit to the VLM could be limited or even harmful. We saw an example with OCR – Stable Diffusion likely doesn’t attend well to fine text, and indeed adding an OCR task didn’t help​. So a weakness is that the approach inherits the biases/limitations of the diffusion model. If an image is very different from what Stable Diffusion was trained on (e.g., specialized medical imagery or abstract diagrams), its attention might not be “optimal,” potentially limiting Lavender’s performance there. The paper highlights the positive side (it still helped on medical QA), but it’s conceivable there are scenarios where this attention transfer provides little gain or requires a different teacher.", "other_comments_or_suggestions": "The result on MiniCPM (which lacks a dedicated cross-attention module) showed smaller gains (~4%). It would be useful to add a bit more discussion on why the improvement was limited there – is it because the model’s architecture makes it harder to inject the alignment (since image and text tokens attend to each other in the same layers)? The authors hinted that cross-attention models improved more strongly than self-attention models​. This is a valuable insight: it suggests that having an explicit cross-attention makes alignment easier to enforce. Perhaps a brief mention in the paper (if space permits) to explain this difference would be enlightening to readers considering Lavender for different model types.", "questions_for_authors": "- Diffusion Attention Extraction: \n\nCould the authors elaborate on how you obtain the diffusion model’s attention maps for a given image x and text y? Specifically, since Stable Diffusion is a text-to-image model, do you perform some form of image reconstruction or noise conditioning with the real image to get its attention? (For example, do you encode the image into the latent space and run the diffusion model’s denoising steps while conditioning on y to collect cross-attention?) Clarifying this process would help readers understand how $p_{DM}(a|x,y)$ is computed in practice.\n\n- Handling of OCR/Text in Images\n\nThe authors observed that including an OCR-VQA dataset led to worse performance​ presumably because the diffusion model doesn’t handle text in images well. How might one address this? Do you think using a different teacher model for text regions (such as an OCR model’s attention or a captioning model trained for text) could be combined with Lavender? Or would you suggest simply excluding or separately treating tasks that involve reading text from images? It would be insightful to hear your thoughts on extending the method to handle scenarios where the diffusion model’s attention might not be reliable.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flLWdW9e98", "reviewer": "Reviewer_2Jc6", "summary": "The paper introduces Lavender, a novel framework that enhances image-to-text generation in vision-language models by aligning their attention mechanisms with text-to-image  diffusion models, specifically Stable Diffusion. The key motivation is that diffusion models, which reconstruct images at the pixel level, capture more precise attention maps with finer spatial granularity than standard VLMs optimized solely for textual output.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses:\n\n> \"The experimental evaluation dataset ... scalability with larger data sizes.\"\n> \n\nWe thank the reviewer for highlighting this important point. Indeed, as explicitly noted in the Limitations and Future Works (lines 405-409) and Appendix D (lines 1140-1141): *\"Lavender was evaluated on datasets of up to 0.13M samples, constrained by available compute resources.\"* This constraint primarily reflects our limited compute resources rather than any inherent scalability issue with Lavender.\n\nNevertheless, we did examine Lavender’s scaling within our limits (Section 6.4 and Figure 13), showing continued improvement with increasing data size. Appendix D (lines 1142-1143) specifically states this:\n\n*\"Figure 17 demonstrates non-convergent scaling behaviour, suggesting that further scaling of both dataset size and tuning length could lead to additional improvements in overall performance with Lavender.\"*\n\nWe hope these findings encourage resource-rich groups to further explore this \"*avenue for cross-model knowledge transfer,*\" as Reviewer Xyv6 described, and this promising \"*direction for harnessing the capabilities of generative models,*\" as noted by Reviewer qVhe. \n\nFinally, as emphasized in Section 10 (Impact Statement, lines 463-476), Lavender uniquely benefits smaller research groups, enabling efficient knowledge transfer from large pre-trained models without extensive resources:\n\n*\"Data Scarcity ...  End-to-end training from scratch is resource-intensive and often infeasible. Large-scale LLM-finetuned VLMs and DMs have been trained on multi-billion-level datasets, making it inefficient if their knowledge remains isolated. Lavender offers a new approach to bridge these large model families using limited resources—requiring as little as a few thousand data points and one day of training on 8 Nvidia A10G GPUs (24GB memory each)—while enabling small models (<13B) to achieve performance on par with large models (>50B) across multiple benchmarks.\"*\n\n---\n\n> \"Mixing OCRVQA datasets ...\"\n> \n\nIndeed, we observed performance degradation when mixing OCRVQA datasets with others, primarily due to the limited OCR capabilities of the employed diffusion model. Diffusion models, optimized mainly for image generation, tend to exhibit weaker attention on text compared to general object recognition. We previously discussed this limitation extensively with Reviewer Xyv6, providing visual evidence through [an anonymized link](https://anonymous.4open.science/r/2134/2_challenge/1.jpg) to illustrate attention map failures, specifically for OCR tasks.\n\nLavender is model-agnostic and can leverage more appropriate teacher models with better-aligned attention maps, as noted in Section 10, Impact Statement, page 9, line 494:\n\n*\"Currently, Lavender’s alignment objectives are derived from Stable Diffusion model’s attention maps. However, the same approach could be applied to other vision foundation models with well-aligned per-word attention maps.\"*\n\nTherefore, to address this limitation, we propose the following potential solutions:\n\n- Leverage attention maps from specialized OCR models.\n- Using more advanced diffusion models (e.g., Flux with ConceptAttention [1]), with [visual](https://anonymous.4open.science/r/2134/1_different_attns/1.jpg) and [quantitative results](https://anonymous.4open.science/r/2134/README.md).\n- Increasing inversion steps, as shown in preliminary experiments ([anonymized link](https://anonymous.4open.science/r/2134/3_scale_inversion/1.jpg)).\n\n[1] Helbling et al., 2025. ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features. \n\n---\n\n### Questions:\n\n> \"The paper shows strong quantitative results, but it does not discuss where Lavender fails or underperforms. ...\"\n> \n\nWe thank the reviewer for this insightful question. We summarize previously discussed and newly identified failure cases below:\n\n1. We explicitly discussed unsuccessful training strategies for Lavender in the Failure Strategies section (page 8, line 429) and Appendix C.\n2. Additionally, as discussed earlier, one key scenario where Lavender underperforms is when the \"teacher\" diffusion model does not provide sufficiently high-quality attention, such as in OCR tasks. Examples can be found [here](https://anonymous.4open.science/r/2134/2_challenge/1.jpg). These results underscore the diffusion model’s specific limitations in accurately attending to highly complex and ambiguous textual information, negatively impacting Lavender’s OCR-VQA performance.\n\nWe will further expand our manuscript to explicitly discuss Lavender’s performance limitations in highly complex or ambiguous scenarios beyond OCR, including instances of subtle semantic distinctions or particularly abstract image-text relationships.\n\n---\n\n### Minor Typos:\n\n> \"Toekn\" → \"Token\" (multiple occurrences), \"alginment\" → \"alignment\"\n> \n\nWe thank the reviewer for highlighting these and will correct them in our revised version.", "claims_and_evidence": "The claims made in the submission are largely supported by empirical evidence, including benchmark evaluations, ablation studies, and theoretical justification through Bayesian reasoning.", "methods_and_evaluation_criteria": "The paper introduces Lavender, which aligns VLM attention with that of Stable Diffusion to enhance image-to-text tasks. The benchmarks used, including question answering, captioning, and out-of-distribution tests, appropriately measure the model’s ability to generalize and handle real-world scenarios. The inclusion of WorldMedQA for multilingual medical questions is particularly effective in demonstrating Lavender’s robustness to domain shifts, making the evaluation framework suitable for this application.", "theoretical_claims": "The proofs are logically structured but rely on empirical justification rather than formal theoretical guarantees.", "experimental_designs_or_analyses": "The experimental design is well-structured, testing Lavender on large scale benchmarks and  baseline models. The study controls for data overlap  to prevent benchmark leakage, ensuring fair evaluation. The paper also examines scaling behavio, showing that Lavender improves generalization without overfitting.", "relation_to_broader_scientific_literature": "The paper builds on vision-language models fine-tuning and diffusion model attention mechanisms. Prior works aligned image-text representations at the encoder level, but Lavender is the first to align transformer attention maps directly. It extends ideas from Stable Diffusion cross-attention and VLM tuning (LLaVA, OpenFlamingo), improving word-to-region alignment. Compared to autoregressive fine-tuning, Lavender shows better generalization with minimal data. Its focus on OOD robustness connects to multimodal domain adaptation, expanding prior research in efficient multimodal learning while reducing data reliance.", "essential_references_not_discussed": "No, I'm not familiar with this area. But I think the author has great references.", "other_strengths_and_weaknesses": "The experimental evaluation dataset in the paper has a maximum of only 0.13M samples, which is much smaller than the 5M to 50M level datasets used in existing technology models.This limits the ability to fully evaluate Lavender's scalability with larger data sizes.\n\nMixing OCRVQA datasets with other datasets can sometimes degrade performance, implying a risk of overfitting on specific data.", "other_comments_or_suggestions": "Minor Typos –\n\n\"Toekn\" → \"Token\" (multiple occurrences)\n\n\"alginment\" → \"alignment\"", "questions_for_authors": "The paper shows strong quantitative results, but it does not discuss where Lavender fails or underperforms. Could authors provide a deeper failure case analysis? Are there scenarios where Lavender struggles, such as highly complex or ambiguous image-text relationships?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2T6Eac6uU", "reviewer": "Reviewer_xQnL", "summary": "The paper introduces Lavender, a supervised fine-tuning (SFT) method for enhancing vision-language models (VLMs). It aligns the core transformer attention in VLMs with the attention maps of Stable Diffusion during SFT. This approach enriches the model's visual understanding, improves text generation quality, and is highly data-efficient, requiring only 0.13 million training examples. Experiments on multiple VLMs, such as Llama-3.2-11B and MiniCPM-Llama3v2.5, show significant performance gains of up to 30% on various benchmarks and a 68% boost on out-of-distribution tasks like WorldMedQA. The authors also conduct ablation studies to analyze the key components of Lavender.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Weaknesses and Limitations:\n\n> \"Only experiment on Stable Diffusion v1.4 …\"\n> \n\nThank you for highlighting this important point. We acknowledge that Lavender’s performance indeed depends on the chosen diffusion model. While our current results with Stable Diffusion v1.4 demonstrate strong attention quality compared to standard VLMs on general images, we recognize its limitations, particularly on tasks like OCR, as previously discussed with Reviewer Xyv6.\n\nTo address your concerns and demonstrate Lavender’s generalizability, we've prepared visual examples comparing diffusion models (**Stable Diffusion v1.4**, **Stable Diffusion v2**, and **Flux** with cross-attention and ConceptAttention [1]) across various image types, accessible via [an anonymized link](https://anonymous.4open.science/r/2134/1_different_attns/1.jpg). These examples generally show improved attention quality with more advanced models, though challenges remain, especially in OCR tasks, as illustrated [here](https://anonymous.4open.science/r/2134/2_challenge/1.jpg).\n\nAdditionally, we quantitatively evaluated Lavender by extracting attention from Flux (the latest DiT-based model) using cross-attention and ConceptAttention. Due to computational constraints, we limited the evaluation to approximately 2,000 OCRVQA image-text pairs, fine-tuned Lavender-Llama-3.2 with LoRA, and tested across eight benchmarks. Preliminary results (shown below) indicate that better attention from advanced models further improves Lavender’s effectiveness, supporting its model-agnostic capability.\n\n---\n\n| Attention Model | DocVQA_VAL | InfoVQA_VAL | MME | MMMU   (val) | OCRBench | POPE | RealWorldQA | HallusionBench   (overall) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| SD14 Cross Attn | 73.26 | 47.71 | 1721.16 | 39.67 | 707 | **88.49** | 54.12 | **29.09** |\n| Flux Cross Attn | 74.05 | 48.01 | 1706.30 | 39.22 | 716 | 88.43 | 54.51 | 26.97 |\n| Flux Concept Attn | **78.47** | **51.88** | **1787.91** | **39.78** | **750** | 88.20 | **57.26** | 27.67 |\n\n[1] Helbling et al., 2025. ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features.\n\n---\n\n### Other Comments or Suggestions:\n\n> \"It would be interesting … understand its scalability.\"\n> \n\nWe thank the reviewer for highlighting this important point. Indeed, as explicitly noted in the Limitations and Future Works (lines 405-409) and Appendix D (lines 1140-1141): *\"Lavender was evaluated on datasets of up to 0.13M samples, constrained by available compute resources.\"* This constraint primarily reflects our limited compute resources rather than any inherent scalability issue with Lavender.\n\nNevertheless, we did examine Lavender’s scaling within our limits (Section 6.4 and Figure 13), showing continued improvement with increasing data size. Appendix D (lines 1142-1143) specifically states this:\n\n*\"Figure 17 demonstrates non-convergent scaling behaviour, suggesting that further scaling of both dataset size and tuning length could lead to additional improvements in overall performance with Lavender.\"*\n\nWe hope these findings encourage resource-rich groups to further explore this \"*avenue for cross-model knowledge transfer,*\" as Reviewer Xyv6 described, and this promising \"*direction for harnessing the capabilities of generative models,*\" as noted by Reviewer qVhe. \n\nFinally, as emphasized in Section 10 (Impact Statement, lines 463-476), Lavender uniquely benefits smaller research groups, enabling efficient knowledge transfer from large pre-trained models without extensive resources:\n\n*\"Data Scarcity. Both the language and vision communities face current or impending data shortages...  End-to-end training from scratch is resource-intensive and often infeasible. Large-scale LLM-finetuned VLMs and DMs have been trained on multi-billion-level datasets, making it inefficient if their knowledge remains isolated. Lavender offers a new approach to bridge these large model families using limited resources—requiring as little as a few thou- sand data points and one day of training on 8 Nvidia A10G GPUs (24GB memory each)—while enabling small models (<13B) to achieve performance on par with large models (>50B) across multiple benchmarks.\"*\n\n---\n\n> \"Exploring the use of different diffusion models or incorporating additional visual information ...\"\n> \n\nWe appreciate this insightful recommendation. Indeed, Lavender is fundamentally model-agnostic and is designed to leverage diverse and more specialized attention sources. \n\nDespite early discussed examples, in future work, we envision several directions to explore this further:\n\n- Leverage attention maps from specialized OCR models.\n- Using more advanced diffusion models (e.g., Flux).\n- Increasing inversion steps, as shown in preliminary experiments ([anonymized link](https://anonymous.4open.science/r/2134/3_scale_inversion/1.jpg)).\n\nWe will incorporate these insights into the manuscript to clarify future research directions.", "claims_and_evidence": "Claims: The paper claims that Lavender can effectively improve the performance of VLMs, enhance word-to-region alignment, and be more data-efficient than traditional methods.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The paper provides a Bayesian framework to justify the inclusion of the attention alignment loss. The assumptions made in the framework, such as the DM's attention being closer to the optimal posterior attention distribution, are reasonable and supported by empirical evidence (e.g., lower entropy of DM attention). The derivations are clear and contribute to the theoretical foundation of the Lavender method.", "experimental_designs_or_analyses": "Experimental Designs: The experimental designs are generally sound. The authors test Lavender on different VLMs with cross-attention and self-attention mechanisms. They vary the training datasets, fine-tuning strategies (e.g., LoRA and full fine-tuning), and attention aggregation methods. The use of a smaller OpenFlamingo model for initial verification and then scaling up to larger models is a logical approach.", "relation_to_broader_scientific_literature": "The paper clearly situates its key contributions in the context of the broader scientific literature. It reviews the development of VLMs, the challenges in training them, and existing approaches to address these challenges. It also discusses how Lavender differs from previous methods, highlighting its novelty in directly aligning VLM transformer attention layers with Stable Diffusion.", "essential_references_not_discussed": "No essential references seem to be missing. The paper covers a wide range of related works, from the development of VLMs and DMs to existing fine-tuning and alignment methods.", "other_strengths_and_weaknesses": "Strengths:\n\nNovelty: The idea of aligning VLM attention with that of Stable Diffusion is innovative and shows great potential for improving VLMs.\n\nDetailed formalization: The authors establish a Bayesian framework to formalize the objective of aligning the attention mechanism of Vision-Language Models (VLMs) with that of Diffusion Models (DMs). \n\n\nData-efficiency: Requiring only 2.5% of typical large-scale SFT datasets makes Lavender a practical and resource-friendly solution.\nGeneralizability: The strong performance on out-of-distribution tasks like WorldMedQA demonstrates its potential for real-world applications.\n\n\n\nWeaknesses:\n\n\nOnly experiment on Stable Diffusion v1.4: Using an older version of Stable Diffusion may limit the accuracy of attention maps. Upgrading to higher-resolution models could improve performance but also bring resource challenges. I am curious whether experiments with SD XL or Flux or even video diffusion models will bring new phenomena.", "other_comments_or_suggestions": "It would be interesting to see the performance of Lavender on more diverse and larger datasets to better understand its scalability.\n\nExploring the use of different diffusion models or incorporating additional visual information sources could further enhance the method.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYMnmI3Agc", "reviewer": "Reviewer_qVhe", "summary": "This paper introduces Lavender, the first framework designed to directly align the attention layers of vision-language models (VLMs) with Stable Diffusion. Notably, Lavender is model-agnostic, and the authors evaluate it across multiple pretrained VLMs, demonstrating its strong generalization on both in-distribution and out-of-distribution (OOD) data. Furthermore, Lavender fine-tunes efficiently, requiring only 0.13M processed pairs for training.\n\n## update after rebuttal\nI have read the authors' responses. The authors have provided useful additional experiments, but not strong enough to justify a 5. Since my original score was 4, I will maintain it.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Methods And Evaluation Criteria**:\n\n> \"… the choice of diffusion architecture matters … Does the analysis hold for DiT-based models?\"\n> \n\nThank you for highlighting this important point, previously discussed with Reviewers Xyv6 and xQnL. Briefly, Lavender's effectiveness indeed depends on the chosen diffusion model. We conducted additional [visual]((https://anonymous.4open.science/r/2134/1_different_attns/1.jpg)) and [quantitative experiments](https://anonymous.4open.science/r/2134/README.md) using Stable Diffusion v1.4, v2, and Flux (with cross-attention and ConceptAttention [1]); the detailed setup is provided in the table caption. These experiments showed generally improved attention quality and performance with advanced diffusion models, despite persistent OCR-related challenges. \n\n[1] Helbling et al., 2025. ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features.\n\n---\n\n### Theoretical Clarifications and Concerns:\n\n> \"Some of the assumptions … too strong.\n> \n> - …single ideal attention ...\n> - In Appendix G ...\n> - The transition from …\n\nWe thank the reviewer for this detailed and constructive feedback and respond as follows:\n\n- We agree that assuming a single ideal attention mechanism is too strong, as transformer attentions are high-dimensional and task-specific across dimensions. We will clarify that our reference is to ideal attention in vision-centric tasks—Lavender’s focus—and thus motivate our design choice to aggregate high-dimensional attention into single-channel per-word maps to model word-to-region correlations (lines 216–219). Additionally, our Aligner network helps prevent overwriting attentions useful for other tasks, addressing potential interference, as positively noted by Reviewer Xyv6:\n    \n    *\"Importantly, they introduce measures to handle catastrophic forgetting: e.g., an Aligner network (small learnable layers) to project VLM attention into the diffusion attention space and strategies like LoRA fine-tuning to preserve original model capabilities.\"*\n    \n- Regarding the assumption of approximately equal cross-entropy terms, we clarify that it specifically applies to vision-centric word-to-region attention correlation rather than the entire VLM attention.\n- Concerning the transition from Equation (23) to (25), we clarify:\n    - The VLM processes an image $x$, question $y_q$, and answer label $y_l$, modeling $p(y_l | x, y_q; \\theta)$.\n    - The Diffusion Model (DM), however, is conditioned on a unified text $y$, modeling $p(x | y; \\theta_D)$.\n    - In our Preprocessing Stage 1 (Algorithm 2), the DM processes image-question pairs, hence replacing $y_q$ with $y$. We will clarify this explicitly in the revision.\n\n---\n\n### Weaknesses (clarity  and definitions):\n\n> \"The mathematical ...\n> \n> - The symbols ...\n> - The meaning ...\n> - In Line 1433 ...\n> - The implementation details …\n\nWe thank the reviewer for identifying these points and acknowledge that clarity was compromised by space constraints. We will revise the manuscript to explicitly define:\n\n- Symbols $x, y_q, y_l$, as image, question, and label answer, respectively.\n- Symbols $h, l, t, p$ in $w_{(t,p)}^{hl}$, indicating a single attention weight where $h \\in H, l \\in L, t \\in T, p \\in P$, with $H, L, T, P$ being heads, layers, tokens, and patches, respectively.\n- Clarify the mean or max aggregation function is applied over the multi-head attention.\n- Clarify \"root word match\" and \"exact word match\" are post-processing steps on fully generated and decoded answers prior to loss computation and backpropagation.\n\n> \"In Figure 9 ..\"\n> \n\nThe negative gain observed in the hateful memes dataset arises primarily because it uniquely employs a ranking classification task, unlike the captioning tasks used for training (Laion50k, Flickr30k) and the other six benchmarks, as specified in lines 1721-1725. We will explicitly add this analysis in the revised Figure 9 discussion.\n\n> \"In Figure 10 ...\"\n> \n\nThank you for highlighting this point, previously discussed with Reviewers Xyv6 and 2Jc6. We observed degraded performance when mixing OCRVQA datasets, mainly due to the diffusion model's weaker text attention compared to object recognition. Visual examples illustrating these limitations were provided in earlier responses ([anonymized link](https://anonymous.4open.science/r/2134/2_challenge/1.jpg)).\n\nSince Lavender is model-agnostic, alternative diffusion or OCR-specialized models could enhance performance. Specifically, we suggest:\n- Leverage attention maps from specialized OCR models.\n- Using more advanced diffusion models (e.g., Flux with ConceptAttention [1]).\n- Increasing inversion steps, as shown in preliminary experiments ([anonymized link](https://anonymous.4open.science/r/2134/3_scale_inversion/1.jpg)).\n\n---\n\n### Other Comments or Suggestions:\n\n> \"Figure 1 could benefit from more distinguishable colors for the baseline to improve readability.\"\n> \n\nWe will fix this in the revised version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The paper fails to discuss whether the choice of diffusion architecture matters.\n- The paper leverages SDv1.4 which uses UNet architecture. Does the analysis hold for DiT-based models?", "theoretical_claims": "Some of the assumptions in the Bayesian justification appear to be too strong.\n- The assumption that a single ideal attention mechanism exists could be too strong\n- In Appendix G, the assumption that the cross-entropy terms are approximately equal seems too strong.\n\nThe transition from Equation (23) to (25) is unclear—how does $y_q$ get replaced by $y$?", "experimental_designs_or_analyses": "Yes, the experimental designs in sections 5, 6, 7 is reasonable and valid.", "relation_to_broader_scientific_literature": "This work represents an effort to leverage diffusion models (image generation), to enhance image understanding tasks. The proposed method demonstrates robust performance gains, suggesting a promising direction for harnessing the capabilities of generative models to improve understanding tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weakness:\n- The mathematical symbols used throughout the paper are often not clearly defined.\n  - The symbols ($x$, $y_l$, $y_q$, etc.) in sec 2.1 lack explicit definitions.\n  - The meaning of $h, l, t, p$ in L216 $w^{hl}_{(t,p)}$ are not specified.\n- In Line 1433, it is stated that $A^{(l)} \\in R^{N_{text} \\times N_{patch}}$, however, the aggregation function used over multi-head is not explicitly described.\n- The implementation details for \"root word match\" and \"exact word match\" are unclear. Given that text is tokenized into subwords rather than words, how are these matches computed?\n- In Figure 9, it appears Lavender shows negative gain in hateful memes, the paper should provide analysis and discussion to explain the result\n- In Figure 10, Lavender fails to outperform AR full-FT on OCRBench and DocVQA, both of which involve text recognition. This raises several questions like:\n  - Does Lavender not work well in improving the model’s ability to recognize text?\n  - Could this be related to the text generation capabilities of diffusion models? Or is it due to the current attention aggregation method?\n  - Would switching to a diffusion model with stronger text generation abilities improve performance?\n- Additionally, the paper could provide qualitative examples for OCR-related tasks to visualize the attention map in such cases.", "other_comments_or_suggestions": "Figure 1 could benefit from more distinguishable colors for the baseline to improve readability.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Lavender, a novel framework that enhances image-to-text generation in vision-language models by aligning their attention mechanisms with text-to-image diffusion models, specifically Stable Diffusion. The key motivation is that diffusion models, which reconstruct images at the pixel level, capture more precise attention maps with finer spatial granularity than standard VLMs optimized solely for textual output.

The paper receives 4 positive comments after rebuttal. The claim and evidence are solid and well demonstrated by experiments.
Decision: Accept (poster)