Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Authors: Simon Park
Abhishek Panigrahi
Yun Cheng
Dingli Yu
Anirudh Goyal
Sanjeev Arora
Venue: ICML 2025 poster (2025)
Keywords: vision language models
modality imbalance
SIMPLE-to-HARD generalization
gradient alignment
Primary area: deep_learning->large_language_models
TL;DR: we define a framework for measuring simple-to-hard generalization on text/visual inputs for VLMs and propose strategies to mitigate modality imbalance

Abstract: Vision Language Models (VLMs) are impressive at visual question answering and image captioning. But they underperform on multi-step visual reasoning---even compared to LLMs on the same tasks presented in text form---giving rise to perceptions of *modality imbalance* or *brittleness*. Towards a systematic study of such issues, we introduce a synthetic framework for assessing the ability of VLMs to perform algorithmic visual reasoning, comprising three tasks: Table Readout, Grid Navigation, and Visual Analogy. Each has two levels of difficulty, SIMPLE and HARD, and even the SIMPLE versions are difficult for frontier VLMs. We propose strategies for training on the SIMPLE version of tasks that improve performance on the corresponding HARD task, i.e., simple-to-hard (S2H) generalization. This controlled setup, where each task also has an equivalent text-only version, allows a quantification of the modality imbalance and how it is impacted by training strategy. We show that 1) explicit image-to-text conversion is important in promoting S2H generalization on images, by transferring reasoning from text; 2) conversion can be internalized at test time. We also report results of mechanistic study of this phenomenon. We identify measures of gradient alignment that can identify training strategies that promote better S2H generalization. Ablations highlight the importance of chain-of-thought.

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Simon Park
Abhishek Panigrahi"
Yun Cheng"
Dingli Yu! Anirudh Goyal Sanjeev Arora
Abstract
reasoning abilities of pre-trained LLMs towards solving visual reasoning tasks (Monajatipoor et al., 2023; Carbune et al., 2024; Zhang et al., 2024a). However, VLMs are usually felt to exhibit more brittle reasoning than the underlying LLM, and recent works have tried to understand this as a modality imbalance problem (Peng al., 2022; Huang al., 2022; Fan al., 2023; Wei et al., 2024). For example, presenting the task in an image form can lead to lower performance than when the same task is presented in text form (Zhang et al., 2023; 2024c; Wang et al., 2024b; Zhang et al., 2024d; Fu et al., 2024). Mitigating this modality imbalance is still an open problem.
Vision Language Models (VLMs) are impressive at visual question answering and image caption- ing. But they underperform on multi-step visual reasoning- even compared to LLMs on the same tasks presented in text form form-giving rise to per- ceptions of modality imbalance or brittleness. To- wards a systematic study of such issues, we intro- duce a synthetic framework for assessing the abil- ity of VLMs to perform algorithmic visual reason- ing, comprising three tasks: Table Readout, Grid Navigation, and Visual Analogy. Each has two lev- els of difficulty, SIMPLE and HARD, and even the SIMPLE versions are difficult for frontier VLMs. We propose strategies for training on the SIM- PLE version of tasks that improve performance on the corresponding HARD task, i.e., simple-to- hard (S2H) generalization. This controlled setup. where each task also has an equivalent text-only version, allows quantification of the modality imbalance and how it is impacted by training strat- egy. We show that I) explicit image-to-tex( con- version is important in promoting S2H general- ization on images, by transferring reasoning from text; 2) conversion can be internalized at test time. We also report results of mechanistic study of this phenomenon. We identify measures of gradi- ent alignment that can identify training strategies that promote better S2H generalization. Ablations highlight the importance of chain-of th-ouuhhttt
Here, we introduce concrete methodology to precisely study such issues. First, we design visual tasks where the image information relevant to the task can also be represented as text (e.g., LaTeX code). This allows a direct comparison of the effect of training strategies in individual modalities and combinations. Second, to allow a clear comparison of different training strategies, we measure the brittleness of learning with simple-to-hard (S2H) generalization, where models trained on SIMPLE examples of a task and evaluated HARD examples.
We create a set of synthetic tasks² that involve algorithmic visual reasoning (Ghosal et al., 2024; Cherian et al., 2023; Zhang et al., 2024b): Table Readout (reading out table en- tries in an order specified visually), Grid Navigation (finding valid paths through grid-like like structures while avoiding obsta- cles), and Visual Analogy (identifying logical patterns across sets of abstract visual examples and applying analogical rea- soning). Each task requires many reasoning steps while dynamically shifting attention over a sequence of small re- gions in the image. SIMPLE and HARD examples differ in the length and complexity of the necessary reasoning steps.
1. Introduction
Many Vision Language Models (VLMs) (e.g., LLaVA- series (Liu et al., 2023c;b; 2024a)) fuse an LLM with visual encoders which allows them to harness the impressive
The SIMPLE tasks are difficult for current frontier VLMS such GPT-40 and Claude-3.5 Sonnet (Achiam al., 2023; Anthropic, 2024) (Appendix I.1.. Since we work with smaller open-parameter models, our methodology consists of using supervised training to precisely inject capability at a task in one modality and then study how variations in train- ing affect the gap in S2H generalization between modalities.
'Equal contribution 'Princeton Language and Intelligence, Princcton University "Meta AI. Correspondence to: <(juhyunp, ap34, yc6206) ) @ princeton.edu>..
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Creating such tasks was more nontrivial than expected, for reasons described in Appendix C.6.
Code available at VL.M-SHH PLI codebase

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Algorithmic Visual Reasoning
Modality Imbalance
(Consecutive Table Readout)
SASD a
S2H Generalization Text
No S2H Generalization on Image
A
6
7
3
6
9
8
tanditabriee)
Veegia
Read numbars from (Row
B, Col B)
E bewditabelar]
Row A Col B
Row A Col C
Row Coll H
Transfer Reasoning Capability across Modulity
1,9
Figure 1. (Left) Example Data Point for Consecutive Table Readout. Input table can be provided as an image or LaTeX code. The task is sequentially read numbers from a start cell to an end cell in row major order. (Right) Illustaaioon of Key Concepts using examples from Consecutive Table Readout. We observe that current models can S2H generalize on text- - when trained to read short sequences from small LaTeX- formatted tables, the models can read longer paths from larger tables, also provided in LaTeX code. However, they fail to length- generalize on images. To address the generalization gap and imbalanced learning of different modalities, our goal is to transfer the generalization behavior from text to image modality.
Since the tasks are difficult for frontier VLMs, we expect the takeaways from our study to be of broader interest.
eralization in the image modality (Section 4). The findings in (ii) should be interpreted as suggesting that simple image-to-text conversion could be a promising intervention to reduce modality imbalance in future VLMs whose base LLM does exhibit S2H generalization in the text modality.
Illustrative example of Consecutive Table Readout: Given table of numbers and indices of two table cells (i, and (k, (), the model needs to output every table entry between these two cells in row-major order. The input table can be provided as an image or as text (i.e.. LaTeX code), allowing the kind of study sketched in Figure 1. In the SIMPLE task, the length of the output sequence is 5 10, whereas in the HARD task, can be as long as 30. Therefore, S2H generalization here is type of length generalization, a well-studied concept in LLMs (Zhou et al., 2024a). SFT on 8 x 10a SIMPL.F-text examples yields 80% accuracy on HARD-text examples. However, training on the SIMPLE-image examples results in only 20% accuracy on the HARD-image examples. The 60%p difference a measure of the modality gap or modality imbalance.
A surprising finding is that even though explicitly training on image-to-text conversion seems necessary for S2H generalization, the final trained model can generate the correct solution without explicitly extracting the image content as text: the image-to-text conversion skill gets inter- nalized! (This also greatly reduces the inference-time cost.) Therefore, try to understand the effectiveness of this key intervention at the level of training gradients. We find that gradients from SIMPLE-image reasoning examples can help reduce loss on HARD-image inputs with the above interven- tion (Section 5); this gradient alignment merits further study.
On tasks where we need to inject reasoning capability on the HARD task, our findings about gradients inspired more effective two-phase training (Section 4.3). The first phase teaches the model to do image reasoning via text conversion on a few SIMPLE examples. We find that inclusion of this phase substantially improves gradient alignment in the ear- lier phases of training, when gradients have larger norms, which allows for more effective S2H generalization on the image modality. This finding is in accord with previous empirical evidence that highlights the importance of visual- language alignment in VLM training (Fan et al., 2024).
1.1. Paper Overview
We study training strategies that incorporate various types of supervision: text-based, image-based, and combinations of the two (Section 2.4). We find that the most reliable way to alleviate the gap is to teach the model image reasoning via text conversion - explicitly extracting information from the image in text form before generating the solution using COT. Specifically, we find: (i) for tasks where the model exhibits S2H generalization in the text modality, training on image reasoning via text conversion greatly helps to mitigate the gap (Section 3): (ii) for tasks where the S2H generalization failed in both modalities, applying the idea from (i) while also injecting reasoning capability on the HARD task in the text modality leads to S2H gen-
2

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
(a) Table Readout
(b) Grid Navigation
(c) Visual Analogy
Examape typs,
April
May
Zota
EIGHT
SEVEN
FOUR
FIVE
Kappa
SIX
FOUR
SEVEN
EIGHT
Particle
TWO
SIX
SEVEN
EIGHT
O
O
A
?
Apex
ONE
ZERO
FOUR
LIGHT
-
Peak
ONE
FIGHT
SIX
FIGHT
Alpha
ZERO
NINE
FOUR
position tion
type
color
AND
OR
X00
progremsion
Figure Illustration of our synthetic tasks: Table Readout involves reading numbers along a specified path a table Grid Navigation involves navigating a grid to collect objects while avoiding obstacles. Visual Analogy involves solving analogical reasoning queries using two in-context examples. More details on Visual Analogy: For the example Visual Analogy above, we only include one in-context example for simplicity and provide annotations for clarity. In the first row, the first cell contains a rectangle, whereas the second and third cells contain a circle and a rectangle. Therefore, the in-context example is consistent with applying the OR relation along the shape type domain. The model then needs to identify the correct option that corresponds to applying the OR relation to the first two cells of the query along some (potentially different) domain. See Appendix for non-annotated example images that are provided to the model.
2. General Setup
order. The number of cells to read in SIMPLE and HARD examples is respectively 5-10 and 25-30. Only for this task, we additionally prepare set of MEDIUM difficulty level, where the number of cells read is 15-20. Training on SIMPLE examples and evaluating on MEDIUM examples can also measure S2H generalization.
2.1. Model
In line with Shi et al. (2025) that show the benefit of com- bining multiple image encoders in VLMs, we trained Eagle- X2 X2-Llama3-8B, variation of Eagle-X5 that uses Llama3- 8B- IB-Instruct (Dubey et al., 2024) as the LLM backbone and CLIP-448 (Radford et al., 2021) and ConvNeXt (Liu et al., 2022) as visual encoders. Since the original paper found only minor benefit beyond the two encoders, we do not use all five visual encoders. See Appendix D) for more details on the training. In Appendix F, replicate some of the experiments on Viven2.5-VL-3B-Innllil (Bai et al., 2025) and observe consistent results.
Grid Navigation: in either image its LaTeX code) from designated start cell to an end cell while collecting all specified objects and avoiding obstacles. SIMPLE examples contain 1-2 objects and type of obstacle (Figure 34). HARD examples involve 2 2 distinct objects and 2 types of obstacles (Figure 35). The task can be solved by depth-first search (DFS). Recent works (Kim et al., 2024; Wu et al., 2024a; Wang et al., 2024b) explored similar synthetic tasks in LLM and VLM evaluation.
The model navigates in a 2D grid (given
and 7B-Instruct
2.2. Tasks
We briefly describe the tasks that we consider and the SIM- PLE and HARD setup of each task below (summarized in Table 1; fully detailed in Appendix C).
Visual Analogy: The model reasons about attributes and relations between geometric figures in puzzle (given in the image or text description). It analyzes two in-context examples and applics an analogous reasoning to choose from 4 options to complete the query. SIMPLE puzzles have examples and query vary along the same attribute following common relation (Figure 36). HARD puzzles have examples and query vary along different attributes following a relation, and the combinations of attribute
The model sequentially reads numbers along a highlighted path a table (given in either image or its LaTeX code). SIMPLE examples consist of 1-4 linear segments in spiral or sinusoidal path patterns with average length of 12 (Figure 32). HARD examples consist of > 4 linear segments, featuring longer and arbitrary compositions of spiral or sinusoidal path patterns with an average length of 35 (Figure 33).
Table Readout:
and relation held-out from training (Figure 37). This task is adapted from Barrett et al. (2018) and Hill et al. (2019).
Consecutive Table Readout: This variant of Table Readout, modified to make the reasoning simpler?. The model sequentially reads numbers in row major
Pattern-Heldout Visual Analogy: This a variant of Visual Analogy, modified to make the reasoning simpler. See Appendix G.2 for more details.
'There is fixed underlying rule for where the cell should be. The model doesn't 't need to make a decision at each step.
3

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
2.3. Training Data
Text
Mix (eval on HARD-text)
Formally, we let f X + Z denote reasoning task, where
Image
Mix (eval on HARD-image)
X' refers set of imput data, further split into XSIMPLE and HHARD and Z refers to set of answers. Each input E X can be presented in text format x1) or in image format x())
MEDIUM
HARD
S2H Accuracy
1.0
0.5
For each pair of data x and solution f(x), we also create a chain-of-thought reasoning trace, which we denote by (x) We also define a prompt Poonvert that we op- tionally prepend at the start of chain-of of-thought to signal explicit image-to-text conversion on image input'. Hence, our training dataset is defined by input x (which can be given either as x(9) or x())), chain-of of-thought CoT (x), and the final answer f(x).
0.0
1
2
4
8
16
1
2
4
8
16
Number of Training Data (x10*)
Figure 3. S2H Generalization of different supervisions for Con- secutive Table Readout to MEDIUM (left) and HARD (right) ex- amples. S2H generalization text of Text (x) outperforms S2H generalization on image of Image (A), highlighting modality im- balance. Mix (+) mitigates this imbalance.
For each task f. we use the same Python script and a fixed template to generate all tuples (x(1), x()) CxT(x), (x))..
2.4. Types of Supervision
Our controlled experiments study the effect of the following types of supervision on SIMPLE examples during training:
adapted supervision strategies will have a + sign appended to represent this additional component (e.g., Mix+ adapted from Mix supervision).
(a) Text supervision: given a text input x(1) E SIMPLL we train on the gold output containing a chain of- of-thought trace CoT and the final answer (xx).
3. Modality Imbalance in Consecutive Table Readout
(b) Image supervision: given an image input (()) E ASIMPLE we train on the gold output containing a chain- of of-thought trace (x) and the final answer fx).
We use Consecutive Table Readout introduced in Section I and Section 2.2 to illustrate the S2H generalization gap between different modalities and propose training strategies needed to address it. We compare different types of supervision by training the prescribed SIMPLE examples and measuring the improvements on the exact match accuracy on two different difficulty levels: (a) MEDIUM: reading 15-20 consecutive numbers and (b) HARD: 25-30 numbers (more challenging).
(c) Image-via -ixt supervision: given image input x(0) E XSIMPLE, train the gold output containing the con- version prompt Poonwent, converted text (f),, chain- of of-thought trace (x), and the final answer (x).
(d) Text+ supervision: we train on an equal mix of Text and Image supervisions.
To demonstrate the modality imbalance, we compare Text and Image supervision. Figure 3 shows that the S2H generalization gap between the two is substantial. For HARD, while Text supervision achieves 80% accuracy on HIARD-txX examples, Image supervision achieves only 20% on HARD-image examples.
(e) Mix supervision: train on equal mix of Text, Image, and Image-via-Text supervisions.
We train the model on one of the above supervision types with auto-regressive loss (1) that takes in the model's logits on an input example and returns the average loss on a se- lected set of tokens. For example, for Image supervision, we will represent the input example as (x(0), (x) fxx), and compute the loss on (CoT((x, (x)). During the evalu- ation, test whether the model predicts fx) correctly for given input.
In order reduce the gap. we consider training strategies that can leverage strong S2H generalization of Text super- vision help S2H generalization of Image supervision. Two candidates are Text+ supervision, which simply mixes in Text and Image supervision, and Image-via-Tee wix- supervision, which trains the model to first convert the image input to its text format and then output the solution. Text+Image supervision induces the model to implicitly make the connection that the image and text formats are equivalent, while Image-via -ixt-- supervision makes this
In Section 4, we will adapt some of the above supervision strategies to also include HARD Text supervision". The
e.g.. "Convert the provided image to text"
"ddnticcll as Text except we use a HARD-text example, i.e. NEaRo We note the subtle difference between "HARD-text example," which refers the data x("), and "HARD Text," which a type of supervision with a prescribed (input, output) structure:
(C),)))(), f(x).. Similarly, a "HARD-image example" distinct from HARD Image
*Correctness requires all generated numbers be in the correct order; see Appendix E.

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
4. Full Study: Table Readout, Grid Navigation, Visual Analogy
HARD-text
HARD-image
S2H Accuracy
Generation Length
We now consider our three main tasks: Table Readout, Grid Navigation, and Visual Analogy (Figure 2). These tasks require the model to generalize to HARD examples by com- posing reasoning patterns learned from SIMPLE training examples, which has been known to be difficult for LLMs (Yu al., 2024; Zhao al., 2024; Wu et al., 2024b; Huang et al., 2023; Dziri al., 2024).
0.4
500
0.2
0.0
Text+ Image
Image- via-Text
Mix
Text+ Image
Image- via-Text
Mix
These tasks are considered non SZI1-generalizing because the model struggles to generalize to HARD instances after being trained on SIMPLE examples. In any of the three settings, training with Text, Image, and Mix supervision (which only include SIMPLE examples) cannot achieve more than 25% S2H generalization on either text or image.
Figure 4. Efffet of Image-via-Tex, -exxt on Consecutive Table Read-
out: S2H Generalization (left) and Generation Length (right) for HARD task. Number of training data is 16 10 Text+Image un- derperforms Mix and Image-via via vix- supervision. Image -ia Text supervision improves performance slightly but at the cost of longer generation due to explicit image-to-text conversion at inference.
The failure to S2HH-generalize in either input modality high- lights the insufficient general reasoning capacity of existing models on these tasks. We then adapt Mix supervision to include HARD Text in training and measure whether the improved performance on HARD-text can result in better S2H generalization in the image modality.
connection explicit. We compare the two training strategies in Figure and show that -eieevvi-TText supervision shows much better performance on IIARD-images.
However, Image-via-Text supervision has a key drawback: trained models have significantly higher inference costs, since the conversion of image to text before generating the solution leads to 3x longer outputs, which limits the real-world practicality. To address this, we propose Mix supervision. which combines Image-via-Texe and Text+ Image supervision. This teaches the model to align the modalities, while also teaching it to not always rely on the image-to-text conversion.
4.1. Improved performance on HA HARD-text can transfer to S2H generalization on image
Mix+ supervision. adapted from Mix from Section 3, trains the model with an equal mix of HARD Text supervision and SIMPLE Mix supervision.
Mix+ supervision shows significantly better image S2H generalization, demonstrating an effective transfer of reasoning capability from text to image.
With only 3 x 10a data, Mix+ quickly improves the model's accuracy on HIARD-text examples to 95%. At the same time, Mix+ supervision leads significant improvement image S2H generalization the model can achieve 64%, 92% and 35% S2H accuracy on HARD- MARD-i (respectively, Table Readout, Grid Navigation, and Visual Analogy) after being trained on 12 x 10 data (Figure 5). We conclude that Mix+ supervision can effectively transfer the injected reasoning on HARD-text to S2H generalization on images.
Mix can mitigate the modality imbalance by improv- ing S2H generalization on images, while maintaining inference cost.
Mix supervision retains most of the S2H generalization per- formance of Image-via vext supervision while reducing gen- cration length by directly solving reasoming tasks from im- ages (Figure 4). In Figure 3 (left), we show that it can almost completely match the S2H generalization performance of Text supervision for MEDIUM. On HARD level, even though it does not fully close the gap between text and image in- put (Figure 3, right), the gap be further reduced with short text-only warm-up training. We discuss this further in Section 6.
4.2. Dual capability of Mix+
Motivated by the observed benefit of Image-via-Ted supervision from Section 3, we also measure the image S2H generalization of Image-via-Text+ supervision (an equal mix of HARD Text supervision and SIMPLE image-via-Text supervision). On Table Readout and Visual Analogy, observe that Image-via-Texde supervision outperforms Mix+ supervision in S2H performance on HARD-image by a substantial (20-30%p) gap (Figure 6).
Consistent results across tasks: In Appendix G.2, we show similar results on Puttern-Heldeee Visual Analogy.
close this gap, we prompt Mix+ models with an addi-
5

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Text (eval on HARD-text)
Image
Mix
Mix+
Align-Mix+
Table Readout
Grid Navigation
Visual Analogy
1.0
52H Accuracy
0.5
0.0
3
6
12
24
0.5
12 Number of Training Data (x10*)
1.5
3
0
3
12
24
Figure 5. Results S2H-ggneralizing tasks: We report the S2H generalization image Table Readout (left), Grid Navigation (middle), and Visual Analogy (right). S2H generalization on text from Text supervision serves as a reference (in gray dashed line). Text, Image, and Mix supervisions fail generalize, highlighting the gap between SIMPL.F. and HARD examples. Mix+ improves performance, while Align-Mixe further enhances generalization with an initial alignment phase.
Table Readout
Visual Analogy
4.3. Benefits of two-phase training
S2H Accuracy
Generation Length
Given previously observe that Image-via-Tex" supervi- sion helps with S2H generalization, we add an initial phase that trains the model with Text and Image-via -Tett super- vision on SIMPLE examples. The goal is to precondition the model (via SIMPLE Image-via-Text supervision) to align text and image reasoning on SIMPLE examples. Intuitively, the preconditioning must be useful generalize this knowledge HARD examples later when trained with Mix+ supervision. We call this two-phase approach Align-Mix+?.
1000
0.5
500
0.0
Image- via-Text+
Mix+
+Convert
Align-
Mix+
0
via-Text+
Mix+
+Convert
Align- Mix+
Image-
Align Mix+ significantly boosts S2H generalization on image to an accuracy of 76%, 96%. and 56% on HARD- images (respectively Table Readout, Grid Navigation, and Visual Analogy) after training on 12 10 data (Figure 5), Align-Mix+ also maintains inference cost (Figure 6).
Figure 6. Image-via-Texee -eeeet+ Table Readout and Visual Analogy: S2H Generalization on image (left) and Generation Length (right) with 12 x 10" training examples. Image-via-Texxe Text+ achieves good performance but with higher inference cost. Mix+ matches the performance of Image-via -ia Text+ appending "Convert" to the prompt (+Convert) or by adding an alignment phase (Align-Mix+).
Align-Mix+ further improves image S2H generalization, while maintaining inference cost.
tional inference time token, "Convert", which appears at the start of the Image-via-Texx responses (Section 2.4). We observe that the models respond with an accurate text conversion before generating the reasoning tokens.
5. A Study on Loss Dynamics and Gradient Alignment for S2H generalization
Our findings show that S2H generalization can be trans- ferred across modalities by simply mixing different types of supervision. This happens without any explicit matching of representations, which motivates us explore training gra- dients obtain insights into how each strategy contributes to S2H generalization. Here, analyze the evaluation loss behavior on HARD Image and HARD mmage-via-Texi examples during training. Similar gradient studies have
Mix+ models exhibit dual capability in reasoning with or without image-to-text conversion.
This is in line with the findings in Su al. (2025) of the dual learning capability of LLMMs in short and long reason- ing. Note that when explicitly prompting Mix+ models to perform image reasoning via text conversion, this still in- curs similar cost in generation length as Image-via -iage-via-Text+ (Figure 6). We discuss more in Appendix 1.6.
'For the main experiments. we use x 10 training examples in the first phase. See Appendix 1.3 for ablations the number and composition of data used in the alignment phase.
Identiccl as Image and Image Text except image input (1) E XEARD-
6

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
been proposed for measuring influence (Koh & Liang, 2017) of training data points on evaluation tasks (Park et al., 2023; Xia et al., 2024; Engstrom al., 2024).
Image
Text+Image
Mix
Gradient Alignment
Salution | Image())))) 5))
0.2
HARD Eval Loss
5.1. A study on Consecutive Table Readout
10
0.1
In Section 3, we showed that Mix outperforms Tmxt+ supervision in S2H generalization images. The key factor driving this improvement was the inclusion of Image-via-Ted supervision. Here, we show that Mix super- vision reduces evaluation loss on HARD Image examples (therefore improving evaluation accuracy) through a better gradient signal. To do so, we measure the alignment between gradients on SIMPLE and HARD Image examples. Let ((r:s)()) (x) denote the loss on solution given image, i.c., (1)
0.0
2-13
2
2-4
0
100
200
SIMPLE Gradient Norm
Training Steps
Figure 7. Analysis of gradients on Consecutive Table Read- out: (Left) Average Gradient Norm SIMPLE Image examples (Exe (s(
vs. Gradient Alignment Score (Equa- tion (3)) for different training checkpoints; (Right) Average Loss on solution given HARD image (((ss) during training. Larger gradi- ents for Mix have higher alignment score compared to Text+Image and Image, showing the importance of Image-via-Tee vix-Texx supervision for generalization.
(u:sy (x) : 1(fo((x ,y)),
where y contains both (x) and the answer f(x). We also denote the loss on solution given HARD image as
Insights from the evaluation loss dynamics: We use the following additional notations to report the average loss on specific tokens on HARD Image-via-Text example and un- NSIMPLE and HEARD (i.e. Exeximms Vl(r,s) (x) and E (l(,,s) respectively)", then we define the gradient alignment score as 10;
(1is)) == ExE (1:5))
If ESIMPLE and SHARD denote average gradients
derstand which components help the model learn to reason on H HARD-images via text conversion, and how it translates to a direct solution on HARD-image examples.
(3) Intuitively, the gradient alignment score measures how much the evaluation loss (on HARD Image) can be reduced by taking gradient updates from the training data (SIMPLE Image), relative to training on evaluation data directly (see Theorem H.1 for formal statement). In Figure 7, we plot this score against the gradient norms on the training data. A stronger gradient alignment larger values of gradient norm is preferred because the evaluation loss can be reduced more when the training gradients are larger.
(ESIMPLE: SHARD)/ (HAAR:: SHARD)
HARD image-to-text conversion: Average loss con- verted text tokens given the image and the conversion prompt "):
(H) == ExE Eraso ((So((x))) Promvert, x ())) ()).
Solution given HARD image and text: Average loss on solution tokens given the image, the conversion prompt, and the converted text:
((H) (T: == Ex ((f((x(()) Pronvert, (1)) y)),)
(5)
Mix achieves a high gradient alignment score, espe- cially when gradient norms are large. This improved alignment leads a significant initial drop in the evalu- ation loss (loss on solution given HIRDD-image), which then continues to improve throughout training.
where y contains both CxT((x and the answer f(x.
In Figure 8, we report the above losses for Mix, Mix+, and Align- Align-Mix+. Since the model does not see HIARD-image examples during training, these losses (along with ( evaluate the S2H generalization on image. We observe:
5.2. A study on Table Readout
1. HARD image-to-text conversion loss (Equation (4)) of Mix matches Mix+, showing that training on SIMPLE Image-via-Texe examples suffices to generalize the con- version subtask to HARD-images.
In Section 4, we showed that Mix+ improves S2H general- ization over Mix, while Align-Mix+ can further improve over Mix+ with an additional alignment training. Here, we study how each included component helps S2H generalization across the training strategies.
2. There is a significant gap in the loss on solution given HARD image and text (Equation (5)) between Mix and Mix+ This implies that including HARD Text is neces- sary to fully generalize reasoning to IIRDD-images.
Following Park et al. (2023), we apply a random projection on gradients to 1096 dimension for an efficient storage. IIn Appendix H.3, we give alternatiee measures of gradient alignment. Our takeaways remain the same.
# indicates the additional conversion prompt Poonvert
7

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Mix
Mix-+Mix+
Mix+
Align-Mix+
Image-to-text (4fe:T)
Solution Image+Text (1f), NT:s))
Solution Image (1
HARD Eval Loss
Image-via-Text+ Final
10-1
10-3
100
200
300
400
100
200
300
400
200 300
400
Training Steps
Figure 8. Analysis of evaluation losses HARD examples Table Readout: (Left) HARD image-to-text conversion loss a1 (8) (Eq,4)); (Middle) loss on solution given HARD image and text (1 (B)
(I+:T)
(T. (Eq.5)); (Right) loss on solution given HARD image (1:S)
(Eq.2)). Mix matches Mix+ in ((4)
showwigg ((#:T)) showing that training on SIMPLE Image-via vix-eexx examples is sufficient for HARD image-to-text
(U.#T;:S) showing the need for HARD Text examples for generalization. Taking intermediate checkpoint of Mix and completing the training with Mix+ (Mix - Mix+) leads to evaluation loss values comparable to Mix+, suggesting that HARD Text examples can be introduced later. Align-Mix+ starts with smaller ((R)
conversion. Mix performs worse in ((H)
(IIIT) and ((R) (I.#T::) losses, which helps the model loss than even Image-vid Text+, that reflects in lower (11)) loss.
achieve lower (44) (35)
As ablation, took an intermediate Mix checkpoint and completed the training with Mix+ supervision This transition resulted in negligible changes to HARD image- to-text conversion loss (Equation (4)), while loss on solu- tion given HARD image and text (Equation (5)) and loss on solution given HARD--maae (Equation (2)) decreased significantly, approaching the values for Mix+.
Align-Mix+ Mix+
Gradient Alignment
0.15
0.10
0.05
Losses on HARD Teate examples start signifi- cantly lower for Align-Mix+ after the alignment phase. This shows that training on SIMPLE Image-via-Tent TTxt ex- amples return favorable starting point, even if they aren't 't sufficient for generalization. It then achieves better loss on solution given HARD image and text (Equa- tion (5)) in the end, which also translates to an improved loss on solution given HARD image (Equation (2)).
0.00
10-5
10-4
10-3
SIMPLE Gradient Norm
Figure 9, Analysis of gradients on Table Readout: Average Gra- dient Norm on SIMPLE Image examples (Fxexxums (5(r,) (x) lla) Gradient Alignment Score for different training checkpoints. Larger gradients for Align-Mite -Mi+ have higher gradient alignment scores. Mix+ has better gradient alignment scores than Mix.
Insights from the gradient alignment score: We can further quantify the differences training strategies with the gradient alignment score (Equation (3)) between SIMPLE and HARD Image examples (Figure 9). Intuitively, higher gradient alignment at each step should accumulate to better generalization on HARD-images. We observe:
6. Further Ablations
We perform several ablation studies to identify critical training components that underlie our findings. We push all the details and discussions to the appendix.
Task interactions in multi-task training: We compare Mix, Mix+ and Align-Mix+ with an equal mix of 3 tasks in Section 4. We observe that multi-task training significantly boosts performance Table Readout and Grid Navigation but hurts Visual Analogy, which shows the effect of task interactions in our strategies. See Appendix L88.
1. Mix exhibits lower gradient alignment compared to Mix+. Training solely on SIMPLE examples fails to provide gradients aligned HARD Image. Including HARD Text examples significantly improves gradient alignment.
On the other hand, Align-Micg Minn-Mix+ has higher gradient align- ment than Mix+ in earlier training steps, when training gradient norms are large. We give detailed analysis in Figure 17 in Appendix H.
Transferring reasoning from image to text: We also experiment with including HARD Image supervision in train- ing and evaluating on HARD-text input, which gives much stronger results (Table 8 in Appendix G.4),
We preserved the optimizer states and learning rate schedule.
8

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Text warm-up pretraining: We add a text warm-up pre- training (TW) phase before the training of VLM simulate the effect of stronger LLM backbone. This pretraining phase completely solves the modality imbalance or further boosts performance of Align-Mix+. See Appendix G.5.
performance by 5.1 5.1%p on descriptive questions. Therefore, our synthetic datasets involve useful skills that can also help improve VLMs real-world benchmarks.
Limitations and possible future directions: We believe Mix or Mix+ may not be the optimal approach to improve image generalization on tasks where the model exhibits S2H generalization in the text modality. Curriculum- strategies (Xie et al., 2024; Mindermann et al., 2022) that dynamically adjust the data mixture could yield better results. However, generalization gap between text and image inputs, which can be bridged by transferring learning from the dominant modality (text) to the weaker one (image). Therefore, we foeus on the effectiveness of our training strategies in transferring knowledge learned on text input to image input. restricted the scope of
Importance of chain-of-thought: Completely removing or progressively internalizing CoT (Deng et al., 2024) fails to achieve image S2H generalization, suggesting that CoT crucial in our strategies. See Appendix 1.7.
goal is to emphasize the HARD
7. Discussion, Limitations and Future work
We explore the modality imbalance in VLMS by measuring S2H generalization. We show that on tasks where VLMs can reliably show generalization on text input after fine-tuning, Mix supervision can induce a similar level of generalization on image input. We then propose algorithmic tasks, where models trained on SIMPLE examples fail to generalize to HARD examples in either modality. Mixing HARD Text examples in training can help the model generalize on HARD-image input, revealing S2H generalization transfer capabilities of these models.
In the interest of crispness, our study with a small set of prompts and limited (and synthetic) image distribution. But doing so allowed a clearer and quantitative look at modality imbalance and how it can be bridged.
Our results highlight that chain-of-thought (CoT) reasoning can play an important role. However, even minor modi- fications CoT significantly affect the transferred S2H generalization results on image inputs, and mitigating this brittleness through robust training strategies beyond Mix+ is crucial. Future work could focus on mechanistic insights into our trained models to design more generalizable strategies targeting specific model components.
Related Works: Current VLM benchmarks are often solvable without the visual input. To remove such bias, we designed controllable tasks and provided framework (S2H generalization) to quantify and mitigate modality imbalance. While S2H generalization has been extensively studied for LLMs, similar investigations remain scarce for VLMs.
Prior strategies to address modality imbalance and cross- modal transfer often rely on matching representations or optimization techniques. However, through gradient align- ment studies, we demonstrate that auto-regressive training effectively aligns reasoning across modalities.
A Acknowledgements
SP, AP, CY, and SA acknowledge fundings from NSF, PLI, DARPA, ONR, and OpenAL. CY is additionally supported by the Francis Robbins Upton Fellowship in engineering We thank Xingyu Zhu, Bingbin Liu, Nikunj Saunshi, Sad- hika Malladi, Samy Jelassi, Misha Khodak, Zirui Wang, Mengzhou Xia, Yihe Dong, Haoyu Zhao, Danqi Chen, Tri Dao, and Benjamin Eysenbach for discussions, suggestions, and proof-reading at various stages of the paper.
For more detailed discussion, see Appendix B.
Extending our findings to real-world scenarios is also left for future work. It will require real-world scenarios with precise gradation of SIMPLE and HARD examples with respect to underlying abstract concepts. Our work suggests that the brittleness of VLMs could be mitigated by training them to create very detailed descriptions of the scene (and this capability could be internalized for faster inference).
Utility to real-world benchmarks:
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. It primarily is a basic scientific ex- ploration of the capabilities of Vision Language Models. It may lead to the development of better VLMSs, but we do not anticipate any negative societal impact.
We note that training even on our synthetically created datasets seems useful for improving the performance of VLMs in real-world settings. Specifically, including our synthetic datasets during pretraining of VLMs yielded significant improvements across different benchmarks (Table 9 in Appendix G.6). For example, including SIMPLE and HARD Image supervision examples from all synthetic datasets can improve performance on MMMU (Yue et al., 2024) by at least 3%p. Similarly, on a chart dataset (Wang et al., 2024c), including our synthetic datasets can improve
9

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
References
Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 7096- 7116, Online, November 2020. Association for Computa- tional Linguistics. doi: 10.18653/v 1/26603///0022mmmlml.. main. 576. URL https::/p///. / /aap:////ala/cc/cco/c antho ogy .rgg 2020. emnlp main. 576/.
Abbe, E., Bengio, S., Lotfi, A., and Rizk, K. Generalization on the unseen, logic reasoning and degree curriculum. Journal of Machine Learning Research, 15-381) 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L.,, Aschenbrenner, L.,, Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023.
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Agrawal, A., Batra, D., and Parikh, D. Analyzing the behavior of visual question answering models. In Su, J., Duh, K., and Carreras, X. (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1955-1960, Austin, Texas, November 2016. Association for Computational Lin- guistics. doi: 10. 10.11863//1/D11--111 //aclanthology. hwwpttthooogyooooggrooo wwg//oo- 1203/.
Carbune, V., Mansoor, H., Liu, F., Aralikatte, R., Baech- ler, G., Chen, J., and Sharma, A. Chart-based reasoning: Transferring capabilities from L.MSs to VLMs. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 989-1004, Mcxico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 1188633//1/2224.indiiggg. 62. URL https: attts:://aclatteee Findings-nase 62/.
URL https:
Agrawal, P. Antoniak, S., Hanna, E. B., Bout, B., Chap- lot, D., Chudnovsky, J., Costa, D., De Monicault, B., Garg, S., Gervet, T. et al. Pixtral 12b. arXiv preprint 2rXiv:2410.07073, 2024.
org/
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:/5044 00325, 2015.
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. Exploring length generalization in large language models. Advances in Neural Information Pro- cessing Systems, 35:38546-38556, 2022.
Cherian, A., Peng, K.-C,, Lohit, S., Smith, K. A., and Tenen- baum, J. B. Are deep neural networks smarter than second graders? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10834- 10844, 2023.
Anthropic. Claude 3.5 sonnet, June 2024. URL https: //www. www.aanthrriiio. ww..cooccccoooomoooooc wtwp////////////cc/////at..tceet sonnet. Arti- ficial intelligence model.
Deng. J., Dong, W., Socher, R., Li, L.JJ,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 255,255, 2009. doi: 1A1100/CVR.22000555000
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015.
Deng, Y., Choi, Y., and Shieber, S. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14833 2024.
Bai, S., Chen, K., Liu, X., Wang. J., Ge, W., Song. S., Dang, K., Wang. P., Wang, S., Tang, J., Zhong, H., Zhu, Y., Yang, M., Li, Z., Wan, J., Wang. P., Ding, W., Fu, Z., Xu, Y., Ye, J., Zhang, X., Xie, T., Cheng, Z., Zhang, H., Yang, Z., Xu, H., and Lin, J. Qwen2.5-vl technical report, 2025. URL https:: // arxiv. org/ abs 20los/22502..22 .19.2...199.3 13923.
Duan, H., Yang, J., Qiao, Y., Fang. X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluat- ing large multi-modality models, 2024.
Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lilli- crap, T. Measuring abstract reasoning in neural networks. In Proceedings of the 35th International Conference on Machine Learning, pp. 511-520. PMLR, 2018.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A.. Al-Dahle. A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The llama 3 herd of models. arXiv preprint arXiv: 2:X.v:2407.21780 2024.
Bhattamishra, S., Ahuja, K., and Goyal, N. On the ability and limitations transformers to recognize formal lan- guages. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.),
Dziri, N., Lu, X., Sclar, M., L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al.
10

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Faith and fate: Limits of transformers compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Hill, F., Santoro, A., Barrett, D., Morcos, A., and Lillicrap, T. Learning make analogies by contrasting abstract rela- tional structure. In International Conference Learning Representations, 2019.
Engstrom, I., Feldmann, A., and Madry, A. Dsdm: Model- aware dataset selection with datamodels. In Forty- first International Conference Machine Learning, 2024. URL 1d=GCBHkKeH@s..
Hsich, C.-Y., Zhang. J., Ma, Z., Kembhavi, A., and Kr- ishna, R. Sugarcrepe: Fixing hackable benchmarks for vision- languaee compositionality. Advances in neural information processing systems, 36, 2024.
wwwpeereeeeeeeeeeorrr..... nerrforum?
Fan, W.-C., Chen, Y.C.,, Liu, M., Yuan, L., and Sigal, L. On pre-training of multimodal language models customized for chart understanding. arXiv preprint MxXiv:2407.14500 2024.
Huang, K., Sun, K., Xic, E., Li, Z. and Liu, X. T2i- compbench: comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neu- ral Information Processing Systems, 36:78723-78747, 2023.
Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 20029- 20038, June 2023.
Huang, Y., Lin, J., Zhou, C., Yang, H., and Huang, L. Modality competition: What makes joint training of multi- modal network fail in deep learning? (Provably). In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, (eds.), Proceedings of the 39th In- ternational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9226-9259 PMLR, 17-23 Jul 2022.
Fan, Y., Du, Y., Ramchandran, K., and Lee, K. Looped transformers for length generalization. In The Thirteenth International Conference Learning Representations, net. forum?
2025. URL
Jelassi, S., d' Ascoli, S., Domingo Ennich, C.,, Wu, Y., Li, Y., and Charton, F. Length generalization in arithmetic transformers. arXiv preprint arXiv:2306. 15400, 2023.
id=2edi gk8yoU.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang. J., Zheng, X., Li, K., Sun, X., and Ji, R. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, June 2023. URL https::////// arxiv. org/ /ats//b//// 2306 13394.
Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Kil, J., Tavazoee, F., Kang, D., and Kim, J.-K. II-MMR: Identifying and improving multi-modal multi-hop reason- ing in visual question answering. In Ku, L.- Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 10698- 10709, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18533/v/2024. findings- acl.ngs--cll666... URI. https: a anthology. org 222422242222 f....ffininngg acl. 636/.
Fu, D., Guo, R., Khalighinejad, G., Liu, O., Dhingra, B., Yogatama, D., Jia, R., and Neiswanger, W. Isobench: Benchmarking multimodal foundation models on isomor- phic representations. In First Conference on Language Modeling, 2024. URL hitps: /ttpps:///ppee http://pppeeeeeeeeeeeeee 2orummi1-=221EErRRJ
Gao, T., Wettig, A., Yen, H., and Chen, D. How to train long- context language models (effectively). arXiv preprint 1XXi:2440.0026600 2024.
Kim, D., Lee, J., Park, J., and Seo, M. How language models extrapolate outside the training data: A case study in textualized gridworld. arXiv preprint a:Xiv:2406.15275. 2024.
Ghosal, D., Han, V. T. Y., Ken, C. Y., and Poria, S. Are language models puzzle prodigies? algorithmic puzzles unveil serious challenges in multimodal reasoning. arXiv preprint arXiv:2403.0 038664 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In The Third International Conference for Learning Representations, 2015.
Goyal, Y., Khot, T., Summers-Stay. D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
2017.

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., and Papail- iopoulos, D. Teaching arithmetic to small transformers. In The Twelfth International Conference on Learning Representations, 2024. URL https: /oppnnevveee. net/ forum? i URS.suRRRbbSS.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976- 11986, 2022.
McLeish, S., Schwarzschild, A., and Goldstein, T. Bench- marking chatgpt on algorithmic reasoning, arXiv preprint arXiv:2404.03441, 2024.
Li, Y. and McClelland, J. Representations and computations in transformers that support generalization on structured tasks. Transactions on Machine Learning Research, 2023.
Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., al. Prioritized training points that are learnable, worth learning, and not yet learnt. In In- ternational Conference Machine Learning, pp. 15630- 15649. PMLR, 2022.
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, X. Z., and Wen, J.-R. Evaluating object hallucination in large vision- language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Liang, P. P., Wu, P., Ziyin, I.., Morency, I..P., and Salakhut- dinov, R. Cross-modal generalization: Learning in low resource modalities via meta-alignment. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 2680-2689, 2021.
Monajatipoor, M.. Li, L. H., Rouhsedaghat, M.. Yang, L., and Chang, K.-W. MetaVL: Transferring in-context learning ability from language models to vision-language models. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 495-508, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 10.18663l11/2023.aclhhhhh
Lin, X., Wang, S., Cai, R., Liu, Y., Fu, Y., Tang, W., Yu, Z., and Kot, A. Suppress and rebalance: Towards generalized multi-modal face anti-spoofing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 211 211-221, 2024.
Liu, B., Ash, J. T., Goel, S., Krishnamurthyy A., and Zhang, C. Transformers learn shorteuts to automata. The Eleventh International Conference on Learning Represen- tations, 2023a. URL hitps: /tttpp:://pppreeeeee opw.revien.eeeeep/e//p/ee nnt// forum? ? idd 2d-De4FYgjFuee
Nesterov, Y. Lectures on convex optimization. Springer Optimization and Its Applications, 137, 2018.
Nguyen, C.,V Le, T--.., Mai, A.-T., and Le, D.-T. Ada2i:
Enhancing modality balance for multimodal conversa- tional emotion recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 9330- 9339, 2024.
Liu, H., Li, C., Li, Y., and Lee, Y.J. Improved baselines with visual instruction tuning. In NeurlPS 2023 Workshop Instruction Tuning and Instruction Following, 2023b.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. In Teirty-seventh Conference on Neural Information Processing Systems, 2023c.
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. Trak: Attributing model behavior at scale. In In- ternational Conference on Machine Learning (ICML), 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a.
Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8238-8247, 2022.
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., and Bai, X. Ocr- bench: on the hidden mystery of oer in large multi- modal models. Science China Information Sciences, 67(12), December 2024b. ISSN 1869-1919. doi: 10. 010007/5114322-024444333333 URL http:///p// / dox. org/10. /1t.0//////110111
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763.. PMLR, 2021.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pp. 216-233 Springer, 2025.
Rahmanzadehgervi. P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vi- sion, pp. 18-34, 2024.
12

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Pro- ceedings of the 26th ACM SIGKDD International Con- ference on Knowledge Discovery& & Data Mining, KDD '20, pp. 3505-3506, New York, NY, USA, 2020. Associa- tion for Computing Machinery. ISBN 9781450379984. doi: 10. A..1444/394448634000660 URL https: /doi.. org 10. 1145/ /0/44/33344488 3406703.
emnlp-main. URL https:://aclaaaeeee 1ttg/2220.emllp-110 162/.
Taylor, A. K., Cuturrufo, A., Yathish, V., Ma, M. D., and Wang, W. Are large-language models graph algorithmic reasoners? arXiv preprint arXiv:2410.22597 2024.
Thrush, T., Jiang, R., Bartolo, M.. Singh, A., Williams, A., Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
Sanford, C., Fatemi, B., Hall, E.,, Tsitsulin, A., Kazemi, M., Halcrow, J., Perozzi, B., and Mirrokni, V. Understand- ing transformer reasoning capabilities via graph algo- rithms. In The ehirty- Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: / //openreviei www.eeee/oorre forum? -dd ifddddddbddwssss.
Tong, S., II, E. L. B., Wu, P., Woo, S., IYER, A. J., Akula, S. C., Yang, S., Yang. J., Middepogu, M., Wang, Z., Pan, X., Fergus, R., LeCun, Y., and Xie, S. Cambrian-l: A fully open, vision-centric exploration of multimodal LLMs. In The Thirty-eighth Annual Conference Neu- ral Information Processing Systems, 2024. URL https: // //openreview. net for Pent-id-iiie -i8AA-VVV......
Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Zhao, Y., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., Catanzaro, B., Tao, A., Kautz, J., Yu, Z., and Liu, G. Eagle: Exploring the design space for multi- modal LLMS with mixture of encoders. The Thirteenth International Conference on Learning Representations, 2025. URL https:: : happps://pppenee http///pppeweeeeeee/e/pee n forum? 1d=Y2RW9EVwhT.
Wang, H., Feng, S., He, T., Tan, Z., Han, X., and Tsvetkov, Y. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024a.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
Wang. J., Ming, Y., Shi, Z., Vineet, V., Wang, X., Li, Y.. and Joshi, N. Is a picture worth thousand words? delv- ing into spatial reasoning for vision language models. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024b. URL https: / / openreview net/
Socher, R., Ganjoo, M., Manning, D., and Ng. A. Zero- shot learning through cross modal transfer. Advances in neural information processing systems, 26, 2013.
Wang. W. Tran, D., and Feiszli, M. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12695-12705, 2020.
Su, D., Sukhbaatar, S., Rabbat, M., Tian, Y., and Zheng, Q. Dualformer: Controllable fast and slow thinking by leam- ing with randomized reasoning traces. In The Thirteenth International Conference on Learning Representations, 2025. URL https: www.r/ewwreeeeeeeooo/e/e. n forum? id=bmbRCRiNDu.
Wang, Z., Xia, M., He, L.., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., Chevalier, A., Arora, S., and Chen, D. Charxiv: Charting gaps in realistic chart understanding in multimodal LLMs. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024c. URL https: /ooeeneview net/ forum? wwc.ddccyym0oom QYae.
Sun, 7., Yu, I.., Shen, Y. Liu, W., Yang, Y., Welleck, S., and Gan, C. Easy-to-hand generalization: Scalable alignment beyond human supervision. In The eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: wpwp//vien.eeeeeeee/ee weee/forrm id=qwgfh2fTeN.
Wei, Y., Feng, R., Wang, Z., and Hu, D. Enhancing multi- modal cooperation via sample-level modality valuation. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 27338- 27347, June 2024.
Tan, H. and Bansal, M. Vokenization: Improving language understanding with contextualized, visual-grounded su- pervision. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.). Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 2066-2080, Online, November 2020. Association for Computational Linguistics. doi: 10. N0.18865/1//20000
Weiss, G., Goldberg, Y., and Yahav, E. Thinking like trans- formers. In International Conference on Machine Learn- ing, pp. 11080- 11000. PMLR, 2021.
13

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Wu, Q.. Zhao, H., Saxon, M., Bui, T., Wang. W. Y., Zhang, Y., and Chang, S. Vsp: Assessing the dual challenges of perception and reasoning in spatial planning tasks for vlms. arXiv preprint arXiv.2407.0186, 2024a.
Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P.,
Zhou, A., Lu, P., Chang, K.-W., Gao, P., et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403.14624, 2024c.
Wu, X., Yu, D., Huang, Y., Russakovsky, O., and Arora, S. Conceptmix: A compositional image generation bench- mark with controllable difficulty. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024b. URL https: //openreview net/ formm? i M 9
Zhang, X., Li, S., Wu, Z.,, and Shi, N. Lost in transla- tion: When gpt-4v (ision) can't see eye to eye with text. a dision-language conplsstgeecsssseneeeec. analysis of vllms and be- yond. arXiv preprint arXiv:2310. 25200, 2023.
Zhang, X., Li, S., Shi, N., Hauer, B., Wu, Z.,, Kondrak, G., Abdul-Mageed M., and Lakshmanan, L. V. Cross-modal consistency in multimodal large language models. arXiv preprint arXiv:241 1.09273, 2024d.
Xia, M., Malladi, S., Gururangan, S., Arora, S., and Chen, D. LESS: Selecting influential data for targeted instruction tuning. In International Conference Machine Learning (ICMI.) 2024.
Zhang, X., Yoon, J., Bansal, M., and Yao, H. Multimodal representation learning by alternating unimodal adapta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11456-27466, June 2024e.
Xia, Y., Huang, H., Zhu, J., and Zhao, Z. Achieving cross modal generalization with multimodal unified representa- tion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Xie, S. M., Pham, H., Dong, X., Du, N., Liu, H., Lu, Y., Liang, P. S., Le. Q. V., Ma, T., and Yu, A. W. Doremi: Optimizing data mixtures speeds up language model pre- training. Advances in Neural Information Processing Systems, 36, 2024.
Zhang, Z., Wang, X., Zhang, Z., Li, H., Qin, Y., and Zhu, W. Llm4dyg: can large language models solve spatial- temporal problems on dynamic graphs? In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. pp. 3350-43611 2024f.
Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., and Arora, S. SKILL-MIX: a flexible and expandable family of evaluations for Al models. In The Twelfth International Conference on Learning Representations, 2024. URL https///opppereeeeeeeeee fet/forum? id=Jf5qplvq1q.
Zhao, H., Kaur, S., Yu, D., Goyal, A., and Arora, S.
Can models learn skill composition from examples? In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: / //openreview net/ forum? d= d=1 dddsssds...bbm.....
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L.,, Shojanazeri, H., Ott, M., Shleifer, S., Desmai- son, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Expe- riences on scaling fully sharded data parallel, 2023. URL https: /ttp://aavvvv/..//ee/eee org/ abs 22./ 11277.
Yue, X., Ni. Y.. Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W.. Sun, Y., et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J. M., Bengio, S., and Nakkiran, What algorithms can transformers learn? a study in length generalization. In The Twelfth International Conference on Learning Representations, 2024a.
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, When and why vision- language models behave like bags-of ofrss-of-ordds and what do about it? In The Fleventh International Conference on Learning Repre- sentations, 2023.
Zhou, Y., Alon, U., Chen, X., Wang, X., Agarwal, R., and Zhou, D. Transformers can achieve length generaliza- tion but not robustly. arXiv preprint 1929911402.09371. 2024b.
Zhang, J., Huang, J., Jin, S., and Lu, S. laniuaela models for vision tasks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Zhang, I.,, Zhai, X., Zhao, Z., Zong, Y., Wen, X., and Zhao, B. What if the tv was off? examining counterfactual reasoning abilities of multi-modal language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21853-21862 2024b.
14

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Appendix
Table of Contents
Appendix Structure
B Related Works
17
C Details Synthetic Tasks
18
C.I Formal description of data generation
C.2 Consecutive Table Readout
19
Table Readout
C.4 Grid Navigation
20
C,5 Visual Analogy
20
C.6 Issues during synthetic data creation
21
Training Details
E Evaluation Details
25
E.I Evaluation on Consecutive Table Readout and Table Readout
25
E.2 Evaluation on Grid Navigation
25
E.3 Evaluation Visual Analogy
25
F
Consistent Results on Another Model Family and Size
26
F.1
Training Details
................................
26
Evaluation Details
................................
F3 Results
26
Continued Discussion From Main Paper
G.1 Comparisons at equal unique samples for Consecutive Table Readout
28
G.2 Additional setting an S2H getereeeraliziig task
................................
28
G.3 Comparison at equal FLOPs for non S2H-generaliziig tasks
28
G.4 Transferring reasoning from image text
................................
G.5 Text warm- pretraining
30
G.6 Utility our synthetic datasets for existing evaluation benchmarks
H Continued Discussion on Gradient Alignment
32
H.1 Proof of Theorem H.1
................................
32
H.2 Additional measure 1: gradient cosine similarity
................................
33
H.3 Additional measure 2: Adam update alignment
................................
33
H.4 Experimental results
34
1 Additional Ablations
36
L1 Performance of other multimodal models
36
T.2 Ablation of the Mix+ supervision
36
1.3 Ablation of the reasoning alignment phase (Align-)
36
1.4 Ablation of the text warm-up pretraining phase (TW):
36
1.5 Requirement of text representation
37
15

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
1.6
Explicit and implicit text conversion
39
1.7 Explicit and implicit CoT
39
1.8 Multi-task training: jointly training on all three non S2H S2H-seHHHHnee generalizing tasks
41
1.9 Ablation repeated HARD examples
42
J
Interpretability Experiments
43
K Analysis of Failure Modes
45
K.1 Table Readout
45
K.2 Grid Navigation
45
K.3 Visual Analogy
46
A. Appendix Structure
The appendix provides omitted experimental details, additional empirical explorations, and theoretical statements, which we outline below.
Related works: In Appendix B, we provide overview of relevant lines of research in VLM benchmarks and evaluations, modality imbalance, cross-modal transfer of generalization, and simple-to-hate generalization. We highlight the contributions that differentiate our work from the similar ones.
Experimental details: We provide all details of our synthetic data generation in Appendix C. We present our data generation algorithm for creating training data in Appendix C.I, details on Consecutive Table Readout, Table Readout, Visual Analogy, and Grid Navigation in Appendices C.2 to C.5 respectively. We show examples from our training data for each synthetic setting in Figures 32 to 37. We present details on training and evaluation in Appendices D and respectively.
Consistent results on another model family and size: We replicate some experiments from the main paper with Dween2.5-VL-3B-nnstree and 1eeen.25-VLL7B--ssrree We report the results in Appendix F.
Continued discussion from main paper: We continue the discussion in the main paper in Appendix G. We present results on Consecutive Table Readout after normalizing the number of unique samples used across training strategies (Appendix G.1), present results Pattern- Hellout Visual Analogy (Appendix G.2), compare training strategies on non S2H-genera tasks by normalizing the total number of training data used (Appendix G.3), discuss further on transferring reasoning from image to text modality (Appendix G.4), discuss further on text warm-up pretraining (Appendix G.5), and report the utility of our created synthetic datasets for real-world benchmarks (Appendix G.6).
S2H-gener version of Visual Analogy
Continued discussion on gradients: that the gradient alignment score connects to the expected drop in evaluation loss with SGD on training gradients (Theo- rem H. We then propose results on additional measures gradient cosine similarity and Adam update alignment score (Appendix H.3) that better capture the Adam gradient updates used for optimization.
We continue our discussion on gradient alignment in Appendix H. We first show
Ablation studies: We conduct extensive ablation studies measure the effect of each experimental design decision in training strategies on non S2HH-generalizing tasks and report the results in Appendix I. We report the performance other multimodal models on our synthetic data (Appendix I.1). We study design choices in Mix+ (Appendix 1.2). design choices in Align-Mix+ (Appendix I.3), design choices in text warm-up pretraining (Appendix I.4), the effect of the choice of a text representation (Appendix I.5), the effect of text conversion (Appendix L66), the role of thain-of-thougee (Appendix I.7), the effect of multi-task training (Appendix 1.8), and the effect of repeated training examples (Appendix I.9).
Interpretability experiments: We further conduct interpretability experiments on our trained models. We use gradient attribution to track the focus of the model on different image pixels during chain-of-though generation (Appendix J). We also report failure modes of models trained on our synthetic data when evaluated on HARD examples (Appendix K).
16

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
B. Related Works
Benchmarks and evaluations for VLMs VLMs are evaluated on benchmarks such as visual question answering (VQA) (Antol et al., 2015), image captioning (Chen et al., 2015), zero-shot image classification (Deng et al., 2009), and compositional reasoning (Thrush et al., 2022; Yuksekgonul et al., 2023; Hsieh al., 2024). However, these benchmarks often suffer from language bias, allowing solutions to use shortcuts with minimal visual information (Agrawal et al., 2016; Goyal et al., 2017; Zhang et al., 2024c). Although recent work (Rahmanzadchgervi et al., 2024; Wang et al., 2024b; Kil et al., 2024) proposed new benchmarks that aim to evaluate the spatial understanding and reasoning of VLM, most evaluation tasks are in the form of VQA questions that only require "single-hop" reasoning or relatively fewer reasoning steps. To create a controlled setting with well-defined SIMPLE and HARD tasks, we focus on algorithmic visual reasoning tasks. These tasks allow us to precisely control the number of steps in the step-by-step reasoning process and the level of dynamic interaction between textual and visual inputs. Closely related works have explored graph-based algorithmic reasoning LLMs (Taylor al., 2024; McLeish al., 2024; Zhang et al., 2024f; Wang et al., 2024a; Sanford et al., 2024) but such studies remain limited for VLMs.
Modality imbalance Studies have shown that models exhibit different learning capabilities and learning speed on multimodal inputs (Wang et al., 2020; Nguyen et al., 2024). The imbalanced contribution of individual modality to the final prediction can result in overreliance on a few dominant, optimized modalities, while underutilizing signals of the weak ones, Peng et al. (2022) and Lin et al. (2024) attempt to rebalance the convergence speed of all modalities by modulating the learning rate or gradients. Fan et al. (2023) propose a representative embedding to guide the slow-learning modality and regularize the fast-learning one. Zhang et al. (2024c) propose an alternating unimodal training to minimize interference between modalities. Despite their success in traditional multimodal joint training, it remains challenging repeat the same for adapter-but adapter-based VLMs due to significant differences in architecture and training pipeline. Our work aims to address this issue specifically for VLMS from the perspective of transferring the strong learning behaviors from the dominant modality (text) to the weak one (image).
Generalization transfer between input modes Given the high cost of training VLMs from scratch, recent research on adapter-based VLMS has been driven primarily by the idea of leveraging pretrained LLM backbones. The success of this approach is built on the idea of cross- cross-modal generalization, which enables the model to harness information from the auxiliary modality (e.g. text) to improve unimodal task on the primary modality (e.g. image classification). This knowledge transference has been exploited for both small-scale multimodal models (Socher et al., 2013; Liang et al., 2021; Tan & Bansal, 2020) and more recent VLMs (Monajatipoor al., 2023; Carbune et al., 2024; Zhang et al., 2024a). However, existing works often require explicit alignment of the modality, such as learning unified representation using contrastive learming (Xia et al., 2023), for models to transfer knowledge across modalities. The cost of curating a large, perfectly aligned multimodal dataset to learn the modality alignment becomes expensive as the model size increases. In our work, we find that transfer of generalization across input modes naturally emerges from auto-regressive training.
S2H generalization Recent studies have explored simple-to-hun generalization in LLMs, with a focus on length generalization transformers. These works evaluate models on tasks requiring longer computations than those seen during training, using synthetic datasets like parity, Dyck-1 languages, decimal addition, structural recursion, and finite state automata (Anil et al., 2022; Lee et al., 2024; Jelassi et al., 2023; Li & McClelland, 2023; Kazemnejad al., 2024; Liu et al., 2023a; Abbe et al., 2024; Bhattamishra et al., 2020; Zhou et al., 2024b; Fan et al., 2025). Zhou et al. (2024a) connect length generalization to the RASP programming language (Weiss et al., 2021), offering a unified perspective. Sun et al. (2024) recently propose easy-to-hard generalization to measure generalizable verification for math and code datasets. OOD generalization beyond human supervision remains an important open question for the advancement of current Al models (Burns et al., 2023).
17

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table Summary of SIMPLE and HARD task setup for Table Readout, Grid Navigation, and Visual Analogy
Setting
Attribute
SIMPLE
HARD
Mean Length
12
35
# Turns
1-4
>
Table Readout
Spiral Sinusoidal
Composition of Spiral / Sinusoidal
Pattern
# DFS steps
[10, 25]
[26, 60]
Grid Navigation
# Objects
[1.2] {1}
(2,3, 4,5,
# Obstacle type
{3, 4,
Example Patterns
Same
Different
Visual Analogy
Query Pattern
Seen
Held-out
C. Details on Synthetic Tasks
C.1. Formal description of data generation
In Algorithm 1, we provide the pseudo-code for generating the training data mixture for the main experiments. Below we provide more details in the setup.
C.I.1. WHEN TRAINING ONLY ON SIMPLE EXAMPLES
For Consecutive Table Readout and for any type of supervision among Text, Image, Text+ Image-via Lange-via-Texxe and Mix:
For each unique data E X and for each type of supervision Text, Image, and Image-via -Text, we choose whether to include in the training data, depending on whether these types of supervision are used for training (Section 2.4).
We denote the number of unique data used from XSIMPLEE N SiMPLE:
We compare all training strategies with the total number of training data used, given by:
SIMPLE Number of epochs Nu
SIMPLE Number of types of supervision per input
For a fair comparison, we keep the number of unique data SIMPLE fixed across Text+ Image-via -ix-t and Mix. Then to match NIMMPLE we set the number of epochs to 1.5 for lmxt+ (50% samples are repeated 2x), 3 for image-via-Texx- and for Mix.
Since our result depends heavily on the success of Text and the failure of Image in Consecutive Table Readout, carefully tune the number of training epochs to achieve optimal performance. We conduct ablations where instead of setting N SIMPLE = NSIMPLE, we also try setting No equal to Naonz or NUMEEE (respectively, the number of epochs is set at 2, 3). The results presented in Figure 3 corresponds to Na Nomes SIMPLE
Note Text and Image for Consecutive Table Readout:
for Text and SIMPLE Namn for Image. We discuss further in Appendix G.I.
1.1.2. WHEN ALSO TRAINING HARD EXAMPLES
For Image-via-Tent -Txt++ Mix+ on non S2H-generalizing tasks:
We set NHARD, the number of data from the HARD task, equal to NIMPLE the number of data from the SIMPLE task.
We generate mixture of NSMMPL examples under image-via-Tens Text or Mix. We include NHARd instances of HARD Text.
C.3. REASONING ALIGNMENT (Align-) OR TEXT WARM-UP PRETRAINING ((TW))
When generating data for the reasoning alignment phase (Align-):
We set N 10 and include an equal number of SIMPLE Text and SIMPLE Image-via -Teeet -Text examples.
When generating data for the text warm-up pretraining phase ((TW)):
We set = 1 and include an equal number of SIMPLE Text and HARD Text examples.
After training on (TW) and/or Align-, we continue with the main phase of supervision (e.g., Mix+ for bixg).
18

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Algorithm 1 Data generation pipeline for main experiments
Require:
Task f X + 2, Dataset X' = NSIMLL U NHARD, Number of data to generate N, Type of supervision
i E Text, Image then
Initialize the number data per difficulty NAMMPLE N, NuARo and the number of unique examples Ngiuon: NSMMPL
clse if (Text=Image, Image-via-Teree Mix } then
Initialize the number of data per difficulty NSIMPLE N, NHARD 0 and the number of unique examples NISIPEE NanIn Mix+| } then Initialize the number of data per difficulty NSMMPEE
else { Image-via-Texxt
2 NHARD
N and the number of unique examples NiiMPLE NenuEs
I
else if { [Alig--] then
Initialize the number of data per difficulty NSIMPLE = N, NBARD 0 and the number of unique examples SiMma. NagIa else if E { (TW) then
2
Initialize the number data per difficulty NSIMPL = 2 end
N.
NHARD = N and the number of unique examples NaMMA NIIMPLE
Initialize S = Q. for ( NiiMLL do Sample ~ XaIME.. I E Text, Text+Image, Image-via-Text: Mix, viiage-va--Tex+ Mix+, Align-, (TW) }, then S - Su ( (C)T())
8 E Image, Text++mage, Image-via-Text, Mix, Image-via-Txt++, Mix+ }, then S + S ((x00) (xT(x), f(x)))
I E {
Image-via-Text, Mix, Image-via-Texte Mix+, Align- J, then +- S ([x (9) Peonvert, 20, (x)(())
end for
Determine number of epochs to repeat NaDICE
Randomly shuffle S and repeat it e times (i.c., take the first e S] elements from repeated copies of S) for NuARD do
Sample XuAuu-
S S U " (o)()),
end for Randomly shuffie S and return S.
Figure 10. Pseudo-code for generating data mixture: For ablation studies, the algorithm might be slightly modified.
C.2. Consecutive Table Readout
Given a table with n, rows and ne columns, a start cell (Fa, c.) and an end cell (e), the model is tasked read all numbers between the start cell and end cell following the given rules.
If a < Tes move left-to-right within each row:
Cx), (Fa, Cs + 1),".. (r., (r. +
+ (1,,,) ... (res 1), (re,2), .., Cre
rf > Te, move right-to-left within each row:
(r,, (ra. 1),- (r., 1), (r. - 1, n, (r. 1, n. 1),"" nc), (re, ne 1),, ... (re,
If r. = Te, move from (rz, c.) to (re. ce).
See example images in Figure 1.
C.3. Table Readout
Given table with n,, rows and n. columns (where n,, ne E [8, 12), start cell (r,,c)) an end cell (res ce), and a path of cells P connecting the two cells (without any loops), the task is to read the numbers the path starting from the start cell and ending at the end cell. Each path is continuous and is concatenation of linear segments, where consecutive segments are separated by 90 degree turns. On the SIMPLE task, each path contains - 4 linear segments, following a spiral or sinusoidal pattern, and has an average length 12. On the HARD task, each path contains > 4 linear segments, following a compositional spiral or sinusoidal pattern, and has an average length of 35. See example images in Figures 32 and 33 and an example pseudo-code create the spiral or sinusoidal patterns Algorithms and 3.
19

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Algorithm 3 Sinusoidal Path Generation that changes diree-
tions as right -oot--wwww>>ett----- -leftt>up- popttt-uu--iggee ropp>>ight>>... where down and up movements contain only 2 cells
Algorithm 2 Spiral Path Generation that changes directions as right->down -liho----ltt------riih-----
Require: Table with n,, rows and ne columns, start cell, linear segments
Require:
Table with n, rows and n2 columns, start cell, & linear segments
Initial Reeg =
Initialize current-cell coordinates as start cell coordinates
Initial Fssg =0 0
Initialize diveentdireetion to "right"
Initialize current-cell coordinates as start cell coordinates
Initialize current-direction to "right"
Initialize Path- 4.
Direction- Change ("right"; "leff", "left" : ""right"]
Initialize Path= Φ
Coordinate- Uprantttte--dtcc----ee- ("right" (0, "left" : (0,- -1)
Direction- Direetion-Change = ("right" : "down", "down"
while Nseg # do
"left", "left" : "up", "up": "right"}
* Coordinate-Update
= '"right" : (0, "down"
Add current cell to Path.
(1.0), "left": (0. 1), "up": (-1,0)
Compute temporary-cell by adding coordinate update vector for current-direction from Coordinate- -Upitt to current-cell. if temporary-cell is out of bounds then 1, break
while reeg * k do
Add current cell to Path.
Compute temporary-cell by adding coordinate update vector for current-dite direction from Coordinate- Update to current-cel cell.
noay
Loop twice
Increment column coordinate by 1 in current-cell Add current-cell to Path. Update current-direction using Direction-Change
If temporary--oo is out of bounds, update current- current-direttoo using Direction- Direction-Change and increment Reseg-
Update current-cell by adding coordinate update vector for current-direction from Coordinate-Updae to current-cell. end while
Increment nssg by 2.
end if
Update current-cell by adding coordinate update vector for current-direction from Coordinate- Update current-cell. end while Return Path
Return Path
Figure 11. Pseudo-code for generating spiral and sinusoidal paths on Table Readout: For simplicity, we present a single variant of
each pattern. By permuting the Direction-Change map, the presented variants can he modified to include other direction patterns.
C.4. Grid Navigation
Given a grid with ny- rows and ne columns (where ney, De [8, 12),, a start cell (rs, Cx), an end cell (re, Ce), and set of objects and obstacles placed at various positions within the grid, the task is to find a path from the start cell to the end cell that collects all specified objects while avoiding all obstacles.
For each generated grid, we randomly select several objects from a set of 30 possibilities: heart, crown, flag, star, flower, umbrella. plane, phone, spark, diamond, queen, hammer, club, gear, arrow, sun, bishop, note, coffee, anchor, cloud, pawn, castle, horse, infinity, moon, null, approx, integral, product, and sum. Each chosen object is represented as an Unicode character, as shown in Figure 12. Obstacles are chosen from the following five symbols: dot, cross, square, triangle, and plus. The names and representations of all these symbols- both objects and abstacles-have been verified using GPT-4o.
The SIMPLE task requires the model to collect k E [1, 2] objects spread across the grid, while avoiding a single kind of obstacle. The HARD task requires the model to collect & E [2,5] 5] objects spread across the grid, while avoiding a composition of o E 53 obstacles. The SIMPLE task requires t E [10, 25] DFS steps, while the HARD task requires E [25, 60] DFS steps.
See example images in Figures 34 and 35.
C.5. Visual Analogy
We create multimodal visual analogy dataset based on the Procedurally Generated Matrices (PGM) data proposed in Barrett al. (2018) and Hill al. (2019). Each instance consists of 2 examples of three images, query of two images, and four answer options. Each instance has latent logical relation (XOR, OR, AND, Progression) that will be applied to both the examples and the query. There are also three latent domains di d2, dquery (for each example and the query, respectively), chosen from {lim [line type, 1ine color, shape type, shape_cc co color, shape size, shapehequuee quantity, shape_ shape_posile itionn. For each example i, the value of the domain d, in the third image follows from applying the relation r to the values in the first two images. The task is to choose one of the four options so that there exists a domain duuery where applying the relation r along dquerr in the first two images of the query leads the chosen option.
20

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
J
L
H
8
S
Σ
Figure 12. Details on Grid Navigation: Unicode characters used for specifying each object.
Table List of all possible attribute values for each domain in Visual Analogy: We reproduce Hill et al. (2019) with slight modifications. The diverse combination of the attribute values results in high complexity of this task, testing various both OOD and compositional generalizability of the model to great extent.
[falling diagonal line, rising diagonal line, horizontal line, vertical line, diamond lines, eircular line, V-shape facing up, V-shape facing left
line type
V-shape facing down, V-shape facing right)
line color
{0 (black), 90 (dark grey); 135 (grey), 189 (light grey))
shape type
{circle, rectangle, triangle, pentagon, hexagon)
shape color
{0 (black), 90 (dark grey), 135 (grey), 189 (light grey), 255 (white))
shape size
{20, 27, 34, 41]
shape quantity
6, {(0,0), (0, 1), (0, (1, (1, (1,2), (1,2) (2,0), (2,,))
shape position
Note that following Hill et (2019), we exclude all spurious correlations of the examples and query such that they follow exactly one pattern (d,r), Furthermore, we create three nontrivial confounding options such that each of them, when combined with the query images, is consistent with exactly
pattern (doption : Foption.) where Toption, Fquery-
We also reserve held-out set of combinations S {(d that does not appear in the training images. On the SIMPLE task, dj = dg = dquery and the query pattern (dquery. (Qquery.qqurry) is never chosen from the held-out set. On the HARD task, d1, d2, dquery are distinct and both (d,, rs) and (dquery lquery) are always chosen from the held-out set S.
See example images in Figures 36 and 37 and the complete list of all possible attribute values in Table 2.
C.6. Issues during synthetic data creation
Here, we outline the primary issues that we faced while creating the synthetic datasets, which might be of value to the general community.
C.6.1. Consecutive Table Readout, Table Readout
The primary issue that we faced during creation of these datasets were as follows:
Resolution issues: For images, we found that representing numbers as their English names (e.g. 9 represented as NINE) improved the OCR performance substantially. When represented numerics, the model often confused between pairs (7, 9), and (0, 8). These issues were largely mitigated by replacing numerics with English names.
Color: The model's S2H generalization can vary drastically depending on the color used to highlight the cells. HARD (15-20) images, the performance of the model trained with Image supervision can vary from 30% 70% depending which color (e.g. purple yellow) used.
CoT Trace: Our original CoT Trace simply outlined the numbers on the path, without any mention of the row number, column number, row name, and column name of the cells in the highlighted path. This resulted poor performance of
21

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
the model when trained with images. We then switched to more verbose CoT, where the model was provided with the above details each step of traversing the highlighted path, and the model's performance substantially improved.
For Consecutive Table Readout, we find that the verbose CoT trace shows S2H generalization, and not the final solution that the model reports. Hence, we report our evaluation performance for Consecutive Table Readout on the CoT trace.
C.6.2. Grid Navigation
The major issue that we faced in Grid Navigation was the design of chain-of of-thought reasoning steps to represent the Depth First Search trace. At multiple points, we found that current VLMs are fragile to read image inputs, and our CoT trace needed to be very explicit to train the model effectively on SIMPLE examples.
An initial version of Grid Navigation: In our first version, we designed an extremely simple dataset, where the grids only had source cell, destination cell, and few cells marked by red color that represented obstacles.
Models failed to train on image-input without verbose details in CoT: following at each DFS step: "[current cell]: [proposed next action]" without iterating through all invalid actions considered before proposing this action. c.g., a 3-step DFS step would look as follows:
Our initial CoT would only provide the
(1, 1): right
- (1, 2): down
(2, 2): backtrack
where we don't explain why we need to "backtrack" at (2, 2). This made the model learn the following:
answer formatting
2. knowing how to retrieve the current location (row, col index) and the destination location
knowing which action is preferred (the one that minimizes the distance towards destination)
but the model never picked up on why we sometimes backtrack sometimes take an action that is not the most preferred. At generation, it would ignore all obstacles and try to take the most preferred action.
On the other hand, we observed that the model could still recognize the reasoning for "backtracking" text input and could get 100% accuracy SIMPLE text for Text supervision, and also 100% accuracy on SIMPLE images for Mix supervision. Thus, for cases where the model couldn't 't train with image-inpon but could train with text input, Mix was useful to train the model even for improving accuracy on in-domain examples. However, this setting was slightly different from our S2H generalization view, and so we decided to make the CoT more verbose.
We iterate through all possible actions at each state, giving reasons why that action is valid/ invalid. c.g. a 3-step DFS trace that starts from cell (1, I) will look as follows
In later attempts, we switched to more verbose CoT:
Current cell: (1, 1): right would lead to (1, 2) which is available and not visited yet, so we can move right.
Current cell: (1, 2): down would lead to (2, 2) which is available and not visited yet, so we can move down
- Current cell: (2, 2): right would lead to (2, 3) but has an obstacle; down would lead to (3, 2) but it has an
obstacle; left would lead to (2, 1) but has an obstacle; we have no more action left, so backtrack
The model now gets almost perfect S2H generalization on both text / image matter which supervision we give. So we couldn't it really compare the performance of different types of supervision. This was because once the model learns how to iterate through different actions and determine its validity, length generalization was trivial.
Thus, we switched our current version of Grid Navigation, where the task additionally involved spatial reasoning of different combinations of objects and obstacles spread across the grid.
C.6.3. Visual Analogy
Here, the main challenge is to recreate the Procedurally Generated Matrices (PGM) dataset first introduced in Hill et al. (2019) and Barrett et al. (2018), as the data generation code is not publicly available. Therefore, we try our best to recreate the data set with slight adaptations. Specifically, we have 10 variations in the attribute values for line_t chape-quant ity, and shape posi t on as in the original paper. For the rest of attributes 1 i inn color, shape type, and shape s size,
22

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
only include < 5 variations of attribute values. Meanwhile, the original papers do list all the attribute values used in the original data generation, values based on the consideration that they are clearly differentiable from human perspective.
was the source code publicly available, decide upon the list of possible attribute
The original papers claim that solving PGM puzzles is challenging vision task. While we acknowledge that our recreated version of the data reduces the complexity compared to the original version, we note that our adaptations do not qualitatively change the challenging nature this task. As mentioned Barrett et al. (2018), the challenge of effective knowledge composition comes mainly from the necessity to represent abstract logical rules discrete symbolic explanations. They show that training with auxiliary information of meta-targets vectors that encode the relation, object, and attribute types as binary string significantly helps abstract reasoning performance, and in particular, in terms of compositional generalization. Our text representations are inspired by the construction of the meta-targets vectors with many tweaks fit into the context length of the model. We observe that by including the discrete representation of knowledge in the form of Image-via -Txt supervision, Mix and Mix+ show a much better S2H generalization on image input, which aligns with previous observations Barrett et al. (2018).
23

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
D. Training Details
We first prepare Bigle-X2-LLama---e a variation Eagle-X5-8B (Shi et al., 2025). We choose Caamaa3-B-Instrucc (Dubey al., 2024) as the LLM backbone for its good reasoning capability. We choose CLIPP-488 (Radford et al., 2021) and ConvNeXt (Liu et al., 2022) as the visual encoders because previous works show that combining the two leads significant improvement, whereas any additional visual encoder leads to marginal improvement (Shi et al., 2025).
At the beginning of the project, the codebase released by Shi et al. (2025) was incomplete. To incorporate the Llama3-8B model architecture and the tokenizer, we adapt the codebase from Tong et al. (2024).
We use the same 595k pretraining data from Liu et al. (2023b) and 1.8M finetuning (visual instruction tuning) data from Shi et al. (2025). We use Deepspeed ZeRO Stage (Rasley et al., 2020) for a Distributed Data Parallel (DDP) training on GPUs on a HPC Cluster. We use the AdamW optimizer with no weight decay (i.e., equivalent to Adam), learning rate schedule with linear warmup of 0.03 and cosine decay zero. We truncate the trail of any text that exceeds the maximum number of text tokens (2048). During pretraining, only the adapter is trained, whereas in all other stages of training, all weights in the model are unfrozen.
With this Eagle-X2-Llama--2e as the base model, we then continuously finetune it on different data mixtures across our synthetic tasks. In Table 3, we report some key hyperparameters.
Table Hyperparameter settings: For all values not reported here, we use the same values as in Shi et al. (2025).
Batch Size
LR
Epochs
Total Data
Max Text Tokens
Pretraining
256
le-3
595k
2048
Finetuning
128
2e-5
1
1,809k
2048
Finetuning on Task
128
2e-5
experiment-specific
2048
24

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
E. Evaluation Details
We extend the VLMEvalKit (Duan et al., 2024) to evaluate the finetuned Eagle- X2-LLama3-88 on held-out data. For generation, we apply greedy decoding and generate up to 2048 tokens.
E.1. Evaluation on Consecutive Table Readout and Table Readout
(x) visits each cell sequentially on the path, by giving the row and column index, row and column names, and the value in the cell (see Figures 32 and 33). The final answer fxx gives the list of numbers again, and also sum of the numbers. We evaluate by simply checking whether the list of numbers are correct. Furthermore, because this list of numbers can be extracted from both the final answer and also the CoT, we report the best performance out of the two, On Consecutive Table Readout, find that we get the best performance on HARD examples by extracting the numbers from CoT. On the other hand, for Table Readout, there isn't t much difference between extracting numbers from CoT and extracting them from the final answer.
E.2. Evaluation on Grid Navigation
(x) records the sequence of visited cells during a depth-first search (DFS) from the start to the end cell. At each visited cell, the trace includes full description of neighboring cells and whether they are available for the next step. The DFS algorithm always prefers directions that minimize the distance towards the nearest uncollected object, or the destination (if all objects are collected). If no directions are possible, we backtrack to the most previously visited cell. The final answer (x) simplified sequence of directions (left, right, up, down) that connect the start and destination cells, where all backtrack movements are removed from the stack (see Figures 34 and 35). We evaluate by simulating the movements in the sequence returned by the model and checking if we arrive the destination after collecting all objects and avoiding obstacles.
E.3. Evaluation on Visual Analogy
(x) enumerates all the values of the tasks-relevant attributes for each panel with the conclusion of whether there exists logical pattern among those values for each attribute domain in the examples. The trace includes summary sentence of what (domain, relation) pattern the two examples demonstrate. After that, the trace performs the same enumeration process for the query panels. It then looks at the options and checks whether it is consistent with the desired relation given the attribute values in the query panels. The final answer fxx identifies the pattern in the form of (domain, relation) (e.g. (line type, XOR) for all examples and the query combined with each option, as well as the final answer of the correct option, The evaluation checks whether the identified patterns and the final answer are correct.
25

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
F. Consistent Results on Another Model Family and Size
F.1 Training Details
We take Qwen2.5-VL-2 mixtures across our synthetic tasks.
and 7B-Instruct (Bai et al., 2025) as the base model and finetune it with on different data
We use the SFTTrainer class in the tzl package. We employ FSDP (Zhao et al., 2023) for training on GPUs on a HPC Cluster. We use the AdamW optimizer with no weight decay (i.e., equivalent Adam), learning rate schedule with linear warmup of 0.03 and cosine decay to zero.
Due to a deficiency in the trl package, we slightly modify Algorithm 1 to ensure that each gradient computation (before gradient accumulation) includes a training example with x(9 unless we train exclusively on x(") (while also maintaining randomness in the data). Instead of concatenating and randomly shuffling the entire dataset, we first shuffle the examples within each supervision, then interleave the individually shuffled data. For example, to construct Mix+, we first shuffle Text, Image, Image-via viaee--va-exx,,, Text, and HARD- Text individually, then construct the final dataset by repeatedly taking the next examples from (Text, Image, Image-via- -ext, HARD- Text, HARD- Text, HARD-Teke respectively.
In Table 4, report some key hyperparameters. Note that Bai et al. (2025) do not report the hyperparameters for their internal training, so we used the hyperparameters for Eagle-X2-Llama- as closely as possible. However, we noticed that for Sive2.5-VI--B: training on Grid Navigation or Visual Analogy with learning rate of 2e-5 often broke the model (e.g.. model starts outputting Chinese tokens), so we had to adjust the learning rate to 5e-6 or 2e-6.
Table Hyperparameter settings for Qwen2.5-VL.
Model Size
Task
Batch Size
LR
Epochs
Total # Data
3B
All
128
2c-5
experiment-specific
7B
Consecutive Table Readout
128
2e-5
experiment-specific
7B
Table Readout
128
2e-5
experiment-specific
7B
Grid Navigation
128
2c-6, 5e-6
experiment-specte
7B
Visual Analogy
128
2e-6, 5e-6
experiment-specific
F.2 Evaluation Details
We extend the VLMEvalKit (Duan et al., 2024) to evaluate the finetuned Qwenn.5-VL-33--nnntte and 7B 1B-Instruct on the same held-out data as used for the main experiments. For generation, we apply the default setting for the Qwen2.5-VE VL family (top p=0.001 and temperature=0000) and generate up to 2048 tokens. We set the maximum number of pixels to be 1280 x 28 x 28.
F.3. Results
In Tables 5 and 6, we report the 22H-generalization on image for most supervision types we consider.
For Consecutive Table Readout, we find that Qwen2.5-VL. VL. (both 3B and 7B models) completely fail to solve HARD-text examples even when HARD Text is a part of the training data (e.g., Mix+). For this reason, we relax the definition of hardness and instead train with MEDIUM-text (if applicable) and evaluate on MEDIUM-tes and MEDIUM-image (see Section for the definitions of MEDIUM and HARD). Even then, @wen2.5-VZ VL-3BB-Insttru fail to solve MEDIUM-text examples even when it is explicitly trained with MEDIUM Text. Therefore, none of our proposed methods can improve -2nngnnerliization on image. However, we find that even though Qwen2.5- ---.2-5--V-----nstseee different models can (Mix+ and Mixg)- successfully improve the S2H-generalizatioo on image.
does not S2H--eneralize on text (which is understandable since on different tasks), our proposed supervision types for non-S2H generalizing tasks
For the other tasks (Table Readout, Grid Navigation, and Visual Analogy), we generally observe a consistent result from the main text: 1) the models do not S2H- -eneeenrallize on either text or image; 2) Mix+ improves -2e-generalization on image by transferring the injected reasoning on HARD-text; 3) Align-Mix+ further improves this generalization. Note that for Grid Navigation, and Visual Analogy on 1-t.----------sttrreee we report the best result between the two learning rates (2e-6, 5e-6).
26

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table Results Qwen2.5-VL-3B-Imstruee For Text supervision. we evaluate on HARD-text, for all other supervision types, we evaluate on HARD-image. For Consecutive Table Readout, we train with MEDIUM Text (if applicable) and evaluate on MEDIUM-image -mmae..
Consecutive Table Readou
Table Readout
Grid Navigation
Visual Analogy
Supervision
30k
30k
60k
30k
60k
30k
60k
Text (eval on HARDD-texi
3
8
11
0
15
0
0
Image
11
10
22
22
0
Text+Image
7
6
14
Image-via-Text
8
13
Mix
14
Image-via-Text+
81
90
67
58
48
Mix+
78
86
77
91
20
27
Align Mix+
66
91
80
91
38
42
Table 6. Results for Swen2.5-VV--7B-Insttee For Text supervision, we evaluate on HARD-text, but for all other supervision types, evaluate on HARD-image. For Consecutive Table Readout, we train with MEDIUM Text (if applicable) and evaluate on MEDIUM-image
Consecutive Table Readout
Table Readout
Grid Navigation
Visual Analogy
Supervision
30k
30k
60k
30k
60k
30k
60k
Text (eval on HARD-text)
22
2
15
18
Image
17
14
29
Text+Image
8
5
11
Image-via-Text
36
9
13
13
18
Mix
52
8
17
15
12
0
0
mnage-via-Text+
73
82
88
75
67
41
44
Mix+
72
13
66
69
85
12
17
Ali+
93
92
36
58
25
34
27

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
G. Continued Discussion From Main Paper
G.1. Comparisons at equal unique samples for Consecutive Table Readout
In Figure 3, we compare Text, Image, and Mix under the same NIMPLE: the total number of training data. Note that Mix is trained for only a single epoch, while the reported results for Text and Image are based on and epochs of training, respectively. We make these choices because, for Text, the S2H generalization performance peaks epochs and then declines sharply, whereas for Image, the S2H generalization performance As an illustrative example, Figure 13 shows the performance of Text and Image when N SIMPLE the number of unique samples, is fixed at x 10 Consequently, in Figure 14, we revisit the results of Figure 3, this time explicitly indicating the number of unique samples N NiMMll used.
slight improvement between 2 and 3 epochs.
MEDIUM
HARD
Text Supervision
Image Supervision
1.0
S2H Accuracy
0.5
1.0
1.5
2.0
2.5
3.0
1.0
1.5
2.0
2.5
3.0
Number of epochs SIMPLE 40k)
Figure 13. Ablation on the number of epochs Consecutive Table Readout: We measure the S2H generalization performance of Text on HARD-text and Image on HARD-image with multi-epoch training, when Vi is fixed as 4 10 We observe that the generalization performance of Text supervision peaks 2 epoch training, after which drastically drops, while the generalization performance of Image supervision increases slightly between and epochs of training.
G.2. Additional setting for an S2H-generalizing task
Here, we consider Pattern- Pattem-Heldout Visual Analogy- - a geneereerl version of Visual Analogy - by defining an alternative version of HARD examples. We keep the definition of SIMPLE examples from Section and Appendix C.5, but modify HARD instances to only measure analogical reasoning on held-out reasoning patterns, without requiring the domain to be different across the in-context examples.
That is, let dı, d2, dquery denote the latent domains of the examples and the query, denote the latent logical relation to be applied on the latent domains, and S denote held-out set of combinations (d,r). r). The SIMPL.F task contains puzzles where di = d2 = dquery and (dquery, Tquery) & S, whereas the HARD task contains puzzles where dj d2 dquery and (diuerry Fquery) E S. Note that Visual Analogy, we had additionally required d1, d2, dquerry to be distinct for HARD puzzles.
In Table 7, we compare the S2H generalization performance of Text, Image, and Mix supervision on Puttern-Heeldot Visual Analogy. The model learns the task more easily on text than on image: while the image S2H generalization for Image supervision is bounded by 32%, the text S2H generalization for Text supervision can reach 49% when trained 24 x 10a data.
On the other hand, Mix supervision can transfer the S2H generalization from text to image and improve the performance on HARDDimage (41% with 12 x 10 training data).
G.3. Comparison at equal FLOPs for non 52H-generalizing tasks
Align-Mic MMx++ uses an additional phase over Mix+, where training sequences from SIMPLE split are utilized. In Figure 5, however, we compare Mix+ and Align-Mix+ only in terms of the amount of training data used in the final phase. This raises a potential concern that Align-Mise might only appear stronger because involves more total training FLOPs. To address this, Figure 15 presents a revised comparison, plotting Align-Mix+ against Mix+ in terms of the total training data employed across all stages. Under these conditions, Align- Mig++ still consistently outperforms Mix+.
28

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Text
Mix (eval on HARD-text)
Image
Mix (eval on HARD-image)
MEDIUM
HARD
S2H Accuracy
1.0
0.5
0.0
2
4
8
16
32
64
2
4
8
16
32
64
N
SIMPLE (x103)
Figure 14. Results Consecutive Table Readout based the number of unique samples s Our observations from Section hold true even when different types of supervision are compared at the same value Nismm instead of NSMeE-
Table Results on Patlern-Heelout Visual Analogy: S2H generalization for Text, Image, and Mix supervision are reported on HARD-text and HARD-image examples after varying the number of training data in each strategy. S2H generalization on HARD-images under Image supervision peaks at 36%, while for HARD-text examples under Text supervision, it reaches 45.6% after 24 x 10 training examples. Leveraging the better performance on HARD-text, Mix supervision improves S2H generalization HARD-images 41% with 12 10* examples.
S2H accuracy on HARD-text
S2H accuracy on HARD-image Number training data
Number training data
Supervision
30k
60k
120k
240k
30k
60k
120k
240k
Text
37.2
32.6
45.6
0.0
0.0
0.0
0.0
Image
0.0
0.0
0.0
0.0
27.0
35.6
34.0
Mix
31.2
42.4
49.0
39.8
24.6
35.0
41.0
39.6
Mix+
-li+
Table Readout
Grid Navigation
Visual Analogy
1.0
S2H Accuracy
0.5
0.0
3
0.5
1.5
12
6
12
24
Total Number of Training Data (across all stages) (x10*)
Figure 15. Results S2H-generalizing tasks based on the total number of training data (Figure but including the amount of data in the alignment stage): Align-Mix+ still outperforms Mix+ when compared at the same amount total training data.
G.4. Transferring reasoning from image to text
In the main experiments, tested whether S2H generalization transfer from text inputs image inputs. In Table 8, observe that the transfer can happen in the opposite direction as well. After 24 x 10a training samples that now includes training data from HARD Image instead of HARD Text, a modified version of Mix+ achieves S2H generalization accuracy of 86.0% on HARD-text input on Table Readout and 85.6% on HARD-text input on Visual Analogy. As a comparison, when trained with the same number of data, Mix+ shows S2H generalization accuracy of 73.2% on HARD-imas input on Table Readout and 4% on HARD-image input on Visual Analogy.
29

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table Ablation on transferring reasoning from image to text: We modify Mix+ include HARD Image examples training, instead of HARD Text examples, while keeping the same SIMPLE Mix supervision. Evaluation is now performed on HARD-texi input. We observe that improving generalization performance on HARD-image input strongly transfers to HARD-text input.
Table Readout
Visual Analogy
Number of training examples
HARD-image (Included in training)
HARD-text
HARD-image (Included training)
HARD-text (Excluded in training)
(Excluded in training)
3 x 10
72.0
34.0
86.8
81.4
6 x 10
98.2
70.4
97.8
80.0
12 x 104
99.4
76.4
94.6
86,4
24 x 104
99.8
86.0
99.6
85.6
Text (eval on HARD-text)
Mix
(TW) Mix
Align-Mix+
(TW) Align-Mix+
Consecutive Table Readout
Table Readout
Visual Analogy
S2H Accuracy
0.75
0.50
0.25
0.00
3
6
12
24 Number of Training Data (x10*)
0
3
6
12
0
3
6
12
24
Figure Effeet text warm-up pretraining: We report the S2H generalization on image with/without text warm-up before Mix+ (Consecutive Table Readout) or Align-Mix+ (Table Readout and Visual Analogy). S2H generalization on text from Text supervision serves as a reference (in gray dashed line). Text warm-up enhances image S2H generalization across tasks. (TW) Mix closes the text-image generalization gap the HARD task for Consecutive Table Readout, while (TW) Align-Mix+ outperforms Align- Miign-Mi++tee for Visual Analogy.
G.5. Text warm-up pretraining
In Section 4, we observe that Mix fails to improve image generalization when the LLM backbone does not show strong generalization on text modality. Furthermore, models trained with Mix+ show significantly better image generalization when the text reasoning capability of the LLM backbone is strengthened by fine-tuning on HARD Text examples. Hence, one may expect that the reasoning capability of the L.M backbone on HARD examples is crucial factor for the reasoning capability to transfer to the image inputs. In this section, we investigate the effect of text generalization of the LLM backbone. Specifically, we simulate different levels of text reasoning ability by including pretraining stage of the model on SIMPLE and HARD Text examples. We call this text-only training (during which only the LLM backbone is updated) before full finetuning of the VLM model text warm-up pretraining (TW). This stage of training only uses a small set of 10 text examples (equal mix SIMPLE and HARD for non S2H-generalizable tasks, just SIMPLE examples for Consecutive Table Readout). Our results are shown in Figure 16.
We observe that the additional TW training further boosts the image generalization. In particular, (TW) Mix closes the modality imbalance, reflected by the image-text generalization gap on the HARD (25-30) task for Consecutive Table Readout. On Visual Analogy, (TW) Ali outperforms Align-Mix+ by 15%p with 12 x 10a training data. These results suggest that future stronger LLM backbone can further close the generalization gap between text and image modalities using our proposed strategy.
G.6. Utility of our synthetic datasets for existing evaluation benchmarks
Existing evaluation benchmarks test the ability of VLMs to perform OCR, chart interpretation, image reasoning, and caption generation. However, they primarily test the ability to generate short answers on a given question. On the other hand, our created tasks evaluate long form reasoning generation from models. As such, our fine-tuned models quickly forget to return short responses during training and struggle on existing benchmarks.
30

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Nonetheless, assess the utility of synthetic datasets to existing benchmarks by including them during visual instruction tuning. That is, we prepare different versions of alternate base model Faggle*Synthetic--22-lie 60 / 120k of our Image training mixture (equal mix of SIMPLE and HARD) has been mixed in with the 1.8M finetuning data; or where 240k of our Mix+ training mixture (80k for each synthetic task) has been mixed in with the 1.8M finetuning data.
1) where 30
Results in Table 9 demonstrate consistent improvements across tasks such as OCR, chart interpretation, and multimodal understanding. However, a decline in performance is also observed on binary classification (yes/no response) benchmarks, such as MME. These findings indicate that the proposed synthetic datasets can be valuable for future research, Further investigation is necessary determine how long reasoning datasets like our proposed tasks be best leveraged to enhance general reasoning capabilities (e.g. Gao al. (2024)).
Table 9. Utility of our synthetic data: We compare the benchmark results of Eagle-X2-Llama--ee solely instruction tuned on E.g dataset, and Engle+SSuthetic-X -xa-rti----------88 instruction tuned on mixture of Eagle-1.88 and our synthetic data mixture. Including our data can improve model's performance OCR and chart reasoning benchmarks, but may hurt performance on benchmarks where models need to output yes/no answer (marked with *) or a short phrase (marked with **). f: performance reported on validation set.
Visual Instruction Tuning Dataset
Eagle-1.88 Beele-1.8M
Eagle- Mix+ mixture
Evaluation
Eagle-1.80 1.8M
SIMPLE Image and HARD Image
Benchmark
(30k)
(60k)
(120k)
(240k)
MMMU
35.4
38.2
38.8
38.7
36.3
MME*
1529
1242
1377
1376
1364
MMBench
67.6
69.2
68.4
67.5
69.2
POPE*
86.6
88.7
88.9
87.6
87.5
TextVQA **
66.8
66.5
66.9
66.8
65.8
OCR(Bench)
47.3
50.9
50.4
47.0
48.2
ChartQA
69.6
71.6
69.8
70.5
69.4
CharXiv-Reasonine
16.8
16.4
16.5
17.0
17.2
CharXiv- Deserpptiee
30.7
28.3
35.8
31.1
34.4
Reported benchmarks: Here is summary of the reported evaluation benchmarks.
MMMU (Yue et al., 2024): Evaluates on multi-discipline tasks measuring college-level subject knowledge and reasoning.
MME (Fu et al., 2023): Evaluates both perception and cognition abilities across 14 subtasks with yes/no answers.
MMBench (Liu et al., 2025): Evaluates on VQA, which includes both multiple-choice and free-form answers.
POPE (Li et al., 2023); Evaluates object hallucination with yes/no answers.
TextVQA (Singh et al., 2019): Evaluates understanding and reading within images with short-phrase answers.
OCR(Bench) (Liu et al., 2024b): Evaluates on Character Recognition (OCR) capabilities across 29 datasets covering text / handwritten mathematical expression recognition, key information extraction, and scene text document VQA.
CharXiv (Wang et al., 2024c): Evaluates chart understanding based on 2323 charts from arXiv papers, paired with descriptive and reasoning questions, covering 8 major academic subjects.
31

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
H. Continued Discussion on Gradient Alignment
Augmentation in notation: Suppose the current model parameters are given by 9. We will slightly augment our notation to include the model's current parameters. At model parameter 6, we will use (1s (x; 0) to denote the loss on Image example for data x and loss ( (0) Xuss XHARD, i.e.
(1:5)) ((:s (x: 0). Then, ESIMPLE (0) and SHARD (0) denote average gradients on XSIMPLE and
gSTMPLR (0) X Vl(u,s) 0), BHARD (0) E (x:1:s((x:0)
Recall that the gradient alignment score from Equation (3) is given by
(EsIpLLe()) BHARD (0)) (gaAR (0), (H)RD(O)).
(6)
In the following theorem, we show that the gradient alignment score quantifies the amount of loss that we can decrease in expectation on HARD Image examples by taking gradients on SIMPLE Image examples, relative to taking gradients on HARD Image examples.
Theorem H.1. Suppose for model fo with parameter 0, loss ((r;)) is Lipschitz and has bounded gradient norm on X around parameters 9, with l|BARRD (f) + O. The following holds true for expected drop in ((H) with SGD when using a random training sample from the SIMPLE task, compared to using random training sample from the HARD task:
(H) (1:5) (0 m1 (1:s)) (0)]
E
Vions (x:0)
lim 7
= (BHARD
(HARD))))(
E
N
((H) (1:5) (0 78))
((H)
(1:5) (0)
2 Vi(r,s (x:0)
The proof follows from standard convergence analysis of gradient descent algorithm (Nesterov, 2018).
H.1. Proof of Theorem H.1
Proof. Say g Vla.s) (x; By Taylor's theorem, have the following for a small enough learning rate ?
((:s) (0 ng) (11s)) (0) - V
(T:S) (ns) (0).
(m + n'ggT (v (0 nog))
for some To E (). We first note that Vi( (0) gHARD (0). Next, since the loss is assumed to be Lipschitz,
gT V (1:5) nug) g - L
where L is the Lipschitz constant for the loss. Since the gradient norms are also assumed to be bounded, we have
(1:)) (0 n
(H)
(HH (1:s) (0) = - (EHARO + O(n²),
First assume x E XIMMLL. By taking expectation over x,
[ (0- rg) 1/4 (0] 7) (EARD (8), ExE (x::s) + O()²)
XEXamm (x(1))(x:0)
- (EUARD (0), ESIMPLE (0)) + 0(7²)
Similarly, assume g = Vl(r,s) (x; where x E MHARD By taking expectation over
E NEAD
[0:)) (6- n)) (11,5) (0)
(H)
17 (guARD (0), ExEXXOD Vl(r,) (x) + 0
= (x()))
(gHARD (0)
Therefore, we have
E
(H) (0- 1//) (H)
(0)
(1:S) (x.)))
77 AHARD (0), BSIMPLE (0)) + 011²) (gaRd (0), SamaL (0)) + O())
E
(1:S) (=V((1))(x0))
Navaa
(H)
(0- m//)
(1:S) (0)
(H)
-1 (gARD (0), BHARD (0)) O(1²)
(HARd (0), gHARD (0)) + O(r)
Note that BHARD (f) and (f))PLE()) do not depend on the value of n. Furthermore, by assumption, |uARD (0)ll2 # 0. We conclude by taking 1) -> on both sides of the equation above.
32

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Mix
Mix+
Align-Mix+
Gradient Alignment
SIMPLE Gradient Norm
10-3
0.10
10-4
0.05
10-5
0.00
100
200
300
400
100 200
300
400
Training Steps
Training Steps
Figure 17. Analysis of gradients Table Readout (additional plots for Figure 9): (Left) Gradient Alignment Score (Equation (3)); (Right) Average Gradient Norm on SIMPLE Image examples Ex Conssa//[[ca (V((s))) (x)]]2. Align-Mix+ has higher gradient alignment score in the initial phases of training, where it also has higher gradient norm. Mix+ shows higher gradient alignment score than Mix during the course of training.
Mix
Mix+
Align-Mix+
Gradient Cosine Similarity
Gradient Cosine Similarity
SIMPLE Gradient Norm
0.6
0.6
10-3
0.4
0.4
10-4
0.2
0.2
10-3
0.0
0.0
10
10-4
10-3
200
400
200
400
SIMPLE Gradient Norm
Training Steps
Training Steps
Figure 18. Analysis of gradients Table Readout (replacing gradient alignment score from Figures 9 and 17 with gradient cosine similarity): (Left) Average Gradient Norm on SIMPLE Image examples (Exe s) (x)l12) vs. Gradient Cosine Similarity (Equation (7)) different training checkpoints; (Middle) Gradient Cosine Similarity; (Right) Average Gradient Norm. Similar results hold.
H.2. Additional measure 1: gradient cosine similarity
We additionally define the gradient cosine similarity score the cosine similarity of gradients from XSIMpLEE and XHARD:
(ESIMPLE (0), BUARD(()
Gradient Cosine Similarity:
(ghaRd (f), BHARD ()) (gSIPPLE (0); ESIMPLE
Note that this measure ignores the norm of the gradients XSIMPLE that the model uses during training. Hence, this measure is not an entirely faithful measure on the alignment of the training updates to the loss on HARD Image examples. Figure 18 shows the gradient cosine similarity score across training strategies for Table Readout, which follows similar pattern as the gradient alignment score Figure 17.
H.3. Additional measure 2: Adam update alignment
The gradient alignment score we defined earlier does not account for the fact that we use Adam optimizer (Kingma & Ba, 2015) during our experiments.
Brief definition of Adam: The Adam optimizer maintains two additional states, each representing the running average of the gradients and their squares during training. If m, and v, denote the two states, then the update rule training step f
33

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
with gradient Ee and learning is given by
(1 81)gr +
0 <- 0 - n/((g.),
where h(g:)
v 32 g: @ g.+ 3 +
m. - (1- B1) + -1m-11
(1 32 0g: + 32
Here, (31, 82,6) are hyperparameters for the Adam optimizer and are set at (0.9, 0.999, 10 10-8).
Adam Update Alignment: A true measure of alignment between SIMPLE and HARD training would be to compare h(-), the update vector under the Adam optimizer. However, that requires saving the Adam optimizer states throughout training. For storage efficiency purposes we propose an alternate approximate measure called the Adam update alignment score. We compute the following two quantities for model fo with parameters 8:
m (0) == E
Vl(i:s) 0)]
(0) == Exe
(xx 0) o Vlu.s) (x: 0)]
m and v are proxy measures for the Adam optimizer states. Then, we measure the alignment between gradients for NHARD and SSIMLL as
E
xE NaNru
(h(g), SHARD (0))
Adam Update Alignment Score:
g
(x:19)(x:9)
(8)
E
XE:Nsso 8: (V11,)(x:0)
(h(g), SHARD (0))
(1 81 + mm
where h(g)= V 32)g g ① g g (2) +e
for any vector g
(H)
Intuitively, this measures how much the loss ((:55) can be reduced in expectation by taking a gradient update step with Adam using SIMPLE Image examples, compared to taking a gradient update step with HARD Image examples, while maintaining the current Adam optimizer states. This can be formalized the following theorem.
Theorem H.2. Suppose for a model fo with parameter loss ((ss)) Lipschitz and has bounded gradient norm on X' around parameters 0, with [HARD (f)ll2 # O. Consider a modified Adam update with learning rate 1) with arbitrary gradient g, as follows:
0 0 nnh(g)
(1 (1- + 3im(()
where li(g)
v(1 - -1- 32) (2v( +
The following holds true for expected drop in (I:S) with modified Adam update when using random training sample from the SIMPLE task, compared to using random training sample from the HARD task:
(H)
E
xe Xamma
Liu -n((g)
-(1ss (0)
(H)
E
(h(g), gHARD (0))
lim
(1:))
g:
(x;)))
E
(0:s) m)(()) (H)
(0)
E
xENua (x()))());)
(h(g), SHARD (f))
B: (1:)))
The proof is similar to that of Theorem H.I.
H.4. Experimental results
In Figure 19, we present the analysis of gradients for different types of supervision on Consecutive Table Readout. Similar to the behavior of gradient alignment score in Figure 7, we observe that when measured against norm of gradients on SIMPLE Image examples, Mix achieves a higher Adam update alignment score than both Text+ and Image. This shows that Image-via -ext supervision improves the alignment between SIMPLE and HARD Image gradients, when taking Adam gradient updates into account.
Similarly, for Table Readout in Figure 20, Mix+ has larger Adam update alignment during training. Align-Mixe further improves the Adam update alignment score when gradient norms are large during training.
Remark: retrieving the actual Adam optimizer states requires an additional storage 138GB per checkpoint.
34

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Image
Text Image
Mix
Adam Update Alignment
Solution (m))e(4
1.0
HARD Eval Loss
0.5
10-1
0.0
2-14
2
2-11
2 2-8
100 Training Steps
200
SIMPLE Gradient Norm
Figure 19. Analysis of gradients on Consecutive Tuble Readout (replacing gradient alignment score from Figure with Adam update alignment score): (Left) Average Gradient Norm on SIMPLE Image examples RR (V(,,))())ll vs. Adam Update Alignment Score (Equation (8) for different training checkpoints; (Right) Average Loss on solution given HARD image ((4) during training. Similar results hold.
Mix
Mix+
Align-Mix+
Adam Update Alignment
Adam Update Alignment
SIMPLE Gradient Norm
0.6
0.6
10-3
0.4
0.4
10-4
0.2
0.2
10-5
0.0
0.0
10-5
10-4
-0
200
400
200
400
SIMPLE Gradient Norm
Training Steps
Training Steps
Figure 20. Analysis of gradients on Table Readout (replacing gradient alignment score from Figures 9 and 17 with Adam update alignment score): (Lefft) Average Gradient Norm Image examples (fRxe-Xammz (x) Adam Update Alignment Score (Fquation (8)) for different training checkpoints; (Middle) Adam Update Alignment Score; (Right) Average Gradient Norm, Similar results hold.
Mix+
Align-Mix+
Image-to-text (g(H)
Solution Image+Text (((H)
Solution Image (1(6)
(7()))))
(T:s)
10"
HARD Eval Loss
10-1
10-1
10-3
10-1
0
250
500
750
250
500
750
250
500
750
Training Steps
Figure 21. 21. Analysis of evaluation losses (repeating Figure for Visual Analogy): (Left) HARD image-to-tex conversion loss (1)
(Eq.4)): (Middle) loss on solution given image and text ((())) (J.H))) (Bq.5)); (Right) loss on solution given image (1.)))
(Eq. Similar results hold.
35

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
I. Additional Ablations
L.1. Performance of other multimodal models
In Table 10, we present the performance of three closed source and two open source multimodal models on our three non -2Hgeenrraiizing tasks. Since we do not train these models, the format of the outputs is more flexible. For convenience, we propose alternative metries to extract and evaluate on the models' predictions. For Table Readout, we instead evaluate with the final answer (sum of the sequence of numbers) For Grid Navigation, evaluate with the same metric as in the main part of the paper- whether the proposed path can move from the start cell to the end without rumning into obstacles. For Visual Analogy, we evaluate with just the final option, which is a more lenient metric than the one suggested in the main part of the paper. Note that a random choice baseline should get 25%.
Table 10, Performance other multimodal models: For convenience, we evaluate the models on a slightly different metric.
Table Readout
Grid Navigation
Visual Analogy
Models
SIMPLE
HARD
SIMPLE
HARD
SIMPLE
HARD
Claude-3.5 Sonnet
30.0
0.0
0.0
0.0
35.0
29.8
GPT-4o
19.0
0.0
0.0
0.0
19.8
18.4
OpenAl ol
29.0
0.0
30.6
Llama3.2- 1- IBB-isiio-Innttruet
4.0
0.0
0.0
0.0
16.2
17.8
Pixtral- 12B (Agrawal et al., 2024)
9.0
0.2
0.0
0.0
24.6
21.2
I.2. Ablation of the Mix+ supervision
The dataset composition of the Mix+ supervision consists of three types of supervision in the SIMPLE task: Text, Image, and Image-via-Text. In this section, we ablate on the importance of each component of the data mixture in the training of Mix+, (TW) Mix+, and (TW) Align-Mix+. Figure 22, we report the S2H generalization performance on image when the SIMPLE Mix supervision is replaced with a varying data composition.
In single-stage training (no text warm-up or alignment), Image-via-Text" -eext is the key component of success, evidenced by the strong performance of Image-via via-Text+ supervision. As noted in Section 4, Mix+ match the performance by explicitly prompting the resulting model to convert the image first, which comes at cost of around 1.7x generated tokens at inference time.
In multi-stage training (either (TW) or (TW) Align-), the benefits of Mix+ are more significant. Specifically, among all other types of supervision with text warm-up, (TW) Mix+ is able to outperform the others by at least. 1.7x, while retaining efficient inference costs, unlike (TW) Image-via Text+. Among all types of supervision with text warm-up and alignment, (TW) -lxx achieves the highest performance.
L.3. Ablation of the reasoning alignment phase (Align-)
We perform two ablations for the first phase of the Align-Mix+ supervision. In Figure 23, we report the S2H generalization generalization performance on image of models trained with a varying amount of data in the first phase of Align-Mix+, with the amount of Mix+ data fixed in the second phase. We don't observe monotonic improvement in performance when increasing the amount of data in the first phase. In Table 11, we report the S2H generalization generalization performance on image of models trained with a varying data composition in the first phase of Align-Mix- Our choice of Text and Image-via-Text from the main section gives the best performance on average on Table Readout and Visual Analogy.
L.4. Ablation of the text warm-up pretraining phase (TW):
We ablate on the effect of the training data size during the text warm-up. In particular, we are interested in whether models with better reasoning capability on text can achieve better image generalization. To do so, we vary the number of training data used for text warm-up between (1, 2. x 104 and plot the performance of the warmed-up LLM on HARD-text examples against the performance of the final trained model on HARD HARD-image examples. We report the performance on Visual Analogy in Figure 24. We observe that model's text performance improves with more training data being used for the text warm-up up as expected. However, there is no clear linear correlation between the capability of the model checkpoint after the warm-up
14The closed source models have access to tool-use, so in theory, this should be an equivalent, if not more lenient, metric.
36

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
No Pretraining
(TW)
(TW) Align-
Generation Length
S2H Accuracy
1000
50
-500
Image+
Text text+mmage+
Image-via-Texx via--ext++
Mix+
Align-Mix++
Image
Text+ +magee+
Image-via-Texxe via-Text+
Mix+
Align-Mix+
Image+
Text Text+ +magee+++++++
Image-via-Texx: via-Text+
Mix+
Image+
Image-via-Tex"" -ex-Text++
Mix+
Image+
Text+ Image++
Image-via-Tent -aa-Text+++
Mix+
Image+
Text+ +magge+
Image-via-Tewxe vee-vii--eet++
Mix+
Text+Imager
Figure 22. Ablation the SIMPLE data composition of Mix+ Visual Analogy: Instead of Mix, differrnnt types of SIMPLE
supervision in (Left) Mix+; (Middle) (TW) Mix+; (Right) (TW) Align-Mix+. The main phase (Mix+) uses 12 x 10" training data.
0.75
S2H Accuracy
SIMPLE data composition
Accuracy (after phase 2)
Corresponding Name
phase (Align-Mix+)
0.70
Table Readout
Align-
Text, Image-via-Text
0.76
Text+Image
Text, Image
0.52
Image-via-Ted vex-
Image-via-Tent
0.77
0.65
Mix
Text, Image, Image-via-Texk -vi--Text
0.74
Visual Analogy
10 20 30
Align-
Text, Image-via-Text
0.66
Number of Training Data (x103)
Text+Image
Text, Image
0.19
Image-via-Texx.
Image-via-Tess
Mix
Text, Image, Image-via via--extt
0.46
Figure 23. Ablation the amount of data for the alignment phase of Align- Mix+ on Table Readout: Second phase (Mix+) 12 10* training data. We don't I observe monotonic improvement in generaliza- tion performance with increasing number of training samples in the first phase.
Table 11. Ablation the SIMPLE data composition for the alignment phase of Align-Mix+: Amount of data for the alignment phase is fixed at 10 Second phase (Mix+) uses 12 10 training data. Performance is reported on validation set with 100 HARD-image examples. Our composition of Text and Image-via-Text the SIMPLE task performs best average on Table Readout and Visual Analogy.
training stage and image S2H generalization of the final model. Specifically, model with x 10 warm-up performs the best for the (TW) Mix+ supervision, while a 10 warm-up works the best with the (TW) Align-Mix+ supervision. Meanwhile, we observe that (TW) Align-Mix- supervision can universally achieve better S2H generalization on image than the (TW) Mix+ across all data scales. conclude that an improved text capability by itself is insufficient to guarantee good transfer image modality. We expect future VLMs with both stronger LLM backbone and better modality alignment can further leverage the text performance and transfer it images.
1.5. Requirement of text representation
One potential limitation of our proposed training strategies is the requirement of a text representation corresponding the image. In Consecutive Table Readout, Table Readout, and Grid Navigation, we use the LaTeX code of the table grid, which is considered be perfectly aligned with the image. In reality, it may be challenging to find an exactly equivalent text description or representation of real-world image, as many minute visual features cannot be captured by language. We show that our proposed training strategy does not require perfect alignment between the text and the image representation to work. For Visual Analogy experiments, the text description of the puzzle in the image lossy: it only enumerates unique values all task-relevant attributes without encoding the object to which each corresponds, so one cannot recover the original image given the description (see examples in Figures 36 and 37). Models trained with our proposed training
37

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
(TW) Mix+
(TW) Align-Mix+
(TW)
1.0
0.8
S2H Accuracy
0.6
0.4
0.2
1
2 Number of Training Data in (TW) phase (x10*)
Figure 24. Ablation the amount of data for the text warm-up phase on Visual Analogy: Second phase (Mix+) uses 12 10* training data. Using more data for the warm-up stage results a stronger L.IM backbone with hetter HARD-text performance (gray dashed line), but does not necessarily lead to better image S2H generalization of the final model trained with our proposed strategy. This suggests that a stronger text capability is not the only factor that induces S2H generalization on image.
strategies (Mix+, Align-Mix+, (TW) Align-Mix+) all demonstrate significant improvements in image generalization (Figure 6), testifying that our methods work with lossy text representation.
Lossless text representation for Visual Analogy: We additionally conduct experiments where the text representation of the puzzle in the image is a lossless representation. We represent the panels in the puzzle as code defining each object as a set of attributes. Each geometric object is represented by the values its 5 attributes: (shape type, shape co 1 or, shape_s ize, thaae-quantitee ity. shape-poss line.color). order to the context length of the VLM, we describe each object in shorthand notations, For example, for panel in the puxxle that contains circle and 2 rectangles, with attribute values (45 (gray-scale), 42 (pixels), 1, top-left) and {{0, 90), (21, 21,,2, top-right, bottom-left). we will represent the panel as
ion], while lines are defined by their attributes: {line.type
We give all details on how to parse the shorthand codes the prompt. On the other hand, for the same example, the (Lossy) text representation would have been
type: circle, rectangle
color: 0,90
size: 21,42
quantity: 1,2
position: top-left, top-right, bottom-left
This substantially reduces the context length on average on our training dataset, and further removes the necessity of parsing code. However, this isn't an exact representation of the image of the puzzle.
Performance lossy and lossless Visual Analogy tasks: In Figure 25, compare Mix+, Mign-Mix+, and (TW) Align- Align-Mix+ for lossless and lossy Visual Analogy tasks at 12 x 10a training examples in the final phase (Mix+). Our observations reveal that a lossless text representation enhances S2H generalization performance on images for Mix+. However, for Align-Mix+ and (TW) Aix+, the lossy text representation leads to better S2H generalization performance on images. This discrepancy could attributed to the complexity the shorthand code in the lossless text representation, which requires additional parsing. We did not investigate this phenomenon further.
38

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
(Lossy) AR
(Lossless) AR
Mix+
Align-Mix+
(TW) Align-Mix+
0.75
S2H Accuracy
0.50
0.25
0.00
0
3
6
12
24
24 Number of Training Data (x10*)
0
3
6
12
0
3
6
12
24
Figure 25. Ablation on lossy vs, lossless Visual Analogy: measure the image S2H generalization of different types of supervision for two different versions of text representation for Visual Analogy. Models can perform better on Lossless Visual Analogy with Mix+. However, the trend can change with Align-Mix+ and (TW) Align-Mixt
L.6. Explicit and implicit text conversion
In Appendix 1.2, we find that explicit text conversion (Image-via-Txxe is the key component in the data composition the Mix+ supervision. At inference time, however, models trained with Mix+ reason directly on HARD images, without explicit text conversion. In Table 12, we observe that the trained models can still perform reasoning with explicit text conversion and that the conversion ability helps it reason.
Mix+ models can convert image to text when prompted. If only an image input x(0) is provided, Mix+ models will always directly predict CoI (x) and fxx) never converting image to text (under greedy decoding). However, since the prompts used in the Image and Image-via-Text examples are the same, we can induce explicit text conversion in the final trained model by additionally providing the first word "Convert" of Poonvert- We find that all trained model are always able to continue with explicit text conversion - they will generate the rest of Pronvert and an attempted conversion (()) before (x). The conversion accuracy around 50% on Visual Analogy and is almost 100% on Table Readout.
Explicit text conversion generally helps the model to reason on image data. Noticeably, the Mix+ (240k) model improves S2H generalization accuracy from 73.2% to 96.6% with almost perfect text conversion accuracy of 99.2% on Table Readout. On Visual Analogy, the Mix+ (120k) model improves S2H generalization accuracy from 35.4% to 51.8% with a text conversion accuracy of 47%. The benefit of explicit text conversion gradually diminishes with multi-stage training strategies.
We also observe a slight drop in performance with prompted text conversion for models trained with (TW) Align-Mix+ on Visual Analogy which corresponds to minor decline in reasoning performance with explicit text conversion. This suggests that the text warm-up training and alignment phase enable the model to close the gap between direct reasoning and reasoning with explicit text conversion, where the model learns to rely more equally on both text and image modalities, and doesn't 't require explicit text conversion for improved generalization performance.
Models are robust against potential errors in the prompted text conversion. For models that are prompted to perform text conversion, we examine any negative side effects of this step. When the model does not correctly convert the image to its text format, we investigate whether to what extent the model's reasoning introduced by the text conversion step. Interestingly, On Visual Analogy, we find that the models trained with Mix+, (TW) Mix+, and (TW) Align-Mix+ are still able to arrive at the correct reasoning solutions with accuracy 44.3%, 35.4%, and 63.1% respectively on evaluation examples where the trained models make a mistake in text conversion.
be affected by the additional noises find that our final trained models are generally robust to such noises.
1.7. Explicit and implicit CoT
We use ofain-of-thougeee (CoT) as technique to boost the model's reasoning ability in all our experiments. In this section, we explore the role of CoT in our proposed strategies, as well as the possibility of transferring the reasoning capability from text to image modality without COT. In Table 13, we report our observations on Visual Analogy. We note that similar observations hold for Consecutive Table Readout and Table Readout.
39

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table 12. Ablation on explicitly prompting for text conversion: When models are additionally prompted with "Convert," " they exhibit the retained ability of text conversion. The conversion accuracy is near perfect on Table Readout. The S2H generalization performance with an additional prompt "Convert" (Propte)))))m)pee) improves from direct inference (Direct). The improvement margin diminishes with stronger Direct performance. All evaluations are on 500 HARD-image examples.
Task
Supervision (Number of Training Data)
Direct
Prompted
Conversion acc
Mix+ (240k)
73.2
96.6
99.2
Table Readout
Align-Mixt (240k)
87.6
98.0
100.0
(TW) Align-Mix+ (240k)
86.2
97.8
99.4
Mix+ (120k)
35.4
51.8
47.0
Visual Analogy
TW) Mix+ (120k)
55.2
62.8
49.0
TW) -lix (120k)
73.6
70.2
49.6
1.7.1. REMOVING COT COMPLETELY
We first consider completely removing CoT from Mix+ and observe the drop in performance measured by image S2H generalization. We experiment with Mix+, (TW) Mix+, and (TW) Align-Mix+ supervision, in which completely remove CoT from the last phase of training which has the Mix+ supervision, while preserving the full CoT in the text warm-up (TW) and/or reasoning alignment (Align-) phases.
Model does not learn when CoT is completely removed: When CoT is completely removed from Mix+, performance drops to almost 0% for all three types of supervision. We manually inspect the model's output and find that the generated reasoning on HARD-image inputs is identical to the expected behavior for SIMPLE instances, which indicates that the reasoning capability on HARD instances failed completely to transfer from the text to image modality.
1.7.2. PROGRESSIVELY INTERNALIZING COT THROUGHOUT TRAINING
The failure above can be expected: for Mix+ supervision, CoT may serve as a crucial technique to elicit good reasoning behaviors while for (TW) Mix+ and (TW) Align-Mixe the transition from training with full CoT to training without CoT can be too drastic for the model adapt. Therefore, consider milder approach that trains the model internalize reasoning by progressively removing CoT from the training (Deng et al., 2024). We train on the first 30% of 12 x 10 Mix+ examples with full CoT, the next 40% of examples with progressively less C and the last 30% of examples with no COT.
Internalizing CoT during the Mix+ phase also fails: In this scenario, we also observe that the model completely fails on image S2H generalization, getting almost 0% S2H generalization on HIARD-image examples for all three types of supervision strategies.
1.7.3. INTERNALIZING COT DURING TEXT WARM-U BEFORE REMOVING CoT COMPLETELY
We also try variant for the multi phltt-ppaseeee approaches, where we internalize the CoT on the text input during a text warm-up ((TW)) stage and continue with Mix+ with CoT completely removed.
CoT can be internalized on text inputs: We internalize the CoT on the text input during a slightly modified text warm-up phase of (TW) Mix+ Specifically, with 10a training data that consists of an equal mix of SIMPLE Text, HARD Text supervision, and Eagle instruction tuning data (randomly sampled from 1.8M examples (Shi et al., 2025)), we train on the first 30% examples with full COT, the next 40% examples with progressively less CoT, and the last 30% examples without CoT as in previous experiments. After the warm-up phase of training, the model can achieve 97.8% accuracy on HARD-text examples, which shows the model's ability to internalize reasoning on text inputs.
Explicit CoT is "necessary" for the internalized reasoning to transfer to image: We then continue with the Mix+ supervision with all CoT removed. The final trained model completely fails with 0% accuracy on the HARD-image examples. Similarly, examining model outputs reveals that the reasoning capability on HARD instances failed completely to transfer from the text to the image modality. Therefore, we conclude that CoT "necessary" for the cross-modal transfer of knowledge happen in our setting.
15 "llli into 101 subsets of equal length, each training with 100%, 99%, ... 0% of total characters the COT.
40

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table 13. Ablation on removing or internalizing CoT on Visual Analogy: Preliminary attempts completely or progressively remove CoT during the Mix+ phase fails to generalize to HARD images, which shows the importance of CoT in our proposed strategies. filll, none, internalizing CoT refer to including full CoT, completely removing CoT. and progressively removing CoT respectively. means the corresponding phase was not included during training. Unless specified, all evaluations are reported on HARD-image examples.
Type of CoT
Type of supervision
S2H accuracy (%)
(TW) (10k)
Align- (10k)
Mix+ (120k)
Mix+ (TW) Mix+
0.6
full
none
0.0
(TW) Align-Mix+
full
full
none
3.6
Mix+
internaiizing
0.0
(TW) Mix+
full
internalizing
0.0
(TW) Align-Mix+
full
full
internalizing
3.4
(TW) Mix+
internalizing
97.8 (IIAR--exx) 0.0
(TW) Mix+
internalizing
none
Image
Image-via-Text+
Table Readout
Grid Navigation Individual
Visual Analogy
Mix+
Align-Mix+
Multi-Task
Mix+
Align-Mix+
Avg. S2H Accuracy
1.0
S2H Accuracy
0.5
X
0.0
3
6
12
12 Number of Training Data per Task (x10f)
3
6
3
12
Figure 26. Ablation on jointly training all three S2H-gener tasks: (Left) Average S2H Generalization image; (Middle, Right) Comparison of Trained Jointly vs. Individually. Similar to training on each task individually. Mix+ and Image-via-Text+ outperform Image, and Align-Mix+ Mix+ matches the performance of Image-via vext+. Multi task SFT boosts image S2H generalization Table Readout and Grid Navigation, while Visual Analogy performance remains unchanged or slightly declines, indicating task interactions drive the cross-modal transfer reasoning capabilities in multi-task training.
All results testify to our claim that CoT is important in our proposed training strategies. As the techniques used to internalize or remove the CoT dependency in our experiments are very preliminary, we are not eliminating the possibility of internalizing CoT in our setting. We note that to do so may require more careful, post hoc approaches, which we leave to future work.
L8. Multi-task training: jointly training on all three non S21l-generalizing tasks
In the main experiments, we have trained on each non S2H-generalizing task separately. In this section, we explore the ablation where we combine and randomly shuffle the training data for Table Readout, Grid Navigation, and Visual Analogy. In Figure 26, we compare the image S2H generalization performance when jointly training on all 3 tasks against training on each task separately.
Similar training on each task individually, the average S2H generalization on image across all 3 tasks is strongest for Image-via vix--extt, followed by Align- -Mi+ and Mix+. When analyzing the effeet of multi-task training each task, observe that it benefits the model's performance on Table Readout and Grid Navigation but hurts performance on Visual Analogy. This is likely because Table Readout and Grid Navigation are similar in nature. They are both represented by LaTeX code in the text modality, require the model to identify the current location neighboring cells. On the other hand, the skills required for Visual Analogy are quite distinct. This suggests that the interactions between tasks during a multi-task training can also affect how much reasoning can transfer across modalities.
table / grid, and reason about

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Mix+
Align-Mix+
Table Readout
Visual Analogy
0.8
S2H Accuracy
0.6
0.4
5 10
20
1
5 10 20 Number of Repetitions
Number of Repetitions
Figure 27. Ablation on the number of repetitions of unique HARD examples, while maintaining the total amount of training data, on Table Readout and Visual Analogy: Image S2H gencralization degrades with more repetitions of HARD Text examples, with the effect Mix+ being more drastic. Here, amount training data is at x 10 with 6 10 examples sampled from the HARD task. Interestingly, performance of Align-Mix+ peaks at 3x repetitions, implying the number of unique HARD Text examples can be reduced by 3x for Align-Mix+.
I.9. Ablation on repeated HARD examples
In the experiments reported in the main paper (summarized in Figure 5), we kept all HARD Text examples unique. In Figure 27, we present the ablation where we repeat each HARD Text example during training, while keeping the total number of training data fixed. Our primary observations are:
Repeating HARD Text examples harms the performance of Mix+. Halving the number of unique HARD Text examples and repeating each example 2 times can drop the performance HARD-image by least 10%p on Table Readout.
On the other hand, Align-Mixe quite robust to repetitions on Table Readout. The number unique HARD Text examples can be reduced by 10x (and repeating each example 10x) with the performance on HARD-image dropping by more than 1-2%p.
On Visual Analogy, while the performance of Align-Mix+ drops with large number of repetitions, the drop in performance is within 1-2%p the number of repetitions up to
Interestingly, the image S2H generalization performance reaches its peak at exactly 3 repetitions for Align-Mix+ on both Table Readout and Visual Analogy. This suggests that we may only require 3x less unique HARD Text examples than reported in 5.
42

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
J. Interpretability Experiments
We use gradient attribution to identify which pixel in the image is important when generating each token in the CoT. For given data x E X and its corresponding image format x(), we label the set of pixels in the image as (xx (8) ). For given gold
output = (x) f(x), we label the sequence of CoT tokens as {yk}: where Y. refers to the subsequence of the CoT tokens, up to the k-th token.
For each pixel x/x E x() and each CoT token ye. we compute the attribution score as:
Pixel Attribute Score:
V Se N...).
x x(0)
Informally, on Image examples, we take the gradient of the loss of the model's output (up to the k-th CoT token) with respect to each pixel, and project on the pixel values. Pixels that show positive alignment with the gradients are marked important for the model's prediction for the token yk--
In Figure 28, we plot the pixel attribute values, averaged across tokens that correspond different segments of highlighted path of an example image from Table Readour. We observe that Align MMix+ improves over Mix+ models by having more focused and concise pixel attributes around the path of highlighted cells and their corresponding row/column names. In Figure 29, also show pixel attribute scores Visual Analogy, where the pixel attributes interest scattered around the grid.
aligned with objects of
43

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Figure 28. Visualization of pixel attribute scores on Table Readout: (Top) Mix+; (Bottom) Align-Mix+. Models are trained with 24 x 10* training data. Pixel attribute scores averaged across CoT tokens that belong the first pixels roughly in the 10th column (left), the next 6 cells in the 8th column (middle), and the last cells in the 6-th column (right). We show the top- pixels with the highest pixel attribution scores (marked red). Mix+ has more diffused pixel attributions in the image, while Align-Mixx focuses more on the path cells (and their corresponding row/column names).
?
?
?
Figure 29. Visualization of pixel attribute scores on Visual Analogy: The model is trained with 12 x 10* training data of (TW) Align Mix+ Pixel attribute scores are averaged across CoT tokens that belong to Example (left), Example 2 (middle), and the query (right) respectively. We show the top-1% pixels with the highest pixel attribution scores (marked as red). The pixel attributes are focused on relevant objects across the grid. Interestingly, when reading relevant object attributes in Example 2, the model still attends to objects from Example 1.
44

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
K. Analysis of Failure Modes
In this section, we briefly discuss the common failure modes of models trained on our synthetic data, when evaluated on examples from the HARD split.
K.1. Table Readout
We analyze the outputs of Text on HARD-text, Image on HARD-image, and Mix+ on both HARD-text and HARD-image, where all models have been trained on 24 x 10 examples.
Since the models perform almost perfectly on the SIMPLE examples, where the total length of the sequence is around 12, may expect the models to read off the first 12 numbers from tables of the HARD split equivalently well but start making errors after the sequence length it was trained on. We find that this is not the case by analyzing the index of the first error; i.e., how many numbers the model reads off correctly before making the first mistake. Although the average index of the first error is around 14.7, about 56% of incorrect generations (equivalently, 26% of total generations) contain mistake before the 12th number in the sequence.
Precision
Recall
1.0-
0.9
0.8
0.7
0.6
Text
Image Type of Supervision
Mix+ (text)
Mix (image)
Figure Analysis of failure modes on Table Readout: (Left) Precision and Recall; (Right) Example of common mistake, Models are trained on 24 x 10 examples of Text, Image and Mix+ supervision and evaluated on corresponding inputs from HARD. Models often hallucinate "shortcut." this case, precision would 12/13 and recall would be 12/29.
To further analyze the behavior of the model when it makes mistake, extend the definition of precision and recall:
Precision Total correctly listed Recall
Total # correctly listed
Total listed
Total # highlighted
where we take the sum in the numerator and denominator across all test examples and mark a cell as correctly listed only the model generation contains it, regardless of the exact position in the sequence. See left of Figure 30 for the evaluation results. Note that for Text and Image, precision is significantly higher than recall, meaning that it rarely hallucinates that a cell is highlighted (when it is not), but fails list off many the numbers that were highlighted. We find that this is mainly because once the model derails from the highlighted path, it just moves directly towards the destination cell, until it rejoins the path, unintentionally creating "shortcut" that skips around 15 cells the original path average. See right of Figure 30 for visualization. However, the recall improves significantly on both HARD-text and HARD-image when trained with Mix+
K.2. Grid Navigation
In Figure 31, we analyze the outputs of Text supervision HARD-text, Image supervision HARD-image, and Mix+ supervision on HARD-image, where all models have been trained on varying number of examples.
successful evaluation on Grid Navigation requires completing multiple intermediate subtasks. The model first needs to correctly identify the source and destination cells from the grid and parse the row/column indices. We observe that the models can easily learn this subtask. Under any of the three types of supervision, the model can get at least 98% accuracy
45

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
on parsing the location of the source and destination cells with only 1.5 10 examples. With 6 10 more examples, the accuracy is always 100%.
Next, we analyze whether the model returns sequence of actions that leads from the source to the destination (ignoring any object or obstacle). We observe that there is some "phase transition" at 3 x 10 examples, where the model's accuracy on this subtask increases sharply. However, whereas Mix+ continues to improve accuracy on this subtask, exceeding 90% at x 10a examples, Text and Image supervision fail to achieve 90% even with 24 10 examples.
We then analyze the average fraction of objects collected while navigating the grid. The evaluation on this subtask also follows a similar "phase transition" at 10 examples. However, whereas Mix+ immediately achieves 90% 3 x 10a examples and continues improve to 96% 24 10* examples, Text and Image supervision fail to improve beyond 50-70%. This subtask becomes strong bottleneck for Text and Image supervision which prevents them from improving S2H generalization performance.
Finally, we analyze the average number of obstacles that the model passes through. Across any of the three types of supervision, the metric improves with more training data. However, this metric drops as low as 0.12 for Mix+ at 24 x 10 examples, whereas Text supervision only achieves 0.78 and Image supervision achieves 0.67.
Text
Image
Mix+
Reach Destination?
Avg. Fraction of Objects
Avg. Number of Obstacles
1.0
1.0
4
0.5
0.5
2
0.0
0.0
0.5
1.5
24
0.5
1.5
3
12
0.5
1.5
6
24
Number of Training Data (x109)
Figure 31. Analysis of failure modes on Grid Navigation: (Left) Whether model generates a sequence of actions that leads to the destination; (Middle) Average fraction of objects collected; (Right) Average number of obstacles passed through. Models trained with Text and Image fail to improve beyond a certain threshold for all three subtasks.
K.3. Visual Analogy
We analyze the outputs of Text on HARD-text, Image on HARD-image, and Mix+ on both HARD-text and HARD-image, where all models have been trained on 12 10 examples. Specifically, we analyze the CoT trace, focusing on the following structural steps as introduced in Appendix E.3 earlier:
To reason about examples:
given an attribute (e.g. shape.type), the model first needs to correctly enumerate the attribute values (e.g.
ci circle) for each image in the examples;
2. the model then needs to decide whether the values in all three images of that example are consistent with logical relation (e.g. XOR);
after repeating the process for both in-context examples, the model summarizes the two relational patterns (d1, and (d2, r2) for the examples;
4. finally, the model needs to identify the target relation 71 = Tquery from the examples.
To reason about the query: the model needs to correctly enumerate the attribute values for each image in the query similarly.
To reason about the options:
1. assuming the query when combined with each option follows relational pattern (domain d, relation r) (e.g. (line type, XOR), the model needs to identify the correct values of the attribute domain d for each option image and the correct relation r;
the model also needs to reason whether the identified relation r the desired target relation Tquery-
46

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Table 14. Analysis of failure modes on Visual Analogy:: Models are trained on 12 x 10" examples of Text, Image, and Mix+ supervision evaluated corresponding HARD inputs. means evaluation considered in-domain, as Mix+ supervision contains HARD-text examples in training. To evaluate the entire CoT (second to last row). we check if the generated output contains all the correct values in reasoning steps about the examples, query, and options. The main sources of emors for each type supervision are highlighted.
Error rate (%)
Types of failures
Text (text)
Image (image)
Mix+ (text" /image)
type values color values
0.0
0.0
0.0/0.0
0.0
0.0
0.0
size values
29.6
26.6
0.0
quantity values
29.6
26.2
0.0 0.0/37.5
Reasoning about examples
position values
29.6
26.2
0.0
held-out re)
86.8
81.0
0.0 42.2
d1 # d2 relation
86.8
80.8
0.0 23.8
35.2
34.4
0.0
type values
0.0
0.0
0.0
color values
0.0
0.0
0.0
Reasoning about query
size values
0.4
7.6
0.0/ 16.8
quantity values
0.4
7.6
0.0 16.2
position values
0.4
7.6
0.0 16.2
attribute domain
79.8
65.2
0.4/ 44.2
attribute values
8.4
32.0
0.0/ 65.0
Reasoning about options
relation
71.4
82.4
0.2/45.0
identify solution
45.2
51.8
0.2 21.0
CoT
100.0
100.0
0.4 79.8
Exact match
100.0
100.0
0.4 64.6
Mix+ supervision enables significant improvement reasoning steps that require compositional generalization where Text and Image supervision fail: As shown in Table 14, we observe that models trained with Text and Image struggle primarily to identify the correct held-out relational pattern (d;, T;) for in-context examples, and in particular to recognize d * d2, that is, the two examples vary along different attributes, with both error rates 2 80%. These two sources of error correspond exactly to the differences between the SIMPLE and HARD split of Visual Analogy, which requires the model to generalize compositional manner. With Mix+ supervision, the model significantly improves on these steps with a much smaller error rate of 42.2% in identifying the held-out (d,, r:) and 23.8% recognizing di + d2-
We observe that even with consistently higher error in identifying attribute values, models with Mix+ supervision achieve lower error rate in both CoT and exact match compared to their counterparts with Text and Image supervision. This makes sense since reasoning depends more identifying the correct logical relation than on identifying the correct attribute values. Although achieving the latter can be an important reasoning step, it is not necessary condition to arrive at the correct solution.
Visual Analogy focuses more abstract relational reasoning rather than object detection:
We also note that the error rate CoT can be higher than the error rate in exact match. This indicates that in some cases the model can still arrive the correct solution even though it makes slight mistakes the reasoning trace: for example, can still conclude with the correct relational pattern without identifying all the attribute values correctly.
Even with Mix+ supervision, the model still exhibits sensitivity to CoT templates and hallucinations: Interestingly, find that the rate in identifying values of si ze, quantityy and position consistently similar. Upon manual inspection of the model output, find that models fail to switch between different reasoning templates about shape and line objects: while the general templates for the two object types are similar, the model needs to reason about five attributes for shapes and only type and co colox for lines. With Mix+ supervision, models can still be sensitive to this small difference in CoT templates and hallucinate about undefined a size, quantity. and position attributes of the line objects. This highlights that models with Mix+ supervision are still brittle.
47

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
The image shows the sales data different products in different months. For readability. values cells have been replaced by their English names, e.g. represented by ZERO. There path of highlighted cells (colored in yellow) starting from (Radiate Tool, January) ending at (EmberGadget, August). Please return the total of values in highlighted cells. Provide
answer as 'Answer: [total].
Rewember
Desember Goteter
March
Fetnerry
Aart
RadisteTool
OME
FOUR
TWO
SEVEN
N1ME
N1142
EIGHT
EIGHT
Shimmerrggarrtuu
7630
1W0
F30
04F
Nitooll
0.4
34C
NRE
FOUR
ZERO
ZERO
ONE
MNC
SEVTS
ONE
EmberGadget
SPVEM
FOUR
SPVEN
SPVPN
THASE
THRTF
StrteContrattion
Panmoouniintrumeet
THARE
EIGHT
TWO
MANT
SuperiorGadgett
2640
RMC
1W0
1900
ERGHT
FAEE
FAVE
2OUR
1. Text Conversion
given table, highlighted (colored yellow) marked with symbol next the values.
image
Vbegini tabular); (1er111:111e)
November Janusty
9 W December
Uctober
March
June
rebruary
August Agril V..
kadiatelool
Onferon Trat
Shimme:Apparatur
6
XiTool 5
EnberGadget
9
sizzleconzrapiicn
BaramountInatrumne
BuperiarGadge:
QuantunApparatua
kend tabular)
leaption (nfrinneeee ssles Data for products in different monthe) \aabel tabibusiness sales)
tablet
2. CoT
We enumerate the relevant now indices, column indices, row names, column names, and their corresponding values.
Row Index, Column Index, Row Name, Column Name, Value Rodi Omnicrennnstrument
January
2
January
ShimmerApparatus
January
Xilool
December
EmberGadget
January
QuantumApparatee
August
SuperioeGadget
August
Puramoumtinstumment
August
SizzleContraption
August
EmberGndget
August
3. Prediction
The total
Answer: 95
Figure 32, A SIMPLE example from Table Readout.
48

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
The image shows the sales data different products in different months. For readability. values cells have been replaced by their English names, e.g. represented by ZERO, There path of highlighted cells (colered in yellow) starting from (esiDevice, September) and coding (Delta Tool, June). Please retum total all values highlighted cells. Provide
[tota]].
Decamber
Ocoieer
Fabrarry
March
SnruDortex
FOUR
CNE
SFVTR
FOLB
FNT
2180
9X
TWO
WaweTool
33/6M
2210
HIGH
2t11
IHH22
23R0
HOUK
IH9Z2
Hachhall
TIREE
504
1OUR
SEVDN
TWO
FOUR
FOUR
1WO
FOUR
ONC
T-RFF
7FRO
TWO
0NF
7580
DIGHT
POUR
1VO
TWO
Bettstee
ApdgW
OKE
FIGHT
SEVTN
THRET
THRRS2
2550
TWO
EIGHT
FIV
TIREE
TWO
FMZ
THAEE
SEVEN
THREE
FaCeelcs
FOUR
MNF
FOUR
7FRD
20UR
SPAFN
RMF
2FR0
SPVRM
Text Conversion
Convert image into a latex code of the given table, where highlighted cells (colored yellow)
marked with symbol the values.
bbggn[[tabls [bc]]
ccentering
beggin
begin[tabular]] tabular) (1111111111)
September December July
February
May
Januazy Karch \)
shiaeDevice
Mavelfool 6
0 &
FlashIsol
Scorchitachine 6 1
EliteMechanise
Ipesiooninstrumenn
2
DeltaTool
ApexMidge: 6
PrierSadget:
PsiDevice
(end(tabular)
leaption /Buioon//Binssss Salen Data for producta in. different nonths]
Lend table)
CoT
enumerate the relevant row indices, volumn indices, row namnes, column names,, and their corresponding
Row Index, Column Index, Row Name, Column Name, Value
10
PsiDevice EmberGadget
September
September
ApexWidget
September
DeltaTool
September
Epsilonlastrument
September
PsiDevice
May
FmberGadget
May
Apex Aidxeidget
May
DellaTool
May
Dolu
June
Prediction
total
4-2-7+4+99 1+0+++++ 9+ 7 50355111111 11111911 233.
+6+9+2+34
Answer: 233
Figure A HARD example from Table Readout.
49

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
image shows dimensional grid. The grid is filled up with objects, which you will be asked to recognize and collect, and obstacles, which you should avoid. Possible objects and obstacles given follows:
Objects: heart, crown, flag, star, flower, umbrella, plane, phone, spark, diamond, queen, hammer, club, gear, arrow, sum, bishop, note, coffee, anchor, cloud, pawn, castle, horse, infinity, moon, approx, integral, product, Obstacles: dot, cross, square, triangle, plus Cells that don't contain any object obstacle left blank. traveller starts at the source (colored in blue) and wishes to go to the destination (colored in green). Your task to give each indicating a movement by cell that the traveller needs follow. The traveller must recognize all objects the grid, collect them avoid obstacles in the process. Provide the final answer as 'Answer: [list of actions]'. path exists, output 'Answer: No path exists'.
of actions left right up down)
Text Conversion
3. Prediction
Convert the image into text version table, where cells colored in blue, green are respectively marked as S, D. Blank cells are given by O. Objects and obstacles mentioned by their names.
Answer: up up left up up left down right down right down down left left left up left left
& &
& & F& & & & & O O & square & square & square & O & square & O & O & square & square
& square
& square square square & flag square square
square square
square & O & square & O & O& O O square 0 h square & square & square & &
F square
CoT
d, looaay
left: closed( square)
down: okay
We follow depth first search,
d. f. Collect flag
visited
always moving towards the destination.
left: d, okay
down: f, closed( square)
left: f'closed( square)
d, e: left: d, okay
right: Fokay
visited
Source: h. g) Destinations( e. b) Collect objects: flag( I)
f.
down: h, okay
left: e, visited
h. g:
Avoid square
h, g:
left: closed( square)
up: d, f fviiited
g.
up: g okay
down: e, closeds square)
down; f, okay
left: g. okay
g. g:
up: closed( square)
f.
g.
up: g okay
right: visited
left: f, closed( square)
up:
okay
f.
No available actions.
up: visited
f,
closedí square)
retrace step
down: g. f'closedi square)
left: okay
left: f, Tokay
d,
right: f, okay
f:
f.
left: d, visited
f.
up: okay
up: fokay
down: okay
left: f. visitee
f.
up: closed( square)
left: okay
Figure 34. A SIMPLE example from Grid Navigation.
50

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
The image shows 2 dimensional grid. The grid is filled up with objects, which you will be asked recognize and collect, obstacles, which you should avoid. Possible objects and obstacles are given as follows: Objects:
heart, crown, flag, star, flower, unbrella, plane, phone, spark, diamond, queen, hammer, club, gear, arrow, sun, bishop, note, coffee, anchor, cloud, pawn, castle, horse, infinity, moon, null, approx, integral, product, sum
Obstacles: dot, cross, square, triangle, plus Cells that don't contain any object or obstacle blank. traveller starts the source (colored in bluc) and wishes to go to the destination (colored in green). Your task is give actions ( lefft right up down) each indicating movement by cell that the traveller needs to follow. The traveller must recognize all objects in the grid, collect them avoid obstacles the process. Provide the final answer as Answwr: [list of actions]. If no path exists, output 'Answer: No path exists'.
*
*
X
X
*
X
X
X
+
*
o
1. Text Conversion
3. Prediction
Convert the image into a text version of table, where cells colored
in blue, green respectively marked as S, D. Blank cells are given by O. Objects and obstacles are mentioned by their names. & & h & & d & f& g& h & &j & & plus & & O O queen & O& & cross
Answer: left up left down down down right right right right right right left up up up up left left down
& plus
cross
& 0
plus
down left left down left right right right right right up right up right right down down right
d cmes & & triangle cross & castle & O O cross &
& &
O cross & cross &
triangle &
cross triangle & triangle
& cross & cross & plus & O& cross
i& O & gear & spark O&ORO
2. CoT
We follow depth first search, always
i,
okay
right: c, L'okay
moving towards the destination,
right: okay i,
g: Collect queen
I:
Sources g, d) Destination: c, k)
left: a, Fooay
right: g okay
Collect objects: geart i, e) spark( i, h)
right: okay
a.
g:
castlia g) queen( a, g) sun( d, b)
Collect gear
left: a, okay
right: c, h closed( plus)
Avoid triangle plus
right: fokny
a.
up: b, okay
i.
down:
g:
down: h, closed( plus)
right: okay
b,
right: okay
right: clased plus)
g:
left: b, d closed( cross)
b,
left: okay
right: h okay
down: c, okay
right: b, closed cross)
e.
i, h: Collect spark
c.
down: c. closedí plus)
closed( cross)
ug: okay
left: v, okay
up: a, okay
right: g. visited
h.
d:
h:
up: f. okay
closed( cross)
left: okay
right: a, okay
f.
left: h, okay
e,
down; visited
h,
down: d, c okay d, c:
down: b, closed cros))
right: f, closed triangle)
up: okay
right: a. okay
left: okay
g. g:
left: d, okay
a.j:
up: okay
d, Collect
down: b, okay
down: okay
g:
right: okay
b.j:
g okay
d.
down: e, okay
right: visited
right: closedí triangle)
down: h, b okay
d. okay
c. okay
right: k okay
h,
4, g: Collect castle
c, c: right: okny d:
right: e closed( cross)
up: c, okay
down: okay
c, g
b.
up: A.g okay
right: c. okay
right: i. okay
b,
Pigure 35. A HARD example from Grid Navigation.
51

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
The image shows a a puzzle in by grid followed by options. The puzzle consists of 2 examples (row and 2), query (row 3), and four options. Fach example contains three images following relation along certain attribute, and this relation is consistent across all examples. The query contains two images. Analyze the
A
A
changes the following attributes for cach example: line type, line color, shape type, shape color, shape size, shape quantity, shape position, and consider the relations: Progression, XOR, OR, and AND. Progression requires the value of a certain attribute to strictly increase decrcase, but necessarily by fixed amount. Please provide your predictions in the format Exxmmpe i: (attribute, relation)' each example and similarly for options. Provide the final answer as 'Answer: [correct option]'.
A
O
?
Text Conversion
CoT
Convert the image into a text version the puzzle.
We first analyze examples by considering the applicable attributes and relations:
Examplc I:
Example
Panel
shapes; type: circle, color:
Image cirele, triangle
90, 255, size: 27, position: 0),
Image 2: trimgle, rectangle
Image 3: circle
(0, 2)
pollem.
Panel 2:
Example 2:
position:
Panel I:
Image 0),
Imuge 2: (0, 2), (1,
Image 3: (0,
Query: Panel
This suggesis the AND relation.
Example 2:
The examples suggest following (shape position, AND), (slape position, AND)).
Options:
common relation.
Option I:
Now, analyse the images the querys
shapes; type: pentagon
type:
Image hexagon, triangle
Image 2: triangle, rectangle
color:
3. Prediction
Looking the options, find:
Example 1: (shape position, AND)
Option has shope color 189, 90, O, so consistent with the OR
Example 2: (shape position, AND)
relation slape
target
Option 1: (shape color, OR), not consistent
Option shape pasition of(1 1J, consistent with the AND relation shape pesition, and AND the rolation. We conclude that option the answer since consistent with the relation.
Option (shape color, XOR), not consistent
Option (shupe type, XOR), not consistent
Option (shape position, AND),
Answer:
Figure 36. A SIMPLE example from Visual Analogy: The common relation is AND and the domains are di d2 dhuarr shape quantity, and the combinations (d, are not in the held-out set S {(line type, XOR), (line color, OR), (shape type, AND), (shape size, XOR), (shape color, Progression, (shape position, OR), (line type, AND), (line color, Progressio.).
52

---

Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
Question
The image shows a a puzzle in by grid followed by options. The puzzle consists of 2 examples (row and 2), query (row 3), and four options. Fach example contains three images following relation along certain attribute, and this relation is consistent across all examples. The query contains two images. Analyze the
changes the following attributes for cach example: line type, line color, shape type, shape color, shape size, shape quantity, shape position, and consider the relations: Progression, XOR, OR, and AND. Progression requires the value of a certain attribute to strictly increase decrcase, but necessarily by fixed amount. Please provide your predictions in the format Fxxampc i: (attribute, relation)' each example and similarly for options. Provide the final answer as 'Answer: [correct option]'.
?
Text Conversion
CoT
Convert image into a text version puzzle.
We analyze examples by considoring applicable attributes and relations:
Example I:
Example
Panel I:
Image L: diamond lincs, borizental line
type: diamond lines, horizontal line, color: 189, 135
Image 2: vertical line
Image falling diagonal
pattern.
Example 2:
color:
Panel 1:
Image 1: 189, 135
shapes: type: pentagon,
Image 2: 189
Image 3:
Query:
suggests the
Panel
Example
The examples suggest following puttern:: (line color, AND), (shape color, AND).
Options:
We conclude that AND is the common relation. Now, we unalyze the images the query: type:
Option 1:
type: V-shape facing left, color:
Image falling diagonal V-stape facing
Image 2: horizuntal line, falling diagonal line
3. Prediction
color:
Looking at the options, we find:
Example (line color, AND)
Option hus line color of 135, so consistent with the XOR
Example (shape color, AND)
relation on line color, but XOR not the target relation.
Option 1: (line color, XOR), consistent
Option (line color,
Option has line color of line color, and AND the target relation. We conctude that option the answer since it is consistent with the AND relation.
consistent with the AND relation
Option 3: (line type, XOR), consistent
Option 4: (line color, AND), consistent
Answer: 4
Figure 37. A HARD example from Visual Analogy: The common relation is AND and the domains distinct: di line color, d2 = shape position, dqeery = line coler, and the combinations (d, r) are in the held-out set S = {(line type, XOR), (line color,oR OR), (shape type, AND), (shape si size, XOR), (shape col or, Progress on), (shape pesition, OR), (line AND), (line color,,Fogression Progressioo) Note that the pattern for the confounding options may not be in S.
53


Reviews:
[{"review_id": "BzkfL7LjkU", "reviewer": "Reviewer_fXux", "summary": "This paper investigates the \"modality imbalance\" problem in Vision Language Models (VLMs), where models perform worse on visual reasoning tasks compared to equivalent text-based tasks. The authors introduce a framework for studying simple-to-hard (S2H) generalization in VLMs using three synthetic tasks: Table Readout, Grid Navigation, and Visual Analogy. Each task has SIMPLE and HARD versions with equivalent representations in both text and image modalities. \n\nThe paper's main contribution is the discovery that VLMs exhibit modality imbalance in S2H generalization, with models able to generalize from SIMPLE to HARD examples in text but failing to do so in vision. Through experiments and gradient alignment analysis, the authors reveal that explicit image-to-text conversion is crucial for transferring reasoning capabilities from text to image modalities. They find that this conversion process can be internalized at test time, and identify gradient alignment measures that predict the effectiveness of different training strategies. Their work provides insights into the mechanisms of cross-modal learning and how the modality gap can be bridged through different training approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Response to Reviewer fXux\n\nWe thank the reviewer for their careful review of our paper. Please find responses to your concerns and questions below.\n\n# Weaknesses\n> Experiments are limited to one architecture\n\nWe used EAGLE because it was the best performing open-source model which also released all details on **its training data and the set of hyperparameters**.\n\nTo test the generalizability of our results, we ran additional experiments by continual finetuning 3B, 7B models from the Qwen2.5-VL family. See the accuracy on HARD-image below. CTR, TR, GN, VA are respectively short for Consecutive Table Readout, Table Readout, Grid Navigation, and Visual Analogy. On the text version of CTR, we notice that the 3B model doesn’t show S2H generalization (which is expected since task difficulty can be model specific). However, across TR, GN, and VA, we generally observe the same conclusion - **Mix+ achieves S2H generalization on the image modality and Align-Mix+ improves the generalization**. \n\n### Qwen2.5-VL-3B-Instruct\n|Supervision|CTR(30k)|TR(30k)|TR(60k)|GN(30k)|GN(60k)|VA(30k)|VA(60k)|\n|:-|-:|-:|-:|-:|-:|-:|-:|\n|Image|0|11|10|22|22|0|0|\n|Text+Image|1|7|6|0|14|0|0|\n|Image-via-Text|1|12|8|13|14|0|0|\n|Mix|1|11|10|14|16|0|1|\n|Image-via-Text+|0|**81**|90|67|58|**48**|**48**|\n|Mix+|**4**|78|86|77|91|20|27|\n|Align-Mix+|-|66|**91**|**80**|**91**|38|42|\n\n### Qwen2.5-VL-7B-Instruct\n|Supervision|CTR(30k)|TR(30k)|TR(60k)|GN(30k)|GN(60k)|VA(30k)|VA(60k)|\n|:-|-:|-:|-:|-:|-:|-:|-:|\n|Image|0|18|17|14|29|0|0|\n|Text+Image|4|8|5|6|11|0|0|\n|Image-via-Text|36|9|13|13|18|0|0|\n|Mix|52|8|17|15|12|0|0|\n|Image-via-Text+|**73**|82|88|**75**|67|**41**|**44**|\n|Mix+|72|13|66|69|**85**|12|17|\n|Align-Mix+|-|**93**|**92**|36|58|25|34|\n\n> Experiments are limited to synthetic settings\n\nPlease see our reply to the first question from Reviewer NF8o.\n\n> Experiments are only on vision-text modalities\n\nOur motivation came from works that identify the modality gap in VLMs. Additionally, we require an open-weights model that has strong enough reasoning capability in each modality, which is available in a VLM. We would love to explore more modalities in the future when open models can also incorporate multiple modalities. \n\n> Task-specific chain-of-thought templates might limit generality of observations\n\nBy keeping the chain-of-thought templates fixed throughout the task, we were able to accurately probe the existing modality imbalance. In the future, we would like to explore more on how the results would generalize to a more flexibly generated CoT trace.\n\n# Suggested Questions\n\n> Can gradient alignment be used to measure cross-modal alignment?\n\nWe found that different supervision strategies could be distinguished only when analyzing gradient alignment between SIMPLE and HARD examples within each modality independently. This may be attributed to our use of a local definition of gradient alignment—examining gradients at each step in isolation. Capturing cross-modal alignment likely requires a more global perspective, as gradients at earlier time steps (e.g., on text) can influence gradients in later steps (e.g., on image). We consider the development of such alignment measures an important direction for future work.\n\n> Consecutive Table Readout vs. Table Readout\n\nBoth tasks involve sequentially reading highlighted cells in a table. In Consecutive Table Readout, the path is always consecutive row-wise, and the model knows where the next cell should be. In Table Readout, the path can take turns in arbitrary directions at arbitrary locations, so the model needs to additionally check **which adjacent cell is highlighted** at every location.\n\nIn the final version, we will list all 4 tasks in the abstract or rename Table Readout to e.g., Path Table Readout and use Table Readout as an umbrella term for both Consecutive and Path Table Readout.\n\n> Why does internalizing CoT fail\n\nWe observed that CoT is crucial for the model to transfer its reasoning from text to image input by repeating the same reasoning steps. Our hypothesis is that during SFT, the model establishes an implicit equivalence between reasoning on text and image inputs via the common CoT tokens. \n\nThere may exist multiple ways, e.g., a more fine-grained curriculum, to internalize CoT. For the scope of this paper, we limit our exploration to simple settings with randomly shuffled data and progressive internalization, and leave a complete study for future. \n\n> Can you include results on Grid Navigation in Figure 6\n\nThe model already performs nearly perfectly on Grid Navigation with 60k examples of Mix+, so we did not compare with Image-via-Text+.\n\n> What is the meaning of Mix$\\to$Mix+ in Figure 8\n\nMix$\\to$Mix+ means we took an intermediate checkpoint of Mix and resumed training with Mix+. This setup allows us to assess the effect of introducing Hard text examples on the loss curves of Mix training, enabling a fine-grained analysis of the differences between Mix and Mix+ (lines 334-341).", "claims_and_evidence": "The claims made in this paper are well-supported by comprehensive empirical evidence. The authors:\n\n1. Demonstrate the modality imbalance problem through controlled experiments on their synthetic tasks, showing significant performance gaps between text and image modalities \n2. Show that their proposed Mix and Mix+ strategies improve S2H generalization in images by effectively transferring reasoning capabilities from text to vision. \n3. Provide gradient alignment analysis that convincingly explains why these training strategies work.\n4. Validate their approaches across multiple tasks with varying complexity, showing consistent improvements over baselines.\n5. Conduct thorough ablation studies on key components like chain-of-thought reasoning, data composition, and text warm-up pretraining.\n\nThe authors are careful to test their approaches in both standard and challenging scenarios (missing modalities). The evidence presented is comprehensive and the conclusions are well-supported by the experimental results.", "methods_and_evaluation_criteria": "The methods proposed in this paper are novel and well-designed. The authors:\n\n1. Create three synthetic tasks that allow for controlled study of S2H generalization in both text and image modalities, with difficulty levels that can be systematically tuned to test generalization capabilities.\n2. Develop multiple training strategies (Mix, Mix+, Align-Mix+) that progressively address more challenging S2H generalization scenarios.\n3. Introduce gradient alignment measures that provide theoretical insights into the effectiveness of their approaches.\n\nThe evaluation criteria are appropriate and comprehensive:\n\n- Performance is measured on both SIMPLE and HARD examples across modalities\n- Comparisons include baselines and ablations to isolate the impact of each component\n- Gradient analysis provides mechanistic understanding of the approaches\n\nThe authors carefully control for factors like training data size to ensure fair comparisons. They also test on both complete and missing modality scenarios to demonstrate the robustness of their approach. Additionally, they provide extensive ablation studies and supplementary experiments in the Appendix, including text warm-up pretraining, explicit and implicit CoT ablation, and multi-task training effects, which further substantiate their claims and provide deeper insights into the mechanisms behind their approaches.", "theoretical_claims": "I reviewed the theoretical claims related to gradient alignment in Section 5. The proofs for Theorem H.1 appear sound. The theorem establishes that (given a small enough learning rate) the gradient alignment score predicts how much loss reduction on HARD Image examples will be achieved by gradient updates from SIMPLE Image examples relative to updates from HARD Image examples directly.\n\nHowever, as the main novelty is on the transfer between HARD text and HARD image examples (without training these HARD image examples), I am not sure whether the analysis is thorough enough. Cross-modality alignment should be examined.", "experimental_designs_or_analyses": "The experimental designs are sound and well-executed. The authors use only one model architecture (Eagle-X2-LLAMA3-8B), which is a limitation but understandable given computational constraints. \n\nTheir synthetic tasks appear simple, but this simplicity actually strengthens the study by enabling precise isolation of reasoning capabilities and clear comparisons across modalities. The authors carefully control for data quantity and quality throughout, ensuring valid comparisons and reliable conclusions.", "relation_to_broader_scientific_literature": "This paper bridges several key research areas in multimodal learning. It introduces controlled algorithmic “visual” reasoning tasks that enable precise multi-step reasoning evaluation, addressing a significant gap in VLM assessment. (add sth like : Furthermore, they allow variable difficulty) It extends work on modality imbalance by uniquely addressing this issue in adapter-based VLMs through knowledge transfer from text to image modalities, moving beyond traditional approaches such as modulating learning rates (Peng et al., 2022). Finally, it extends simple-to-hard generalization research from language-only settings (Abbe et al., 2024; Zhou et al., 2024) to cross-modal contexts.\n\nPeng et al., 2022 : Peng, Xiaokang, et al. \"Balanced multimodal learning via on-the-fly gradient modulation.\" *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*. 2022.\n\nAbbe et al., 2024 : Abbe, Emmanuel, et al. \"Generalization on the unseen, logic reasoning and degree curriculum.\" *Journal of Machine Learning Research* 25.331 (2024): 1-58.\n\nZhou et al., 2024 : Zhou, Hattie, et al. \"What algorithms can transformers learn? a study in length generalization.\" *arXiv preprint arXiv:2310.16028* (2023).", "essential_references_not_discussed": "The literature review seems comprehensive and contains all the relevant references. The authors (in the main text and Appendix B.) have appropriately cited related works in modality imbalance, generalization transfer between input modes, and S2H generalization.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The paper introduces a novel and practical approach to address an important problem in multimodal learning.\n2. The synthetic tasks are well-designed to isolate and study S2H generalization.\n3. The gradient alignment analysis provides valuable mechanistic insights.\n4. The approaches are effective across a range of tasks and scenarios.\n5. The paper includes thorough ablation studies and analyses.\n6. The proposed methods (Mix, Mix+, Align-Mix+) are simple yet effective, making them likely to be adopted in practice.\n\n**Weaknesses:**\n\n1. The experiments are limited to one model architecture\n2. While the synthetic tasks are useful for controlled study, more real-world examples would strengthen the paper's impact.\n3. The paper focuses mainly on vision-text modalities; it would be interesting to see if the insights generalize to other modality pairs.\n4. The approach requires task-specific chain-of-thought templates, which might limit its generality.\n\nOverall, the strengths significantly outweigh the weaknesses, and the paper makes a valuable contribution to the field.", "other_comments_or_suggestions": "I wonder if computing gradient alignment scores between modalities (rather than just between SIMPLE and HARD examples within a modality) might offer valuable insights. This cross-modal gradient alignment could potentially reveal how information transfers between vision and language representations for identical inputs presented in different formats.\n\nIn section 4.2 discussing the benefits of two-phase training, it would be helpful to cite the specific figure or table showing the 76%, 96%, and 56% performance metrics mentioned. This would make it easier for readers to connect your analysis with the supporting evidence.\n\nIn Figure 6, you only showed results for Table Readout and Visual Analogy. Could you also show the remaining task (Grid Navigation) to provide a complete picture of how Image-via-Text+ performs across all tasks?\n\nI am a bit confused about the naming here : are “Consecutive Table Readout” the same as “Table Readout”? Appendix D suggests that they are a simliar but different tasks. However, in the Abstract, you state that you test on three tasks, which make this even more confusing (is the Consecutive Table Readout a “fourth” task?). Clarifying their difference in the main text would help with clarity. \n\nI'm particularly intrigued by your findings on Chain of Thought reasoning. Given your ablation studies in Appendix I.7 showing that attempts to internalize CoT consistently fail to achieve image S2H generalization, could this suggest something fundamental about how reasoning occurs in these adapter-based VLMs? Perhaps the architectural design, with vision adapters attached to a text-based backbone, necessitates text-structured reasoning paths that CoT provides explicitly?", "questions_for_authors": "I may have missed it, but it seems that Figures 5 and 7 themselves are in the main text but not the figures are not referenced anywhere in the main text, making it a bit confusion to read. (For example, it seems that the last parts of Page 6 are referring to Figure 7, but this is never mentioned in the main text)\n\nRegarding Figure 2, I noticed the grey dotted line representing text supervision on the right panels lacks a corresponding entry in the legend, though it's explained in the caption. Adding this to the legend (with a name such as “Text Supervision on Text”) would improve readability at a glance.\n\nSimilarly, I believe that the description of “Text” (grey dash) in Figure 5 could be more clearly stated. In the caption, it says that they are S2H generalization on text from Text supervision, but the legend seems to imply that “Text” is S2H generalization on Image from Text Supervision. As with the Figure 2, I believe giving the “Text” legend name a clearer name (e.g., “Text supervision on Text”) would be better.\n\nIn Figure 8 in the legend, there is “Mix → Mix+”, but there is no explanation of what this means.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L7qrXj0dNl", "reviewer": "Reviewer_pko5", "summary": "This work investigates the modality imbalance in simple-to-hard generalization of VLMs. The main findings are: Explicit image-to-text conversion is important in improving S2H generalization on images, and the conversion can be internalized at test time.\n\n## update after rebuttal\nThe rebuttal partly solves my concerns. I will maintain my evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Response to Reviewer pko5\n\nWe thank the reviewer for their thoughtful comments and suggestions regarding the paper. Please find our responses to your comments below.\n\n# Weaknesses\n> Experiments are limited to one base model\n\nPlease see our reply to Reviewer fXux. We observe the same conclusion from Qwen 2.5 VL 3B and 7B.\n\n> The simple and hard table readout tasks are more like scaling up the recognition and counting complexity, rather than requiring significantly better reasoning capabilities, e.g., chess puzzles.\n\nWe agree that the reasoning in Table Readout is simpler compared to e.g., solving chess puzzles. However, such simple synthetic tasks allow us to clearly define SIMPLE and HARD examples and enable a more controlled analysis of model behavior. \nThat being said, Table Readout does not only involve recognition and counting, but is analogous to Grid Navigation. At any cell, the model needs to know where to read the next number from. E.g., to realize the next number is on the left, the model needs to check that up/right/down is not highlighted but left is. This decision process mirrors the one in Grid Navigation, where all non-highlighted cells are “obstacles” to avoid. \n\n> Please discuss IsoBench\n\nThank you for suggesting the reference. We will include the citation in lines 020-021 (page 1, right).\n\n# Questions\n> How would ‘Hardness’ impact the performance? The choice of threshold to split SIMPLE and HARD needs to be clarified.\n\nAs S2H generalization measures OOD performance, “hardness” is relative with respect to the difference between training and test distributions and is model-specific. On Consecutive Table Readout, we consider **two hardness levels**: HARD (15-20) and (25–30), each reading 15-20 or 25–30 consecutive numbers (line 150) and find **modality gap gets wider as the task becomes more challenging** (line 210-212, Fig. 2). For the other three tasks that also require compositional generalization, we decided the hardness by either increasing the number of components to be composed (Table Readout and Grid Navigation) or introducing held-out compositions (Visual Analogy).\n\nWe agree that ultimately the SIMPLE-HARD split is a matter of judgment. We tried to choose the most intuitive splits between SIMPLE and HARD while being aligned with existing studies (e.g., Hill et al., 2019 and Barrett et al., 2018). Further justifications for these decisions are in Appendix D and the specific thresholds in Table 1.\n\n> How did previous works approach S2H generalization in general? Any of them worth being used as a baseline (only for the text supervision)? \n\nIn Appendix B, we discuss several prior works on S2H generalization that explore ICL with scratchpads (Anil et al., 2022), positional encodings (Kazemnejad et al., 2024), train set priming (Jelassi et al., 2023), curriculum learning (Abbe et al., 2024), and looped transformers (Fan et al., 2024). However, as we focus on modality imbalance, we do not compare with these methods. The aforementioned works (1) focus only on LLMs and are not directly adaptable to a multimodal setup and (2) study only length generalization, but not compositional generalization. \nMore recent efforts such as self-improvement (Lee et al., 2025) and generalizable verifiers (Sun et al., 2024) examine S2H generalization in different setups that rely on curriculum learning with progressively harder tasks or require reward models. In contrast, our study is restricted to supervised fine-tuning (SFT) approaches.\nWhile we acknowledge that Mix or Mix+ may not be the optimal strategies for improving image generalization on our proposed tasks (line 429), it is still notable that reasoning transfer from the text to image modality—enabling S2H generalization on images—emerges naturally through autoregressive SFT. This, in our view, is both non-trivial and interesting.\nWe will include an explicit Related Works section in the main paper to clearly differentiate our contributions from prior work.\n\n# Comments on Presentation / Figures\nWe appreciate the reviewer’s detailed feedback on this matter. We hope to address these issues in the final version of the paper. \n\n>  The authors interchangeably use  “image” and “text”, which might create confusion\n\nTo reduce the number of new terms introduced, we used “Text/Image” (capitalized) for the supervision that trains on the corresponding modality. However, we understand the potential for confusion. \n\n>  The authors can do a better job to help the audience focus on the main storyline\n\nIn Section 1.1, we presented a compressed overview of the paper to direct the readers to relevant sections. In the final version, we plan to trim some of the ablations but instead expand on the discussions that better highlight the main storyline.\n\n[1] Lee, et al., Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges, 2025\n\nall other citations in Related Works / References of the paper", "claims_and_evidence": "Most claims are well supported by experimental results, while it can be improved by showing consistent results with other base models of different sizes and backbones.", "methods_and_evaluation_criteria": "The methodology is well designed and the evaluation metrics are appropriately selected.", "theoretical_claims": "I tried to check the proof in the Appendix F, I did not find obvious issues about it. But it is possible that I overlook some details.", "experimental_designs_or_analyses": "This work can benefit from more experiments with different settings of the threshold to split simple and hard examples, as well as insights from the “hardness” (degree of how hard it is) of the examples.", "relation_to_broader_scientific_literature": "The finding that image-to-text conversion can be internalized at test time is particularly insightful, which could potentially benefit further research in Mechanistic Interpretability", "essential_references_not_discussed": "- There are missing related works that directly relate to modality imbalance and VLM evaluation, including but not limited to:\n    - https://arxiv.org/abs/2404.01266", "other_strengths_and_weaknesses": "- The choice of threshold to split SIMPLE and HARD needs to be clarified.\n- From the examples in Figure 32-33, the simple and hard table readout tasks are more like scaling up the recognition and counting complexity, rather than requiring significantly better reasoning capabilities, e.g., chess puzzles.\n- This paper is generally well written and easy to follow. However, it’s hard for the readers to get used to the many interchangeably used “image” and “text”. For example, I find it difficult to distinguish whether the “text/image” denotes S2H setting or training strategy in Figure 2. It’s probably better to rename the types of supervision, especially (a) and (b).\n- The study on loss dynamics and gradient alignment is particularly interesting and insightful", "other_comments_or_suggestions": "- Although the authors pointed out using only one base model is a limitation, I would like to mention it here again for visibility. This work can be significantly strengthened if the authors can show the main results and the conclusion with other base models with different backbones and sizes. Especially when models scale up, findings might not be 100% consistent.\n- I understand that the authors want to compress as much crucial results and discussions as possible into the main content. However, my feeling of reading this paper is that too many concepts/discussions/paragraphs/subtitles seem to be equally important, but they are in fact not. I am not opposed to any style of writing, but the authors can do a better job to help the audience focus on the main storyline.\n- Note that not everyone will read the appendix, so the main content must be self-contained. Without a proper related work SECTION, it’s hard to position this work among many other works in the literature. For example, how did previous works approach S2H generalization in general? Any of them worth being used as a baseline (only for the text supervision)? Such questions need to be answered in the main content to make it self-contained.", "questions_for_authors": "How would the “hardness” impact the performance? How well can it generalize from simple to relatively hard vs extremely hard? Can relatively hard generalize to extremely hard?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Hpd8fBq9P", "reviewer": "Reviewer_d6eT", "summary": "This paper studies visual reasoning using vision-language models (VLMs). The authors focus on three tasks: Table Readout, Grid Navigation, and Visual Analogy. They run experiments under simple to hard settings to test each task's generalization.\nThey propose distilling knowledge from large language models (LLMs) by converting images into text and extracting chain-of-thought reasoning. Their findings suggest that (1) converting images to text can improve visual reasoning and (2) applying this form of knowledge distillation enables the VLM to learn the transferred chain of thought directly.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Response to Reviewer d6eT\n\nWe thank the reviewer for their comments and suggestions. Please find our responses to your comments below.\n\n> Is the paper related to knowledge distillation or prompt engineering?\n\nWe would like to clarify that **we do not employ any knowledge distillation**. All of our data (image, its text description, the CoT trace, and the solution) is generated with a Python script. When we say the reasoning transfers from the LLM to a VLM, we mean that the innate (or learned) reasoning capability of the LLM backbone (i.e., Llama-3-8B-Instruct) of an adapter-based VLM helps the entire VLM learn to perform the same task in the image modality. There is no external LLM/VLM here.\n\n> The datasets are self-created without detailed descriptions, and public benchmarks are not used\n\nWe propose to measure the modality imbalance in VLMs by looking at the different simple-to-hard generalization behaviors (e.g., length generalization, compositional generalization). For this, we need a dataset where 1) there is a clear level of difficulty and 2) each image has an equivalent text description. There is no public dataset that fits this description. We also mention in the Related Works paragraph that “current VLM benchmarks are often solvable without the visual input,” which makes our analysis impossible. Additionally, we provide details in generating the datasets in Appendix D and provide the example data points in Figure 32-37. We were planning on releasing the entire codebase with the final version of the paper, but here is an anonymous link to the code to generate the data: [https://github.com/asjdifpjadsi/VLM_S2H](https://github.com/asjdifpjadsi/VLM_S2H).  Also see our response to Q1 of Reviewer NF8o.\n\n> Please provide Comparisons with “Multimodal chain-of-thought in Language Models”\n\nWe thank the reviewer for the suggestion. We would like to clarify that our paper focuses on **modality imbalance** and its measurement through **simple-to-hard generalization**, aiming to improve a VLM’s reasoning ability on images to a comparable reasoning performance on **equivalent** text data. We would like to reiterate that  **we do not employ any knowledge distillation**.\n- Firstly, although both Multimodal-CoT and our paper use CoT to boost the model’s reasoning capability, we would like to point out that it is a very common technique in practice.\n- Multimodal-CoT would be mostly comparable to our Image-via-Text Supervision as both attempt to leverage extra text generation to assist reasoning. However, a significant difference is that Multimodal-CoT optimizes for CoT while ours only relies on CoT with a fixed template to assist long-chain reasoning. \n- Furthermore, our method converts an image to an **equivalent** text – a step that is later internalized – which is not equivalent to (either human-annotated or AI-generated) CoT. This conversion introduces no new information, so the help of text conversion in the model’s reasoning, if there is any, is entirely from a difference in modality, as opposed to a more optimal reasoning trajectory. Our work is more aligned with the idea of vision-depicting-prompting in Zhang et al. (2023) and many more we cited in the Related Works section (Appendix B, mentioned in page 8). Instead of prompting, we perform SFT and **propose a testbed to quantify** the benefit of an extra step of text conversion in mitigating **modality imbalance** in terms of **S2H generalization**.\n\n\n# Comments on Presentation / Figures\n> English needs further improvement and a thorough grammar check\n\nWe thank the reviewer for the comment. We have taken a lot of care when writing the paper. We will modify any remaining grammatical mistakes, if any, in our final version.\n\n[1] Zhang et al., Lost in Translation: When GPT-4V(ision) Can’t See Eye to Eye with Text A Vision-Language-Consistency Analysis of VLLMs and Beyond, 2023.\n\n[2] Zhang et al., Multimodal Chain-of-Thought Reasoning in Language Models, 2023.", "claims_and_evidence": "No, the paper is hard to read. It lacks clearly defined sections for the $\\textit{method, dataset, and experiments}$. Instead, these sections are mixed together, making it difficult to identify the evidence supporting their claims.", "methods_and_evaluation_criteria": "The datasets in this paper are self-created and not publicly available, and they lack clear descriptions. As a result, it is hard to assess the quality and difficulty of the datasets for Table Readout, Grid Navigation, and Visual Analogy. It would be better to propose a new dataset if none exists or to use existing benchmarks.", "theoretical_claims": "The paper does not present theoretical claims or proofs; it relies on experimental findings.", "experimental_designs_or_analyses": "The paper mixes experimental designs and methods, which makes it hard to follow the claims. For example, Section 3 should focus on the methods, but it gets interrupted by experiments (see Figure 2), so it's not clear how the methods lead to the conclusions.", "relation_to_broader_scientific_literature": "The paper studies how vision-language models (VLMs) reason with images, similar to the way large language models (LLMs) reason. It is relevant to chain-of-thought, prompt engineering, and knowledge distillation.", "essential_references_not_discussed": "Multimodal Chain-of-Thought Reasoning in Language Models", "other_strengths_and_weaknesses": "Strengths:\n1. The abstract is well written and easy to read.\n\nWeaknesses:\n1. The main sections are not clearly defined, and the English needs improvement.\n2. The paper deals with knowledge distillation and multimodal chain-of-thought but does not compare or discuss existing methods.\n3. The datasets are self-created without detailed descriptions, and public benchmarks are not used.", "other_comments_or_suggestions": "1. The paper does not separate methods, datasets, and experiments into clear sections. Instead, they are mixed together, which makes the manuscript hard to follow.\n2. The fonts in the figures are too small and difficult to read.\n3. The English needs further improvement and a thorough grammar check. For instance, the sentence in Line 106 (page 2, left column) is very confusing and its meaning is unclear. \"for tasks where the S2H generalization failed in both modalities, the same idea as (i) led to S2H generalization in the image modality after text-only supervision was used to inject reasoning capability on HARD task in text modality\", it's hard to understand what's the meanning.\"", "questions_for_authors": "1. The paper transfers the reasoning process (Chain-of-Thought) from LLM to VLM through tuning. However, it is unclear how this approach differs from [1].\n2. Since the work relates to multimodal Chain-of-Thought and knowledge distillation, it should discuss related work in these areas, such as the findings in \"Multimodal Chain-of-Thought Reasoning in Language Models\" [1].\n\n[1] Multimodal Chain-of-Thought Reasoning in Language Models", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "The paper does not have ethical concerns"}}, {"review_id": "HOTtbuREF3", "reviewer": "Reviewer_NF8o", "summary": "The paper investigates modality imbalance issues in existing VLMs by designing multiple tasks with diverse settings. Comprehensive experiments reveal interesting findings, and gradient analysis further illustrates how different settings impact the final results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' Response to Reviewer NF8o\nWe thank the reviewer for their time and effort to review our paper. Please find responses to your comments and questions below.\n\n# Questions\n\n> Experiments are restricted to Synthetic Tasks\n\nWe use synthetic tasks to enable controlled studies of modality imbalance and simple-to-hard generalization in VLMs. These tasks are carefully designed to require diverse skills such as OCR, spatial navigation, and visual reasoning, which are essential to real-world VLM applications. As shown in **Table 7 (Appendix I.1)** (referenced in lines 048–049), frontier VLMs struggle with these tasks, suggesting that even simplified settings can reveal gaps in current models. Importantly, incorporating these synthetic tasks into pre-training improves performance on a broad range of real-world benchmarks, show in **Table 6 (Appendix G.6)** (referenced in lines 427–428). Thus, the synthetic tasks that we use are relevant to the real-world applications and could be utilized to motivate training choices to boost VLM’s general abilities.\n\n> What are the challenges for the remaining modality gap?\n\nThank you for the interesting question. Our work highlights a modality gap in simple-to-hard (S2H) generalization between text and image inputs, which we attribute in part to current VLM training paradigms. Current VLMs rely on adapter-based architectures that loosely integrate small visual encoders with large pre-trained LLMs, which have been pre-trained separately on image and text pre-training data.\n\nOur approach uses image-to-text conversion to better align image inputs with the model’s strong capabilities in language, partially mitigating this gap. However, fully closing it likely requires rethinking the model design itself. Moving beyond adapter-based setups towards early-fusion architectures or joint multi-modal pretraining could help models learn more unified representations and improve generalization across modalities.\n\n> How do the authors propose to mitigate additional inference cost with image-to-text conversion\n\nWe would like to clarify that our proposed approaches, Mix, Mix+, and Align-Mix+ supervision types, use image-to-text conversion only during training the model (please see lines 151-152, 245-247, 256-257). During inference, the models trained with these supervision types **don’t explicitly perform image-to-text conversion**. Please see **Figures 3 and 6** in the main paper and lines 205-209, 257-264 for discussions on lengths of inference-time generations for Mix, Mix+, and Align-Mix+ supervisions. \n\n> Further analysis of gradient alignment for more models and datasets\n\nThanks for the question! We evaluated gradient alignment on the Consecutive Table Readout and Table Readout datasets. Due to the high cost of saving frequent checkpoints, we did not extend these studies to other settings. Instead, we focused on fine-grained alignment measures (see Figures 17–20) and provided theoretical support in Theorems H.1 and H.2. Together, these offer strong initial evidence for the robustness of our alignment metric as a generalization indicator.\n\n> More details on CoT Generation\n\nAll of our data (image, its text description, the CoT trace, and the solution) is generated with a Python script. When we generate an image (e.g., using matplotlib), we store relevant metadata (e.g., sequence of (row index, col index) of highlighted cells in Table Readout) and insert it in a fixed chat template. We give examples of our CoT data in **Figures 32-37** in the appendix. See Appendix D.6.2 (mentioned in footnote 1 of page 1) for alternate versions of CoT templates.", "claims_and_evidence": "Mostly, but with some caveats. While the experimental results support the idea that modality imbalance exists in VLMs and that Image-via-Text conversion helps alleviate this, the evidence could be more comprehensive. For instance, the paper focuses on synthetic tasks, which are controlled and may not fully represent the complexity and variety of real-world visual reasoning tasks. Also, the performance on the HARD-image tasks, even with the proposed strategies, is still far from perfect. This indicates that while the strategies help, they may not provide a complete solution to the problem, and there might be other underlying issues that are not addressed in the paper.", "methods_and_evaluation_criteria": "Yes, but limited by synthetic tasks. The methods and evaluation criteria (s2h generalization) are reasonable for controlled experiments. However, relying on synthetic tasks like Table Readout and Grid Navigation might limit the generalization of the findings. These tasks are constructed to highlight specific issues (such as reasoning across modalities), but they do not reflect the real-world diversity of tasks VLMs are typically applied to. More diverse, real-world tasks would provide a better test of the proposed methods' effectiveness and generalizability.", "theoretical_claims": "No, I did not identify any formal proofs within the text. However, the paper discusses theoretical aspects like gradient alignment and the internalization of image-to-text conversion, which are supported by experimental data and loss function analysis. These are more empirical observations than formal mathematical proofs.", "experimental_designs_or_analyses": "Sound but with limitations. The experimental designs are reasonable, but there are several points of concern:\n\n- Task limitations: As mentioned earlier, the tasks are quite specific and synthetic. The paper could benefit from including more varied tasks or even applying the methods to existing, large-scale benchmarks like VQA (Visual Question Answering) or other real-world multimodal reasoning benchmarks.\n\n- Modality imbalance: While the paper focuses on mitigating modality imbalance, the results show that the gap between text and image reasoning remains significant, even with the proposed methods. This suggests that the current approaches (Image-via-Text and Mix supervision) may not fully solve the problem. In fact, the model's heavy reliance on image-to-text conversion at inference time indicates that while the training can alleviate the gap, it doesn't eliminate it, which may limit the practical applicability of the approach.\n\n- Inference time cost: One major issue that the paper glosses over is the increased inference time cost due to the image-to-text conversion. While the authors acknowledge this, they do not explore alternative strategies to mitigate this overhead, which could be a significant limitation in real-world applications.\n\n- Gradient alignment study: The paper’s analysis of gradient alignment and its impact on S2H generalization is interesting but lacks depth. While the alignment scores provide some insight into how well the model is learning to generalize across modalities, the paper doesn't go into much detail on how these gradients are computed or how reliable they are as a measure of generalization. Moreover, this analysis could benefit from more ablations or sensitivity analyses to assess the robustness of the observed improvements.", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. **Originality and Innovation**:\n   - The paper introduces a novel perspective on mitigating modality imbalance in Vision-Language Models (VLMs), which is a relatively underexplored area. The idea of using simple-to-hard (S2H) generalization and specifically the concept of training with Image-via-Text supervision for improving VLMs’ reasoning capabilities across modalities is an innovative approach. This contribution is significant as it goes beyond just testing VLMs on a standard benchmark; it provides a systematic methodology to address a key issue in VLM performance.\n   - The idea of \"internalizing\" the image-to-text conversion is also quite creative, offering an elegant solution to reduce inference time, even if the initial training strategy is more costly. This speaks to a potential real-world application of reducing the computational overhead in production systems without sacrificing performance.\n\n2. **Experimental Rigor**:\n   - The paper presents a comprehensive set of experiments with well-defined tasks, which allow the authors to study the issue of modality imbalance in great detail. The ablation studies, gradient alignment analysis, and task-specific results provide useful insights into how various training strategies impact performance. These experiments help demonstrate the practical effectiveness of the proposed methods.\n\n3. **Clarity**:\n   - Overall, the paper is well-structured and presents its methods, experimental design, and results in a clear and logical manner. The methodology is sufficiently detailed for readers to follow the steps and replicate experiments, which is essential for scientific transparency. Figures and diagrams complement the text and are helpful for understanding complex ideas like the training strategies and results.\n\n**Weaknesses:**\n\n1. **Limited Applicability of Synthetic Tasks**:\n   - While the synthetic tasks used in the experiments are designed to probe specific aspects of VLM reasoning, they do not fully capture the complexity of real-world multimodal tasks. This limits the external validity of the results. Real-world applications often involve much more varied, noisy, and dynamic data, which may not be sufficiently represented by controlled tasks like Table Readout or Grid Navigation. As a result, the paper could have explored how these methods perform on real-world benchmarks such as VQA.\n\n2. **Inference Time Cost**:\n   - The increased inference time due to image-to-text conversion remains a significant concern. While the paper addresses this cost, it doesn't fully explore how this trade-off might impact practical deployments, especially in environments where latency or real-time responses are crucial. A more detailed analysis of how to mitigate this overhead would make the paper more practical.\n\n3. **Lack of Theoretical Depth**:\n   - The paper focuses heavily on empirical results, which are certainly valuable. However, there is a lack of deep theoretical exploration behind the proposed methods. For example, while the paper discusses gradient alignment, it doesn’t offer a formal mathematical formulation or sufficient explanation of why this approach works. The relationship between gradient alignment and generalization across modalities could have been made more rigorous.\n\n4. **Modality Imbalance Still Present**:\n   - Despite the promising results, the paper acknowledges that a gap remains in generalization performance between text and image inputs, even with the proposed strategies. This indicates that the solution is not fully comprehensive, and the challenge of modality imbalance is far from being solved. A deeper exploration of why some gaps remain and further refinement of the approach would strengthen the paper's claim.\n\n5. **Experimental Results on Large-Scale Tasks**:\n   - The experiments rely on relatively smaller synthetic datasets and do not provide results on large-scale, real-world benchmarks. While this helps control for specific variables, the results may not generalize well to larger, more complex datasets. It would be beneficial to see how the proposed strategies scale when applied to more realistic data, where other complexities such as noise, data sparsity, and multimodal interactions come into play.\n\n6. **Insufficient Discussion on Limitations**:\n   - While the paper briefly mentions limitations such as inference cost and the restricted scope of the tasks, it doesn't delve deeply into potential shortcomings of the proposed approach. A more balanced discussion of when and where these methods might fail would provide readers with a clearer understanding of the boundaries of the work.", "other_comments_or_suggestions": "no", "questions_for_authors": "1. The paper relies on synthetic tasks like Table Readout and Grid Navigation to study modality imbalance. How do you justify using these tasks, and how well do you think the findings apply to real-world VLM applications? Can we generalize the results beyond these controlled tasks?\n\n2. Even with the strategies you’ve proposed, there’s still a noticeable S2H generalization gap between text and image inputs. What do you see as the remaining challenges in addressing this gap, and are there any components or techniques missing that could help achieve full generalization?\n\n3. You mention that the image-to-text conversion method helps but comes with increased inference time. What are the real-world implications of this, and do you have any suggestions or strategies for minimizing the extra cost during inference, especially in production environments?\n\n4. The gradient alignment study is an interesting part of the paper, but I’m curious about how reliable those alignment scores are as a measure of S2H generalization. Have you tested their stability across different datasets or training conditions, and do you think they are robust enough to be a key indicator of generalization?\n\n5. Lastly, how do you generate the Chain-of-Thought (CoT) data for the tasks you’ve presented? Could you walk us through the process a bit more?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission introduces a synthetic framework for assessing the ability of VLMs to perform algorithmic visual reasoning.  Reviewers in general like the setup and believe the submission is a meaningful contribution to the community.  Some concerns, in particular those about the experimental validation, are mostly addressed in the rebuttal.  One reviewer remained negative, though they did not provide any justifications beyond minor language problems; their recommendation is therefore ignored.  Given all these considerations, the AC would like to recommend acceptance.  The authors should revise the submission based on the reviews and rebuttal in the camera ready.
Decision: Accept (poster)