Beyond the Linear Separability Ceiling: Aligning Representations in VLMs

## Abstract

Abstract A challenge in advancing Visual-Language Models (VLMs) is determining whether their failures on abstract reasoning tasks, such as Bongard problems, stem from flawed perception or faulty top-down reasoning.
To disentangle these factors, we introduce a diagnostic framework centered on the Linear Separability Ceiling (LSC), the performance achievable by a linear classifier on a VLM’s raw visual embeddings.
Applying this framework to state-of-the-art VLMs, we uncover a pervasive “alignment gap”, where most models fail to generatively outperform the linear separability of their own representations.
We find that the few models surpassing this ceiling do so via two mechanisms: by further refining visual representations into a more linearly separable format or by executing non-linear decision logic.
We demonstrate that this bottleneck is not a fundamental limitation but a solvable alignment issue.
By augmenting standard next-token prediction with a contrastive objective, our fine-tuning method activates dormant reasoning pathways,
systematically improving the linear structure of representations to significantly surpass the LSC.

## 1 Introduction

A challenge for state-of-the-art Visual-Language Models (VLMs) (Radford et al., 2021) is understanding the root of their frequent failures on abstract tasks.
Is the bottleneck flawed bottom-up visual perception, which builds representations from sensory input (Marr, 2010),
or a flaw in the top-down reasoning that interprets that input using prior knowledge (Gregory, 1970)?
This question highlights a persistent gap between machine and human cognition,
particularly on visual puzzles (Wüst et al., 2024) and tasks VLMs solve in text but not in visual formats (Park et al., 2025).
While previous studies have explored this perception-reasoning interface within modern VLMs for abstract tasks like Bongard problems,
the primary cause of failure has been difficult to isolate, lacking an empirical method to quantify this gap (Vaishnav & Tammet, 2025; Małkiński et al., 2025).

To address this ambiguity, this paper introduces a diagnostic framework centered on the Linear Separability Ceiling (LSC),
a measure of the performance a linear classifier can achieve on a VLM’s raw visual embeddings.
The LSC provides a baseline for the quality of the initial visual representations,
establishing a benchmark that the model’s end-to-end (non-linear) reasoning must surpass to demonstrate added value.

Our focus on computational non-linearity is complemented by recent work on representational geometry by Engels et al. (2025) that challenges the Linear Representation Hypothesis (LRH).
They find that concepts like days of the week and months of the year form irreducible, circular features in activation space.
This *representational* non-linearity implies a need for the *computational* non-linearity we investigate;
as they show, models use these exact circular features to perform tasks involving modular arithmetic.
Their work thus offers a concrete, mechanistic example of the structured features our hypothesized (non-linear) reasoning pathways may operate on.

Applying this framework reveals our central finding: an “alignment gap,”
a concept that extends known representational issues like the “modality gap” (Yaras et al., 2022) to the interface between perception and reasoning.
We find that a model’s reasoning pathways are often not aligned with its own high-quality visual representations.
This causes the generative performance of most leading VLMs to be statistically no better than the LSC on their own visual embeddings.

The models that surpass this ceiling do so via two strategies: by further refining representations into a more linearly separable format,
effectively extending the bottom-up perception process, or by executing a non-linear decision logic that functions as a form of top-down reasoning.
Strong evidence for this second pathway comes from the success of postfix tuning (Li & Liang, 2021; Lester et al., 2021),
a method which by design cannot alter the initial representations.

## 2 Related work

Abstract visual reasoning in VLMs.
The evaluation of VLMs has evolved from foundational benchmarks for single-image understanding,
such as captioning (Lin et al., 2014) and VQA (Goyal et al., 2019), to more rigorous tests of reasoning.
The complexity of these evaluations has increased along two axes.
First, at the vision-language interface, benchmarks now demand deeper reasoning.
This includes VQA variants that test for compositional structure and spatial skills like GQA (Hudson & Manning, 2019),
or probe commonsense understanding as in VCR (Zellers et al., 2019).
Complementing these, datasets like Winoground (Thrush et al., 2022) specifically isolate visio-linguistic compositional abilities.
Second, in the purely visual domain, a distinct set of benchmarks evaluates abstract reasoning by removing the linguistic component entirely.
This category includes Raven’s Progressive Matrices (RPMs) (Zhang et al., 2019),
the Abstract Reasoning Corpus (ARC) (Chollet, 2019), and Bongard tasks (Bongard, 1970), which are the focus of this work.

VLM architectures and modality fusion.
A key architectural differentiator in VLMs is the strategy for fusing modalities (Gadzicki et al., 2020; Shukor et al., 2025).
While early-fusion models like Chameleon (Team, 2024) create a joint representation from raw inputs,
the now-dominant late-fusion approach first processes images with a dedicated vision encoder.
In its most common form, exemplified by LLaVA (Liu et al., 2023), the resulting visual embeddings are mapped into the language model’s space via a simple projection layer.
A more deeply integrated variant is cross-attention fusion, pioneered by Flamingo (Alayrac et al., 2022) and BLIP (Li et al., 2022),
which insert cross-attention layers within the LLM (Lin et al., 2021).
Our framework is designed for late-fusion models where initial visual representation can be isolated.

Linear Representation Hypothesis (LRH) posits that concepts within neural networks are represented by linear structures in activation space.
This is commonly observed as compositional linearity, where the embeddings of composite concepts can be decomposed into the sum of their constituent parts (Trager et al., 2023),
and appears fundamental to transformer decoders, whose sequential layer transformations are often nearly linear (Razzhigaev et al., 2024).
Theoretical work supports these empirical findings,
suggesting that the next-token prediction objective inherently biases models towards learning representations that are linear transformations of latent concepts (Liu et al., 2025).
This hypothesis suggests that the learned representations should be linearly separable — that is,
distinguishable by a simple linear classifier.
This property is critical for generalization, enabling linear transferability where a classifier trained on a source domain remains effective on a new,
related target domain (HaoChen et al., 2022).
The emergence of such a well-ordered geometric space is not a coincidence but a predictable consequence of high-dimensional geometry.
Stochastic separation theorems establish that in high dimensions, any given point in a random set can be separated from the others by a hyperplane with high probability,
even if the number of points in the set grows exponentially with the dimension (Sidorov & Zolotykh, 2020).

Representation alignment and the modality gap.
Research into representation alignment has identified critical bottlenecks in VLMs, which can occur either in the initial visual encoding or in the subsequent reasoning process (Chia et al., 2024; Zhang et al., 2024).
A key example of such an alignment challenge is the modality gap:
a widely observed phenomenon where the learned representations for different modalities exhibit distinct structures or occupy separate regions within the joint embedding space, even when they are semantically aligned (Yaras et al., 2022).
This systematic separation, particularly noted in contrastively trained models, can hinder cross-modal alignment and the seamless integration of information (Qiu et al., 2024), potentially impacting the discriminative power of these representations.
Our framework allows us to quantitatively assess the impact of such issues by establishing the linear separability ceiling as baseline.
This concept of a representational performance ceiling is not new; for instance, prior work has identified a “contrastive learning ceiling” in semantic textual similarity (Zhang & Li, 2024).

Training and adapting.
To bridge the modality gap and enhance reasoning, various adaptation strategies have been developed.
While some work proposes novel architectural modifications (Bigverdi et al., 2025; Kolner et al., 2025)
or reinforcement learning techniques (Li et al., 2025; Huang et al., 2025), our work focuses on training objectives.
Vision encoders are typically pre-trained with objectives like contrastive learning, as seen in CLIP (Radford et al., 2021),
or combined contrastive-generative objectives, as in CoCa (Yu et al., 2022).
Building on this, some methods augment the standard next-token prediction objective with an auxiliary contrastive loss
to explicitly align representations (Ouali et al., 2025; Wu et al., 2025; Ak et al., 2024).
While these approaches typically tackle the modality gap by aligning image-to-text pairs, we align image-to-image pairs.
To apply such strategies efficiently, parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) (Hu et al., 2022)
and prompt tuning (Li & Liang, 2021; Zhou et al., 2022) are widely adopted (Lester et al., 2021; Jahan et al., 2025).
We investigate whether strategies that co-optimize for enhanced representational discriminability
alongside generative accuracy enable VLMs to better leverage their internal visual representations.

## 3 LLM usage

The authors utilized a LLM to refine grammar and sentence structure, and to aid in discovering related work.
The authors reviewed all generated content and assume full responsibility for this publication.

## 4 Experimental setup

Our analysis centers on abstract visual reasoning using bongard-style tasks,
which require a model to infer a rule from positive and negative examples to classify a query image.
A representative example of this task structure is shown in Figure [1](https://arxiv.org/html/2507.07574v2#S4.F1).

Figure: Figure 1: An example of the bongard HOI task. Model must infer a common rule, in this case, "a person is performing a jump on a motorcycle," from the positive examples and determine if the query image follows this rule, which the negative examples do not.
Refer to caption: bongard_images/positive_examples.png

Datasets.
We use two Bongard-style datasets: Bongard OpenWorld (Wu et al., 2024), split into train (500), validation (100),
and test (500) samples with distinct semantic components across splits; and Bongard HOI (Jiang et al., 2022).
For HOI, we use its original splits, with a 4000-sample balanced training set.
Its validation ($4\cdot 100$) and test ($4\cdot 200$) sets are categorized by concept novelty (seen/unseen object/action).
Each sample contains 6 positive, 6 negative, and 1 query image.
For cross-domain generalization, we also evaluate on the Winoground benchmark, a text-image retrieval task designed for testing compositional reasoning.

Models.
Baseline performance was established on multiple SoTA VLMs: Phi 3.5 vision 4.2B (Microsoft, 2024), Pixtral 12B (MistralAI, 2024),
Gemma3 4B and 27B (Google, 2025), InternVL3 14B (Chen et al., 2024), and Qwen 2.5 VL 7B and 72B (Alibaba, 2025).
Subsequently, we applied PEFT to Gemma3 4B, Phi and Pixtral.

Evaluation.
For all fine-tuning experiments, we saved model checkpoints every epoch and selected the one with the best performance on the validation set.
Baselines and best-performing checkpoints were evaluated on their respective test splits.
Models demonstrate consistent performance across all conceptual novelty splits (see Appendix [I](https://arxiv.org/html/2507.07574v2#A9)), so we report the average for brevity.
Furthermore, our evaluations use balanced test sets, as models can exhibit class-specific variability (Appendix [K](https://arxiv.org/html/2507.07574v2#A11)).

## 5 A framework for decomposing VLM reasoning

To diagnose whether VLM failures on abstract reasoning tasks stem from flawed perception or reasoning, we introduce a diagnostic framework centered on a non-parametric linear probe.
This framework assesses the discriminability of a model’s visual embeddings, independent of its generative process, allowing us to disentangle the contributions of initial representation quality from subsequent LLM processing effectiveness.
First, we benchmark performance across various prompt structures, including interleaved vs. labeled image presentation and direct vs. Chain-of-Thought (CoT) prompting (see Appendix [A](https://arxiv.org/html/2507.07574v2#A1)).

### 5.1 Methodology

We then extract multi-token image embeddings at two key stages: the vision stage, which captures initial representations from the vision encoder, and the final stage, which captures contextualized representations from the LLM’s final hidden state.
We aggregate these sequences into single vectors, $\vec{v}_{i}$, using mean pooling and L2 normalization.
This is a standard technique for producing a single embedding from a sequence of tokens (Reimers & Gurevych, 2019), that preserves angular properties relevant to cosine similarity.
To quantify the discriminative quality of these embeddings, we probe them using a nearest-centroid classifier.
We selected this nearest-centroid strategy as it proved more performant than a nearest-neighbor approach as it mitigates the influence of outlier examples and captures a more robust prototype of the abstract concept (Hastie et al., 2009),
as shown in Appendix [B](https://arxiv.org/html/2507.07574v2#A2).
We then compute prototype vectors for the positive ($P$) and negative ($N$) example sets by averaging their respective image vectors (e.g., $\vec{c}_{P}=\text{mean}(\vec{v}_{p_{1}},\dots,\vec{v}_{p_{k}})$).
A query image ($Q$) is then classified based on its cosine similarity to these centroids (i.e., comparing $\cos(\vec{v}_{Q},\vec{c}_{P})$ to $\cos(\vec{v}_{Q},\vec{c}_{N})$).

### 5.2 Decomposition of performance

Our framework dissects VLM performance by statistically comparing three key accuracies: the end-to-end generative accuracy ($acc_{gen}$), the linear probe accuracy on initial *vision* embeddings ($acc_{vision}$), and on *final* contextualized embeddings ($acc_{final}$).
Grounded in the linear representation hypothesis, which posits that concepts are encoded as linear structures within a model’s activation space, we define $acc_{vision}$ as the linear separability ceiling (LSC).
The LSC quantifies the baseline discriminative power of the visual representations, a primary goal of contrastive pre-training.

To classify a model’s performance, we perform a statistical significance test on the difference between its generative accuracy and its LSC.
This approach accounts for the uncertainty in both measurements by analyzing the standard error of their difference (Burns & Dobson, 1981).
Performance is classified as surpassed if the generative accuracy is found to be statistically superior to the LSC (p < 0.05).
Conversely, performance is classified as exhibiting an alignment gap if the accuracy is not statistically superior to the LSC.
This is because a non-linear model is expected to outperform a linear probe on its own representations;
a failure to do so suggests its generative capabilities are not fully aligned with its encoded knowledge.

For models that surpass the LSC, we diagnose the underlying computational strategy by analyzing the geometric evolution of the visual representations.
The comparison between the LSC and the final representation accuracy allows us to distinguish between two mechanisms:

- -
Enhanced linear separability, identified when generative accuracy and linear separability of the final embeddings both significantly surpasses the LSC.
- -
Non-linear decision logic, inferred when generative accuracy significantly surpasses the LSC even though the linear separability of the final embeddings does not.

A Chi-squared ($\chi^{2}$) test is then used to assess the statistical dependence between the generative model’s predictions and the outcomes from our linear probes (p < 0.05).

## 6 A pervasive alignment gap

Our framework, applied to 8 SoTA VLMs across two datasets and tested with 8 prompt formats each, reveals a pervasive alignment gap.
As shown in Figure [2](https://arxiv.org/html/2507.07574v2#S6.F2), a model’s generative performance ($acc_{gen}$ is highly variable and frequently fails to surpass its own LSC.
We identify this widespread issue as an alignment gap, where a model’s reasoning pathways are not effectively aligned with its own visual representations.
Full results are in Appendix [C](https://arxiv.org/html/2507.07574v2#A3).

Figure: Figure 2: Generative accuracy vs. linear separability for all baseline models. The shaded areas show the outcome of a statistical comparison that accounts for the 95% confidence intervals of both metrics.
Refer to caption: bongard_images/LSC_benchmark.png

Strategies for surpassing the LSC.
While most configurations are constrained by this bottleneck, a few successful models offer insight into how it can be overcome.
Given that the iterative refinement of representations is a core mechanic of all transformer-based models,
our analysis reveals that successful models channel this process into two distinct computational strategies.
These strategies are distinguished by the geometric transformations they apply to the visual embeddings,
with Table [1](https://arxiv.org/html/2507.07574v2#S6.T1) highlighting the divergent outcomes.
The first enhances linear separability, effectively continuing the bottom-up perceptual work of the vision encoder.
In contrast, the second implements a non-linear decision logic,
a form of top-down reasoning that computes a solution not linearly readable from the visual embeddings themselves.

**Table 1: Performance metrics for selected SoTA models on the OpenWorld dataset, comparing direct and CoT generative accuracy with initial (LSC) and final representation accuracies.**
| Model | Direct acc (%) | CoT acc (%) | LSC ($acc_{vision}$, %) | Final rep. ($acc_{final}$, %) |
| --- | --- | --- | --- | --- |
| Pixtral 12B | $79.4\pm 3.5$ | $84.2\pm 3.2$ | $76.0\pm 3.7$ | $\mathbf{88.0\pm 2.8}$ |
| Gemma3 27B | $93.2\pm 2.2$ | $87.2\pm 2.9$ | $\mathbf{88.6\pm 2.8}$ | collapsed (50.0) |
| Qwen2.5-VL 72B | $93.6\pm 2.1$ | $93.2\pm 2.2$ | $\mathbf{86.8\pm 3.0}$ | $74.8\pm 3.8$ |
| Human | $91.0$ | - | - | - |

Enhancing linear separability.
The first and rarer of the two strategies is to enhance the linear separability of the visual representations,
a strategy successfully employed only by Pixtral.
Its reasoning pathway further processes the initial visual embeddings, transforming them into a more linearly separable format ($acc_{final}>LSC$).
This suggests that the language model’s layers are functioning as an extension of the perception system,
refining features in a manner that aligns with the principles of contrastive learning (Alshammari et al., 2025).
Consequently, its generative success is characterized by a strong statistical alignment between its final output and the classifications from a simple linear probe on these improved final representations.

**Table 2: Breakdown of strategies which surpassed the LSC across 128 configurations.**
| Successful strategy | OpenWorld | HOI |
| --- | --- | --- |
| Enhanced linear separability | 2 | 0 |
| Non-linear decision logic | 14 | 8 |

Non-linear decision logic.
The vast majority of successes, however, were achieved by implementing a non-linear decision logic,
which leverages the full computational depth of the transformer to form a non-linear decision boundary, as shown in Table [2](https://arxiv.org/html/2507.07574v2#S6.T2).
The initial visual features serve as inputs to this computation, but the intermediate and final visual embeddings are not necessarily constrained to be linearly separable.
Gemma3 represents an extreme and brittle case, where its final embeddings undergo a representational collapse with respect to linear separability ($acc_{final}\approx 50\%$),
making performance highly sensitive to the prompt format.
Qwen provides a more robust implementation; although its final representation accuracy also drops below its LSC,
its subsequent non-linear computations are stable enough to achieve consistent, beyond-LSC performance.
This pathway highlights that a high LSC is a valuable starting point, but ultimate success can also depend on a powerful computational process that operates effectively within a non-linear representational geometry.

**Table 3: Summary of $\chi^{2}$ statistical dependence tests between generative performance and linear probe outcomes across 128 VLM configurations, where (+) and (-) denote significant positive/inverse dependence (both methods tend to agree/disagree), and (Ø) for no significant dependence.**
| Probe location | + | - | Ø |
| --- | --- | --- | --- |
| Vision embeddings | 95 | 2 | 31 |
| Final embeddings | 74 | 29 | 25 |

The disconnect between separability and generation.
As shown in Table [3](https://arxiv.org/html/2507.07574v2#S6.T3), more linearly separable initial visual representations (higher LSC) generally correlate with better generative performance.
However, a deeper look reveals a disconnect between the *final* embedding separability and the generative output.
While a model’s processing attunes its final representations to its generative logic, the relationship is inverse for 29 configurations — meaning the model is more likely to generate an opposing answer compared to what the linear probe on its final embeddings suggests.
The model’s reasoning can therefore be disconnected from, or even opposed to, the logic of a linear probe on the same embeddings.
While high final separability is not a prerequisite for success, this suggests a hypothesis for creating more robust and interpretable models; train them to improve the linear discriminability of their final embeddings while ensuring that structure is positively aligned with the generative process.

## 7 Surpassing the LSC

Our baseline analysis in Section [6](https://arxiv.org/html/2507.07574v2#S6) revealed a widespread alignment gap where VLMs struggle to surpass their LSC.
Through targeted fine-tuning, we systematically probe intervention points within the VLM, from the vision-language interface to the LLM’s reasoning pathways, to pinpoint and resolve the bottleneck.

### 7.1 Experimental setup

To isolate the most effective intervention points for resolving the alignment gap, we compare several PEFT methods across three key areas of the model.
First, to test the vision-language interface, we fine-tune only the projection layer.
Second, to provide input-level guidance without altering core weights, we use prompt tuning, which prepends and appends trainable soft prompts.
As a key methodological control, we also test postfix tuning, using only the appended prompts to ensure the intervention cannot alter the initial processing of visual representations.
Finally, for full model adaptation, we use LoRA to inject trainable matrices into the attention and MLP layers.
Full hyperparameters for all methods are detailed in Appendix [D](https://arxiv.org/html/2507.07574v2#A4).

Evaluation. To evaluate structural robustness, we test performance on unseen prompt formats.
The in-distribution (ID) evaluation uses the standard interleaved prompt structure seen during training.
For the out-of-distribution (OOD) test, we use a labeled prompt structure that groups all images by category at the end of the prompt.
This OOD setup challenges how well the optimization targets generalize beyond the syntactic structure on which they were trained.
Furthermore, OpenWorld test set consists entirely of new concepts unseen in train set, verifying conceptual generation.

Objectives. We train the PEFT parameters using two objectives.
The first is Next-token prediction ($\mathcal{L}_{\text{NT}}$),
the standard language modeling objective of maximizing the likelihood of the ground-truth output tokens.
The second is an InfoNCE + NTP ($\mathcal{L}_{\text{combined}}$) objective,
which augments $\mathcal{L}_{\text{NT}}$ with an explicit contrastive loss ($\mathcal{L}_{\text{sim}}$) designed to identify a positive sample from a set of negative samples (van den Oord et al., 2019) by means of a weighted sum.

$$ $\mathcal{L}_{\text{combined}}=w_{n}\mathcal{L}_{\text{NT}}+w_{c}\mathcal{L}_{\text{sim}}$ $$

This contrastive term explicitly encourages the final embeddings of a query image to be closer to its true category centroid than the alternative.
The motivation is to directly improve the linear separability of the final representations, thereby raising the separability ceiling for the task while co-optimizing for generative correctness,
which is further detailed in Appendix [E](https://arxiv.org/html/2507.07574v2#A5).

### 7.2 Pinpointing the location for intervention

To pinpoint the source of performance gains, we conducted a targeted ablation study on the Phi model with the OpenWorld dataset.
The results, summarized in Table [4](https://arxiv.org/html/2507.07574v2#S7.T4), reveal that the *location* of the intervention is the critical factor in surpassing the alignment gap.

**Table 4: Reported accuracies (%) for the Phi model on the OpenWorld dataset. The ID evaluation uses the interleaved prompt structure seen during training. For the OOD test, we use a labeled prompt structure that groups all images by category at the end of the prompt. In the generative columns, bold indicates the LSC is surpassed.**
| Tuning method | ID | OOD | LSC |
| --- | --- | --- | --- |
| Direct baseline | $59.0$ | $79.4$ | $84.0$ |
| Projector ($\mathcal{L}_{\text{NT}}$) | $\mathbf{90.2}$ | $73.8$ | $84.0$ |
| Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{94.2}$ | $\mathbf{90.2}$ | $84.2$ |
| Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{90.4}$ | $83.4$ | $84.2$ |
| LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{92.2}$ | $\mathbf{89.8}$ | $84.4$ |
| LoRA ($\mathcal{L}_{\text{NT}}$ llm-only) | $\mathbf{92.4}$ | $\mathbf{89.6}$ | $84.2$ |

Isolating the alignment failure in the VLM.
Minimalist interventions at the vision-language interface lead to structural overfitting rather than robust reasoning, as seen in Table [4](https://arxiv.org/html/2507.07574v2#S7.T4).
Since interface-level adjustment encourages superficial learning, we froze the projector for all subsequent experiments.
Our results pinpoint the bottleneck to the LLM’s reasoning pathways, not the vision encoder’s perception.
An ablation study confirmed this (Appendix [G](https://arxiv.org/html/2507.07574v2#A7)), as applying LoRA to the vision encoder resulted in nearly identical predictions and no additional performance gain compared to adapting the LLM alone.
This suggests any adaptations within the vision encoder are likely a form of naïve loss minimization (Prieto et al., 2025),
merely scaling existing features rather than learning more discriminative ones.
The core issue is therefore not a perceptual deficit but a deeper failure in how the LLM processes its visual inputs.

With the bottleneck identified in the LLM’s reasoning module, we find that targeted interventions can robustly unlock latent abilities.
Our findings reveal a critical distinction between two approaches: activating latent skills versus adapting core weights, with the required method depending on the reasoning task.
The success of postfix tuning, a methodological control that cannot alter visual representations, provides evidence that VLMs possess powerful, dormant reasoning pathways.

**Table 5: Reported accuracies (%) for Phi model on both datasets, where bold indicates the the LSC is surpassed.**
| Method | OpenWorld | HOI |
| --- | --- | --- |
| Direct baseline | $59.0$ | $52.1$ |
| LSC | $84.0$ | $71.9$ |
| Postfix tuning | $\mathbf{94.2}$ | $63.2$ |
| LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{92.2}$ | $\mathbf{78.6}$ |

Activation vs. adaptation.
As shown in Table [5](https://arxiv.org/html/2507.07574v2#S7.T5), performance on OpenWorld is an activation issue.
Prompt-based methods are highly effective, achieving performance comparable or even superior to LoRA.
This suggests the model’s inherent skills only need to be steered by an optimal input,
a process similar to prompt engineering (Burns et al., 2023; Brown et al., 2020).
On the relational HOI task, activation methods prove insufficient, requiring deeper adaptation via LoRA to substantially boost performance.
This reveals a core principle, holding true for all tested models (Appendix [H](https://arxiv.org/html/2507.07574v2#A8)),
that activation suffices for atomic semantic concept comparison while adaptation is required for reasoning over relational semantic concepts.

### 7.3 Comparing objectives

The $\mathcal{L}_{\text{NT}}$ objective enables a generative model to achieve high accuracy by implementing non-linear decision boundaries.
However, this objective provides no explicit incentive to improve the linear structure of the model’s representations.
The contrastive loss term, $\mathcal{L}_{\text{sim}}$, is introduced to explicitly optimize for more linearly separable final representations.
The combined objective, $\mathcal{L}_{\text{combined}}$, therefore aligns the two goals of generative reasoning and representational clarity.
Evidence for this alignment is the consistent, significant dependence between generative outcomes and final-layer similarity predictions,
which is absent in models trained only with $\mathcal{L}_{\text{NT}}$.
Table [6](https://arxiv.org/html/2507.07574v2#S7.T6) consolidates the performance of different PEFT strategies,
comparing their generative and embedding similarity-based classification accuracy against relevant baselines.

**Table 6: Summary of PEFT performance on Bongard tasks. Reported accuracies presented as $accuracy\pm ME$. In the generative columns, bold indicates the LSC is surpassed. Superscripts on separability scores denote a significant dependence (p < 0.05) with the predictions of the corresponding generative method (G).**
| Model | Dataset | Method | Generative (%) | LSC (%) | Repr. acc. final (%) |
| --- | --- | --- | --- | --- | --- |
| Phi | OpenWorld | Direct baseline | $59.0\pm 4.3$ | $84.0\pm 3.2^{G}$ | $76.4\pm 3.7^{-G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{92.2\pm 2.4}$ | $84.4\pm 3.2^{G}$ | $74.8\pm 3.8$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.6\pm 1.8}$ | $84.4\pm 3.2^{G}$ | $93.8\pm 2.1^{G}$ |
|  | HOI | Direct baseline | $52.1\pm 3.5$ | $71.9\pm 3.1$ | $60.5\pm 3.4^{-G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{78.6\pm 2.8}$ | $71.9\pm 3.1^{G}$ | $63.6\pm 3.3^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{79.2\pm 2.8}$ | $71.9\pm 3.1^{G}$ | $82.0\pm 2.7^{G}$ |
| Pixtral | OpenWorld | Direct baseline | $72.4\pm 3.9$ | $76.0\pm 3.7^{G}$ | $87.2\pm 2.9^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{93.4\pm 2.2}$ | $76.6\pm 3.7$ | $87.2\pm 2.9^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.0\pm 1.9}$ | $74.4\pm 3.8$ | $96.2\pm 1.7^{G}$ |
|  | HOI | Direct baseline | $57.8\pm 3.4$ | $62.7\pm 3.4^{G}$ | $70.2\pm 3.2^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{78.0\pm 2.9}$ | $61.6\pm 3.4^{G}$ | $74.9\pm 3.0^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{79.6\pm 2.8}$ | $63.1\pm 3.3^{G}$ | $77.8\pm 2.9^{G}$ |
| Gemma3 4B | OpenWorld | Direct baseline | $76.0\pm 3.7$ | $89.8\pm 2.7$ | $\text{collapsed (50.0)}^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.4\pm 2.3$ | $89.8\pm 2.7^{G}$ | collapsed (50.0) |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.6\pm 1.8}$ | $89.8\pm 2.7$ | $96.6\pm 1.6^{G}$ |
|  | HOI | Direct baseline | $56.5\pm 3.4$ | $74.1\pm 3.0$ | $\text{collapsed (50.0)}^{G}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{84.2\pm 2.5}$ | $74.1\pm 3.0^{G}$ | collapsed (50.0) |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{84.2\pm 2.5}$ | $74.1\pm 3.0^{G}$ | $83.2\pm 2.6^{G}$ |
| Human | OpenWorld |  | 91.0 | - | - |
|  | HOI |  | 91.4 | - | - |

Contrastive learning ceiling.
For models trained with the $\mathcal{L}_{\text{combined}}$ objective,
the convergence of generative and linear probe accuracy indicates that performance is limited by representational quality,
as the model’s reasoning pathways have reached a ceiling.
We observe evidence for this in the model’s internal attention mechanisms,
where the most significant differences are concentrated in the later layers.
This finding is consistent with research suggesting that reasoning mainly happens in middle-to-late layers,
whereas perception is encoded in early layers (Chen et al., 2025).
A visual analysis of these attention changes is available in Appendix [M](https://arxiv.org/html/2507.07574v2#A13).

Conceptual Generalization.
The performance gains from fine-tuning stem from genuine conceptual understanding rather than memorization.
This is demonstrated on the Bongard-HOI dataset,
where our models show robust generalization by maintaining high accuracy on test splits with entirely unseen objects and actions (Appendix [I](https://arxiv.org/html/2507.07574v2#A9)).
This finding is reinforced by strong results on the OpenWorld test set, which consists entirely of new concepts by design.

**Table 7: Text retrieval on Winoground, comparing baselines with HOI-trained LoRAs. The CLIP model is included as an LSC baseline for the Phi model.**
| Model | Method | Acc. (%) |
| --- | --- | --- |
| CLIP (ViT-L/14) | Baseline | 27.75 |
| Phi | Baseline | 16.75 |
|  | $\mathcal{L}_{\text{NT}}$ | 18.25 |
|  | $\mathcal{L}_{\text{combined}}$ | 29.25 |
| Pixtral | Baseline | 28.75 |
|  | $\mathcal{L}_{\text{NT}}$ | 43.50 |
|  | $\mathcal{L}_{\text{combined}}$ | 54.75 |
| Gemma3 4B | Baseline | 5.25 |
|  | $\mathcal{L}_{\text{NT}}$ | 5.75 |
|  | $\mathcal{L}_{\text{combined}}$ | 12.25 |

Generalization under domain shift.
Cross-domain evaluations reveal that the nature of the skill learned during fine-tuning is a key determinant of its transferability.
For instance, the relational reasoning skills acquired from the HOI dataset transferred broadly across both our cross-Bongard task and the Winoground benchmark,
whereas the atomic concept comparison skills from the OpenWorld dataset proved less generalizable (Appendix [J](https://arxiv.org/html/2507.07574v2#A10)).
This distinction is particularly evident on Winoground, where only the HOI-trained models achieved substantial performance gains
(Table [7](https://arxiv.org/html/2507.07574v2#S7.T7)).
Here, our $\mathcal{L}_{\text{combined}}$ objective achieved the most successful transfer
by consistently improving text retrieval accuracy for all models, a success we attribute
to the inherent need for inter-image comparison in Bongard-style problems.
Furthermore, this out-of-domain evaluation validated our LSC framework by demonstrating both its broader applicability and the difficulty of surpassing the LSC,
which a fine-tuned Phi model could not significantly surpass.

Limitations.
A key limitation of the $\mathcal{L}_{\text{combined}}$ objective is its tendency for catastrophic forgetting,
a phenomenon where optimizing for one objective degrades performance on another.
This vulnerability became apparent when testing model robustness to changes in prompt format (Appendix [H](https://arxiv.org/html/2507.07574v2#A8)) on the HOI dataset.
On this task, all three models trained with $\mathcal{L}_{\text{combined}}$ experienced performance degradation when the prompt format was altered,
whereas only one of the three models trained with the standard $\mathcal{L}_{\text{NT}}$ objective performed worse.
This suggests that the strong contrastive pressure,
while successfully restructuring the embeddings, causes the generative pathway to overfit to the training syntax.
As a result, the model stops using more general reasoning heuristics,
preventing it from interpreting its own well-structured representations in a new format.

## 8 Conclusion

This work demonstrates that the reasoning failures of VLMs in abstract visual tasks are often not an issue of poor visual perception, but a solvable misalignment in the subsequent reasoning process, revealing that these models possess far greater latent capabilities than their out-of-the-box performance suggests.
We introduce the Linear Separability Ceiling (LSC), a measure of raw embedding quality, not as an absolute performance limit, but as an observational benchmark that baseline models struggle to surpass due to this alignment gap.
We demonstrate that this alignment gap is solvable, by applying parameter-efficient fine-tuning with a combined objective, which augments standard next-token prediction with a contrastive loss to explicitly improve the linear separability of final representations.
As a result, every model trained with this method successfully surpassed its LSC on both the OpenWorld and HOI datasets.
This unlocked their dormant reasoning abilities, enabling them to surpass human-level accuracy on the OpenWorld task and substantially close the performance gap on the more complex relational reasoning of the HOI task.

Our key contributions include:

- •
showing that the reasoning failure stems not from a fundamental capability deficit but from an alignment gap between vision and/or text;
- •
a novel diagnostic framework centered on the linear separability ceiling, which formalizes and quantifies the alignment gap in visual tasks;
- •
the identification of two distinct computational strategies that successful models use to surpass the LSC: enhancing the linear separability of representations or executing non-linear decision logic;
- •
and a validated fine-tuning methodology using a combined objective that resolves the alignment gap by structurally improving vision representations,
leading to performance gains that are more robust and transferable across domains.

This framework opens several avenues for future work,
from developing training methods that avoid the trade-off between performance and structural robustness,
to using the LSC as a live diagnostic during model development.
Ultimately, our work provides both validated methods for enhancing VLM reasoning and novel insights into the representational properties of multimodal models,
paving the way for more capable and interpretable AI.

## References

- Ak et al. (2024)
Kenan Emir Ak, Jay Mohta, Dimitris Dimitriadis, Saurav Manchanda, Yan Xu, and
Mingwei Shen.
Aligning vision language models with contrastive learning.
*ECCV Workshop on Unlearning and Model Editing*, 2024.
- Alayrac et al. (2022)
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds,
Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina
Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy
Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikoł aj Bińkowski, Ricardo
Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.
Flamingo: a visual language model for few-shot learning.
*NeurIPS*, 2022.
- Alibaba (2025)
Alibaba.
Qwen2.5-vl technical report.
*arXiv*, 02 2025.
- Alshammari et al. (2025)
Shaden Naif Alshammari, John R Hershey, Axel Feldmann, William T Freeman, and
Mark Hamilton.
I-con: A unifying framework for representation learning.
*ICLR*, 2025.
- Bigverdi et al. (2025)
Mahtab Bigverdi, Zelun Luo, Cheng-Yu Hsieh, Ethan Shen, Dongping Chen, Linda G
Shapiro, and Ranjay Krishna.
Perception tokens enhance visual reasoning in multimodal language
models.
*CVRP*, 2025.
- Bongard (1970)
M Bongard.
Pattern recognition(book on automatic recognition of visual patterns
and mechanization of creativity, presenting computer program termed
arithmetic).
1970.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris
Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever,
and Dario Amodei.
Language models are few-shot learners.
*NeurIPS*, 2020.
- Burns et al. (2023)
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt.
Discovering latent knowledge in language models without supervision.
*ICLR*, 2023.
- Burns & Dobson (1981)
R. B. Burns and C. B. Dobson.
*Standard error of the difference between means*.
Springer Netherlands, Dordrecht, 1981.
- Chen et al. (2025)
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong,
Manling Li, and Junxian He.
Bring reason to vision: Understanding perception and reasoning
through model merging.
*ICML*, 2025.
- Chen et al. (2024)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong,
Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and
Jifeng Dai.
Internvl: Scaling up vision foundation models and aligning for
generic visual-linguistic tasks.
*CVPR*, 2024.
- Chia et al. (2024)
Yew Ken Chia, Vernon Toh, Deepanway Ghosal, Lidong Bing, and Soujanya Poria.
Puzzlevqa: Diagnosing multimodal reasoning challenges of language
models with abstract visual patterns.
*ACL*, 2024.
- Chollet (2019)
François Chollet.
On the measure of intelligence.
*arXiv*, 11 2019.
- Engels et al. (2025)
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark.
Not all language model features are one-dimensionally linear.
*ICLR*, 2025.
- Gadzicki et al. (2020)
Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche.
Early vs late fusion in multimodal convolutional neural networks.
*IEEE*, 2020.
- Google (2025)
Google.
Gemma 3 technical report.
*arXiv*, 03 2025.
- Goyal et al. (2019)
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra,
and Devi Parikh.
Making the v in vqa matter: Elevating the role of image understanding
in visual question answering.
*CVPR*, 2019.
- Gregory (1970)
R. L. Gregory.
*The intelligent eye*.
Weidenfeld & Nicolson, London, 1970.
- HaoChen et al. (2022)
Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma.
Beyond separability: Analyzing the linear transferability of
contrastive representations to related subpopulations, 2022.
- Hastie et al. (2009)
Trevor Hastie, Robert Tibshirani, and Jerome Friedman.
*The Elements of Statistical learning, Second Edition : Data
mining, inference, and Prediction*.
Springer, 2 edition, 2009.
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*ICLR*, 2022.
- Huang et al. (2025)
Ting Huang, Zeyu Zhang, and Hao Tang.
3d-r1: Enhancing reasoning in 3d vlms for unified scene
understanding.
*arXiv*, 2025.
- Hudson & Manning (2019)
Drew A. Hudson and Christopher D. Manning.
Gqa: A new dataset for real-world visual reasoning and compositional
question answering.
*CVPR*, 2019.
- Jahan et al. (2025)
Prottasha Nusrat Jahan, Upama Roy Chowdhury, Shetu Mohanto, Tasfia Nuzhat,
Abdullah As Sami, Md Shamol Ali, Shohanur Islam, Hafijur Raman, Md Kowsher,
and Ozlem Ozmen Garibay.
Peft a2z: Parameter-efficient fine-tuning survey for large language
and vision models.
*arXiv*, 2025.
- Jiang et al. (2022)
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima
Anandkumar.
Bongard-hoi: Benchmarking few-shot visual reasoning for human-object
interactions.
*CVPR*, 2022.
- Kolner et al. (2025)
Oleh Kolner, Thomas Ortner, Stanisław Woźniak, and Angeliki Pantazi.
Mind the GAP: Glimpse-based active perception improves
generalization and sample efficiency of visual reasoning.
*ICLR*, 2025.
- Lester et al. (2021)
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
*EMNLP*, 2021.
- Li et al. (2022)
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation.
*ICML*, 2022.
- Li & Liang (2021)
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
*ACL-IJCNLP*, pp.  4582–4597, 2021.
- Li et al. (2025)
Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu,
Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, and Dong Yu.
Self-rewarding vision-language model via reasoning decomposition.
*arXiv*, 2025.
- Lin et al. (2021)
Hezheng Lin, Xing Cheng, Xiangyu Wu, Fan Yang, Dong Shen, Zhongyuan Wang, Qing
Song, and Wei Yuan.
Cat: Cross attention in vision transformer.
*ICCV*, 2021.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva
Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft coco: Common objects in context.
*ECCV*, 2014.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
*NeurIPS*, 2023.
- Liu et al. (2025)
Yuhang Liu, Dong Gong, Erdun Gao, Zhen Zhang, Biwei Huang, Mingming Gong, Anton
, and Javen Qinfeng Shi.
I predict therefore i am: Is next token prediction enough to learn
human-interpretable concepts from data?
*arXiv*, 03 2025.
- Małkiński et al. (2025)
Mikołaj Małkiński, Szymon Pawlonka, and Jacek Mańdziuk.
Reasoning limitations of multimodal large language models. a case
study of bongard problems.
*ICML*, 2025.
- Marr (2010)
David Marr.
*Vision: A Computational Investigation into the Human
Representation and Processing of Visual Information*.
The MIT Press, 07 2010.
- Microsoft (2024)
Microsoft.
Phi-3 technical report: A highly capable language model locally on
your phone.
*arXiv*, 04 2024.
- Microsoft (2025)
Microsoft.
Phi-4-mini technical report: Compact yet powerful multimodal language
models via mixture-of-loras.
*arXiv*, 03 2025.
- MistralAI (2024)
MistralAI.
Pixtral 12b.
*arXiv*, 10 2024.
- Ouali et al. (2025)
Yassine Ouali, Adrian Bulat, Alexandros Xenos, Anestis Zaganidis,
Ioannis Maniadis Metaxas, Georgios Tzimiropoulos, and Brais Martinez.
Vladva: Discriminative fine-tuning of lvlms.
*CVPR*, 2025.
- Park et al. (2025)
Simon Park, Abhishek Panigrahi, Yun Cheng, Dingli Yu, Anirudh Goyal, and
Sanjeev Arora.
Generalizing from simple to hard visual reasoning: Can we mitigate
modality imbalance in vlms?
*ICML*, 2025.
- Prieto et al. (2025)
Lucas Prieto, Melih Barsbey, Pedro  , and Tolga Birdal.
Grokking at the edge of numerical stability.
*ICLR*, 2025.
- Qiu et al. (2024)
Longtian Qiu, Shan Ning, and Xuming He.
Mining fine-grained image-text alignment for zero-shot captioning via
text-only training.
*AAAI*, 2024.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language
supervision.
*ICML*, 2021.
- Razzhigaev et al. (2024)
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko,
Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov.
Your transformer is secretly linear.
*ACL*, 2024.
- Reimers & Gurevych (2019)
Nils Reimers and Iryna Gurevych.
Sentence-bert: Sentence embeddings using siamese bert-networks.
*EMNLP-IJCNLP*, 2019.
- Shukor et al. (2025)
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu
Cord‡, Joshua Susskind, and Alaaeldin El-Nouby.
Scaling laws for native multimodal models.
*arXiv*, 04 2025.
- Sidorov & Zolotykh (2020)
Sergey Sidorov and Nikolai Zolotykh.
Linear and fisher separability of random points in the d-dimensional
spherical layer and inside the d-dimensional cube.
*IJCNN*, 2020.
- Team (2024)
Chameleon Team.
Chameleon: Mixed-modal early-fusion foundation models.
*arXiv*, 05 2024.
- Thrush et al. (2022)
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe
Kiela, and Candace Ross.
Winoground: Probing vision and language models for visio-linguistic
compositionality.
*CVPR*, 2022.
- Trager et al. (2023)
Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder
Bhatia, and Stefano Soatto.
Linear spaces of meanings: Compositional structures in
vision-language models.
*ICCV*, 2023.
- Vaishnav & Tammet (2025)
Mohit Vaishnav and Tanel Tammet.
A cognitive paradigm approach to probe the perception-reasoning
interface in vlms.
*arXiv*, 05 2025.
- van den Oord et al. (2019)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding.
*arXiv*, 01 2019.
- Wu et al. (2024)
Rujie Wu, Xiaojian Ma, Qing Li, Wei Wang, Zhenliang Zhang, Song-Chun Zhu, and
Yizhou Wang.
Bongard-openworld: Few-shot reasoning for free-form visual concepts
in the real world.
*ICLR*, 2024.
- Wu et al. (2025)
Shengguang Wu, Fan-Yun Sun, Kaiyue Wen, and Nick Haber.
Symmetrical visual contrastive optimization: Aligning vision-language
models with minimal contrastive images.
*ACL*, 2025.
- Wüst et al. (2024)
Antonia Wüst, Tim Tobiasch, Lukas Helff, Devendra Singh Dhami, Constantin A
Rothkopf, and Kristian Kersting.
Bongard in wonderland: Visual puzzles that still make ai go mad?
*Sys2-Reasoning*, 2024.
- Yaras et al. (2022)
Can Yaras, Siyi Chen, Peng Wang, and Qing Qu.
Mind the gap: Understanding the modality gap in multi-modal
contrastive representation learning.
*NeurIPS*, 2022.
- Yu et al. (2022)
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and
Yonghui Wu.
Coca: Contrastive captioners are image-text foundation models.
*TMLR*, 2022.
- Zellers et al. (2019)
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
From recognition to cognition: Visual commonsense reasoning.
2019.
- Zhang & Li (2024)
Bowen Zhang and Chunping Li.
Pcc-tuning: Breaking the contrastive learning ceiling in semantic
textual similarity.
*EMNLP*, 2024.
- Zhang et al. (2019)
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu.
Raven: A dataset for relational and analogical visual reasoning.
*CVPR*, 2019.
- Zhang et al. (2024)
Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp.
Can vision-language models be a good guesser? exploring vlms for
times and location reasoning.
*WACV*, 2024.
- Zhou et al. (2022)
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Learning to prompt for vision-language models.
*IJCV*, 2022.

## Appendix A Baseline prompt definitions

Strategies evaluated in Table [A.2](https://arxiv.org/html/2507.07574v2#A3.T2) varied primarily in how images were presented relative to descriptive text, the position of the query image, and whether CoT reasoning was asked.

Interleaved strategy.

Interleaved query first strategy.

Labeled strategy.

Labeled query first strategy.

One of the following string variables is appended to the output of the core prompt.

Direct conclusion prompt string.

CoT prompt string.

## Appendix B Single vs batched context

We performed a preliminary analysis comparing a nearest-centroid classifier approach against a simpler nearest-neighbor classifier.
This allows us to demonstrate that aggregating visual information into a conceptual prototype is more effective than treating each piece of visual evidence in isolation.

We refer to these two distinct classification strategies as:
Single context, a nearest-neighbor approach where the query image is classified based on its cosine similarity to the single closest example image vector from the combined positive and negative sets.
This method is sensitive to the features of individual, and potentially outlier, examples;
Batched context, a nearest-centroid approach selected for the LSC. The query image is classified based on its cosine similarity to a prototype vector (centroid) representing the entire set of positive or negative examples.
This method is designed to capture the underlying abstract rule of the set.
The results of this comparison are presented in Table [A.1](https://arxiv.org/html/2507.07574v2#A2.T1).

**Table A.1: Comparison of classification accuracy for single vs. batched context.**
| Model | Dataset | Context | Sim. acc (vision, %) | Sim. acc (final, %) |
| --- | --- | --- | --- | --- |
| Phi | OpenWorld | Single | $78.8\pm 3.6$ | $78.8\pm 3.6$ |
|  |  | Batched | $84.0\pm 3.2$ | $87.0\pm 2.9$ |
|  | HOI | Single | $64.9\pm 3.3$ | $66.4\pm 3.3$ |
|  |  | Batched | $71.9\pm 3.1$ | $69.6\pm 3.2$ |
| Pixtral | OpenWorld | Single | $70.4\pm 4.0$ | $78.6\pm 3.6$ |
|  |  | Batched | $76.0\pm 3.7$ | $88.2\pm 2.8$ |
|  | HOI | Single | $62.4\pm 3.4$ | $68.0\pm 3.2$ |
|  |  | Batched | $62.9\pm 3.3$ | $72.4\pm 3.1$ |
| Gemma3 4B | OpenWorld | Single | $77.8\pm 3.6$ | $81.6\pm 3.4$ |
|  |  | Batched | $89.6\pm 2.7$ | $76.6\pm 3.7$ |
|  | HOI | Single | $69.1\pm 3.2$ | $69.8\pm 3.2$ |
|  |  | Batched | $74.1\pm 3.0$ | $70.2\pm 3.2$ |
| Gemma3 27B | OpenWorld | Single | $78.2\pm 3.6$ | $78.4\pm 3.6$ |
|  |  | Batched | $88.6\pm 2.8$ | $78.8\pm 3.6$ |
|  | HOI | Single | $68.4\pm 3.2$ | $69.8\pm 3.2$ |
|  |  | Batched | $73.4\pm 3.1$ | $64.8\pm 3.3$ |
| InternVL 14B | OpenWorld | Single | $69.4\pm 4.0$ | $74.0\pm 3.8$ |
|  |  | Batched | $70.2\pm 4.0$ | $78.4\pm 3.6$ |
|  | HOI | Single | $63.7\pm 3.3$ | $64.4\pm 3.3$ |
|  |  | Batched | $65.1\pm 3.3$ | $68.0\pm 3.2$ |
| Qwen2.5-VL 7B | OpenWorld | Single | $81.0\pm 3.4$ | $75.4\pm 3.8$ |
|  |  | Batched | $88.2\pm 2.8$ | $81.2\pm 3.4$ |
|  | HOI | Single | $69.1\pm 3.2$ | $66.8\pm 3.3$ |
|  |  | Batched | $72.1\pm 3.1$ | $65.9\pm 3.3$ |
| Qwen2.5-VL 72B | OpenWorld | Single | $79.0\pm 3.6$ | $71.2\pm 4.0$ |
|  |  | Batched | $87.0\pm 2.9$ | $70.2\pm 4.0$ |
|  | HOI | Single | $69.2\pm 3.2$ | $60.6\pm 3.4$ |
|  |  | Batched | $72.0\pm 3.1$ | $58.5\pm 3.4$ |

The data indicates that the batched context method provides a significantly more accurate and stable measure of linear separability.
By averaging embeddings to create a conceptual prototype, the classifier becomes more resilient to individual outlier examples and better reflects the abstract visual rule.
For this reason, all LSC scores reported in the main body of this paper are calculated using this batched context methodology.

## Appendix C Separability ceiling in VLMs

**Table A.2: Generative performance vs. the LSC ($acc_{vision}$) and its final representation accuracy ($acc_{final}$). Superscripts on embedding scores denote significant (p<0.05) dependence (negative indicating inverse) with generative predictions (D=Direct, C=CoT).**
| Model | Dataset | Prompt Strategy | Direct acc (%) | CoT acc (%) | Sim. acc (vision, %) | Sim. acc (final, %) |
| --- | --- | --- | --- | --- | --- | --- |
| Phi | OpenWorld | Interleaved | $59.0\pm 4.3$ | $80.6\pm 3.5$ | $84.0\pm 3.2^{D}$ | $76.4\pm 3.7^{-D}$ |
|  |  | Interleaved query first | $52.4\pm 4.4$ | $68.0\pm 4.1$ | $84.0\pm 3.2^{D}$ | $66.0\pm 4.2^{D,C}$ |
|  |  | Labeled | $79.4\pm 3.5$ | $78.8\pm 3.6$ | $84.0\pm 3.2$ | $78.2\pm 3.6$ |
|  |  | Labeled query first | $57.2\pm 4.3$ | $64.0\pm 4.2$ | $84.0\pm 3.2^{D}$ | $65.0\pm 4.2^{D,C}$ |
|  | HOI | Interleaved | $52.1\pm 3.5$ | $59.9\pm 3.4$ | $71.9\pm 3.1^{C}$ | $60.5\pm 3.4^{-D}$ |
|  |  | Interleaved query first | $51.5\pm 3.5$ | $55.5\pm 3.4$ | $71.9\pm 3.1^{D,C}$ | $56.5\pm 3.4^{D,C}$ |
|  |  | Labeled | $58.8\pm 3.4$ | $57.4\pm 3.4$ | $71.9\pm 3.1^{C}$ | $60.6\pm 3.4^{-C}$ |
|  |  | Labeled query first | $55.1\pm 3.4$ | $55.8\pm 3.4$ | $71.9\pm 3.1^{D}$ | $56.1\pm 3.4^{D,C}$ |
| Pixtral | OpenWorld | Interleaved | $72.4\pm 3.9$ | $83.8\pm 3.2$ | $76.0\pm 3.7^{D,C}$ | $87.2\pm 2.9^{D,C}$ |
|  |  | Interleaved query first | $71.2\pm 4.0$ | $81.0\pm 3.4$ | $75.8\pm 3.8^{D,C}$ | $76.2\pm 3.7^{C}$ |
|  |  | Labeled | $79.4\pm 3.5$ | $84.2\pm 3.2$ | $76.0\pm 3.7^{D}$ | $88.0\pm 2.8^{D,C}$ |
|  |  | Labeled query first | $62.4\pm 4.2$ | $76.0\pm 3.7$ | $75.8\pm 3.8^{D,C}$ | $75.0\pm 3.8^{D,C}$ |
|  | HOI | Interleaved | $57.8\pm 3.4$ | $66.9\pm 3.3$ | $62.7\pm 3.4^{D,C}$ | $70.2\pm 3.2^{D,C}$ |
|  |  | Interleaved query first | $53.5\pm 3.5$ | $67.2\pm 3.3$ | $63.2\pm 3.3^{C}$ | $62.6\pm 3.4^{-D,C}$ |
|  |  | Labeled | $60.6\pm 3.4$ | $66.6\pm 3.3$ | $62.7\pm 3.4^{D,C}$ | $70.5\pm 3.2^{C}$ |
|  |  | Labeled query first | $53.4\pm 3.5$ | $65.4\pm 3.3$ | $63.2\pm 3.3^{C}$ | $61.1\pm 3.4^{D,C}$ |
| Gemma3 4B | OpenWorld | Interleaved | $76.0\pm 3.7$ | $80.2\pm 3.5$ | $89.8\pm 2.7^{C}$ | $50.0^{D,-C}$ |
|  |  | Interleaved query first | $50.2\pm 4.4$ | $51.4\pm 4.4$ | $89.8\pm 2.7^{D,C}$ | $50.0^{D,C}$ |
|  |  | Labeled | $68.6\pm 4.1$ | $83.0\pm 3.3$ | $89.8\pm 2.7^{C}$ | $50.0^{D}$ |
|  |  | Labeled query first | $52.0\pm 4.4$ | $52.4\pm 4.4$ | $89.8\pm 2.7^{D,C}$ | $50.0^{D,C}$ |
|  | HOI | Interleaved | $56.5\pm 3.4$ | $62.6\pm 3.4$ | $74.1\pm 3.0^{C}$ | $50.0^{D,-C}$ |
|  |  | Interleaved query first | $50.4\pm 3.5$ | $50.7\pm 3.5$ | $74.1\pm 3.0^{D,C}$ | $50.0^{D,C}$ |
|  |  | Labeled | $54.2\pm 3.5$ | $63.9\pm 3.3$ | $74.1\pm 3.0^{C}$ | $50.0^{D,-C}$ |
|  |  | Labeled query first | $50.7\pm 3.5$ | $51.7\pm 3.5$ | $74.1\pm 3.0$ | $50.0^{D,C}$ |
| Gemma3 27B | OpenWorld | Interleaved | $91.8\pm 2.4$ | $88.6\pm 2.8$ | $88.6\pm 2.8^{D,C}$ | $50.0^{-D,C}$ |
|  |  | Interleaved query first | $60.6\pm 4.3$ | $71.4\pm 4.0$ | $88.6\pm 2.8^{D,C}$ | $50.0^{D,C}$ |
|  |  | Labeled | $93.2\pm 2.2$ | $87.2\pm 2.9$ | $88.6\pm 2.8^{D}$ | $50.0^{C}$ |
|  |  | Labeled query first | $54.6\pm 4.4$ | $67.2\pm 4.1$ | $88.6\pm 2.8^{D,C}$ | $50.0^{D,C}$ |
|  | HOI | Interleaved | $75.1\pm 3.0$ | $75.2\pm 3.0$ | $73.4\pm 3.1^{D,C}$ | $50.0^{-D,C}$ |
|  |  | Interleaved query first | $54.4\pm 3.5$ | $65.0\pm 3.3$ | $73.4\pm 3.1^{D,C}$ | $50.0^{D,C}$ |
|  |  | Labeled | $76.2\pm 2.9$ | $75.1\pm 3.0$ | $73.4\pm 3.1^{D,C}$ | $50.0^{C}$ |
|  |  | Labeled query first | $52.2\pm 3.5$ | $59.6\pm 3.4$ | $73.4\pm 3.1^{D,C}$ | $50.0^{D,C}$ |
| InternVL 14B | OpenWorld | Interleaved | $79.4\pm 3.5$ | $80.6\pm 3.5$ | $70.2\pm 4.0$ | $61.6\pm 4.3$ |
|  |  | Interleaved query first | $80.4\pm 3.5$ | $70.4\pm 4.0$ | $70.2\pm 4.0$ | $59.4\pm 4.3^{C}$ |
|  |  | Labeled | $55.8\pm 4.4$ | $71.8\pm 3.9$ | $70.2\pm 4.0^{D}$ | $62.4\pm 4.2^{-D}$ |
|  |  | Labeled query first | $66.6\pm 4.1$ | $73.0\pm 3.9$ | $70.2\pm 4.0^{D,C}$ | $54.0\pm 4.4^{D,C}$ |
|  | HOI | Interleaved | $58.2\pm 3.4$ | $63.6\pm 3.3$ | $65.1\pm 3.3^{D,C}$ | $52.0\pm 3.5^{-D}$ |
|  |  | Interleaved query first | $64.2\pm 3.3$ | $60.4\pm 3.4$ | $65.1\pm 3.3^{D,C}$ | $52.8\pm 3.5^{D,C}$ |
|  |  | Labeled | $52.0\pm 3.5$ | $58.8\pm 3.4$ | $65.1\pm 3.3^{D}$ | $53.1\pm 3.5^{-D}$ |
|  |  | Labeled query first | $56.5\pm 3.4$ | $62.7\pm 3.4$ | $65.1\pm 3.3^{D,C}$ | $51.5\pm 3.5^{D,C}$ |
| InternVL3 78B | OpenWorld | Interleaved | $73.0\pm 3.9$ | $69.9\pm 4.1$ | $75.6\pm 3.8^{C}$ | $72.4\pm 3.9^{-C}$ |
|  |  | Interleaved query first | $88.4\pm 2.8$ | $79.4\pm 3.8$ | $75.6\pm 3.8^{D,C}$ | $69.8\pm 4.0^{C}$ |
|  |  | Labeled | $64.4\pm 4.2$ | $67.9\pm 6.7$ | $75.6\pm 3.8$ | $73.4\pm 3.9^{-D}$ |
|  |  | Labeled query first | $84.6\pm 3.2$ | $76.6\pm 4.8$ | $75.6\pm 3.8^{D,C}$ | $67.2\pm 4.1^{D,C}$ |
|  | HOI | Interleaved | $59.0\pm 3.4$ | $62.1\pm 3.4$ | $66.1\pm 3.3^{D,C}$ | $57.6\pm 3.4^{-C}$ |
|  |  | Interleaved query first | $66.1\pm 3.3$ | $62.0\pm 3.4$ | $66.1\pm 3.3^{D,C}$ | $58.6\pm 3.4^{-D,C}$ |
|  |  | Labeled | $54.0\pm 3.5$ | $60.4\pm 3.7$ | $66.1\pm 3.3^{D}$ | $60.4\pm 3.4^{-D,-C}$ |
|  |  | Labeled query first | $66.5\pm 3.3$ | $63.0\pm 3.5$ | $66.1\pm 3.3^{D}$ | $57.1\pm 3.4^{D,C}$ |
| Qwen2.5-VL 7B | OpenWorld | Interleaved | $86.4\pm 3.0$ | $84.6\pm 3.2$ | $87.8\pm 2.9^{D,C}$ | $58.6\pm 4.3^{-D,-C}$ |
|  |  | Interleaved query first | $51.2\pm 4.4$ | $78.2\pm 3.6$ | $87.8\pm 2.9^{-D,C}$ | $54.0\pm 4.4^{-D,C}$ |
|  |  | Labeled | $92.2\pm 2.4$ | $87.2\pm 2.9$ | $87.8\pm 2.9^{D,C}$ | $57.0\pm 4.3^{-C}$ |
|  |  | Labeled query first | $54.8\pm 4.4$ | $80.6\pm 3.5$ | $87.8\pm 2.9^{-D,C}$ | $54.8\pm 4.4^{-D,C}$ |
|  | HOI | Interleaved | $63.0\pm 3.3$ | $67.4\pm 3.2$ | $72.2\pm 3.1^{D,C}$ | $52.2\pm 3.5^{-D,-C}$ |
|  |  | Interleaved query first | $50.4\pm 3.5$ | $60.8\pm 3.4$ | $72.2\pm 3.1^{C}$ | $51.1\pm 3.5^{-D,C}$ |
|  |  | Labeled | $70.4\pm 3.2$ | $68.4\pm 3.2$ | $72.2\pm 3.1^{D,C}$ | $50.5\pm 3.5^{D,-C}$ |
|  |  | Labeled query first | $50.6\pm 3.5$ | $63.0\pm 3.3$ | $72.2\pm 3.1^{C}$ | $51.7\pm 3.5^{-D,C}$ |
| Qwen2.5-VL 72B | OpenWorld | Interleaved | $93.6\pm 2.1$ | $89.2\pm 2.7$ | $86.8\pm 3.0^{D,C}$ | $59.0\pm 4.3$ |
|  |  | Interleaved query first | $91.6\pm 2.4$ | $91.8\pm 2.4$ | $86.8\pm 3.0^{D,C}$ | $68.2\pm 4.1^{D,C}$ |
|  |  | Labeled | $93.6\pm 2.1$ | $93.2\pm 2.2$ | $86.8\pm 3.0^{D,C}$ | $74.8\pm 3.8^{D,C}$ |
|  |  | Labeled query first | $91.8\pm 2.4$ | $92.2\pm 2.4$ | $86.8\pm 3.0^{D,C}$ | $65.2\pm 4.2^{C}$ |
|  | HOI | Interleaved | $79.9\pm 2.8$ | $78.5\pm 2.8$ | $72.1\pm 3.1^{D,C}$ | $53.9\pm 3.5^{-C}$ |
|  |  | Interleaved query first | $77.1\pm 2.9$ | $77.2\pm 2.9$ | $72.1\pm 3.1^{D,C}$ | $59.4\pm 3.4^{D,C}$ |
|  |  | Labeled | $80.2\pm 2.8$ | $79.2\pm 2.8$ | $72.1\pm 3.1^{D,C}$ | $63.7\pm 3.3^{D,C}$ |
|  |  | Labeled query first | $76.0\pm 3.0$ | $77.5\pm 2.9$ | $72.1\pm 3.1^{D,C}$ | $56.2\pm 3.4^{D,C}$ |

## Appendix D Hyperparameter details

LoRA and projector tuning employed a learning rate of $1e^{-4}$ and a batch size of 25.
The LoRA experiments utilized a rank of $r=8$ and a scaling factor of $\alpha=8$.
These hyperparameters are consistent with established practices for fine-tuning VLMs and training LoRA adapters (Hu et al., 2022; Liu et al., 2023; Microsoft, 2025).

Prompt tuning method reported in the main paper employs a hybrid approach,
prepending and appending learnable soft prompt embeddings to the input sequence concurrently.
The final structure consists of a prefix prompt, 100 learnable embeddings prepended to the standard interleaved prompt.
And instructional postfix prompt, a fixed, human-readable instruction designed to encourage meta-learning, followed by 100 learnable embeddings.

The fixed instruction text is:

Postfix tuning method employs postfix part of prompt tuning only.

Hyperparameter search revealed that an aggressive configuration with a small batch size (2) and a high learning rate (0.01), was highly effective for postfix tuning.
Prompt tuning uses a learning rate of $1e^{-4}$ and a batch size of 25.

## Appendix E Combined loss details

Inspired by the effectiveness of similarity measures in assessing embedding quality (Section [5](https://arxiv.org/html/2507.07574v2#S5)), we explored an additional objective for tuning.
Instead of relying solely on the standard next-token prediction loss ($\mathcal{L}_{\text{NT}}$), we incorporated a similarity-based contrastive loss ($\mathcal{L}_{\text{sim}}$) designed to directly enhance the discriminability of the final embeddings with respect to the task structure during tuning.

Specifically, let $V_{T}$ be the normalized mean-pooled embedding vector derived from the final hidden states corresponding to the query image $T$.
Similarly, let $C_{P}$ and $C_{N}$ be the normalized mean-pooled centroids derived from the concatenated final hidden states of all images in the positive set $P=\{p_{1},...,p_{k}\}$ and negative set $N=\{n_{1},...,n_{k}\}$, respectively.
These embeddings are obtained after the model processes the full input sequence, including the tunable soft prompt tokens.
We calculate the cosine similarities between the query image embedding and the set centroids: $s_{P}=V_{T}\cdot C_{P}$ and $s_{N}=V_{T}\cdot C_{N}$.

The similarity-based loss, $\mathcal{L}_{\text{sim}}$, is then formulated as a cross-entropy loss over these similarities, akin to InfoNCE, encouraging the query image embedding to be closer to the centroid of its true category set:

$$ $\mathcal{L}_{\text{sim}}=\text{CrossEntropy}([\frac{s_{P}}{\tau},\frac{s_{N}}{\tau}],y)$ $$

where $\tau$ (fixed to 0.07) is a temperature hyperparameter scaling the logits, and $y$ is the target label ($y=0$ if the ground truth category for $T$ is the positive set $P$, and $y=1$ if it is the negative set $N$).
This contrastive loss term was combined with the standard next-token prediction loss, weighted by hyperparameters $w_{n}$ and $w_{c}$:

$$ $\mathcal{L}_{\text{combined}}=w_{n}\mathcal{L}_{\text{NT}}+w_{c}\mathcal{L}_{\text{sim}}$ $$

The gradients from this combined loss $\mathcal{L}_{\text{combined}}$ were used to update the parameters.
This approach directly optimizes final embeddings to be well-clustered according to the Bongard task’s positive/negative distinction, complementing the language modeling objective.

## Appendix F Combined loss hyperparameter search

To find hyperparameters for $\mathcal{L}_{\text{combined}}=w_{n}\mathcal{L}_{\text{NT}}+w_{c}\mathcal{L}_{\text{sim}}$, we fixed the $w_{n}$ to 1 and scanned through $w_{c}$ values with different schedules.
Constant schedule had same $w_{c}$ throughout different epochs, which proved unstable for larger values.
Linear schedule had a target value where every epoch the $w_{c}=\frac{1+\text{cur_epoch}}{\text{total_epochs}}\cdot\text{target_value}$.
Using linear schedule, scan was conducted over [0.1, 0.2, 0.4, 0.8, 1.6, 2.4, 3.2] target values.
For Phi and Gemma3 the best value was 0.4 and for Pixtral best value was 1.6, which is shown on Figure [A.1](https://arxiv.org/html/2507.07574v2#A6.F1).

Figure: Figure A.1: Pixtral contrastive loss hyperparameter scan
Refer to caption: bongard_images/pixtral_hoi_contrastive_loss_scan.png

Afterwards, target values were tested with different learning rates and since they proved stable, we picked these values.
However, as seen with target value of 1.6 with learning rate 0.0001, the accuracy peaked quite early on.
To address this early peaking and ensure a more controlled dynamic where the contrastive loss provides an initial strong signal that gradually yields to the NTP objective for final model refinement,
we introduced a cosine learning schedule which has two separate schedules for $w_{n}$ and $w_{c}$, governed by the overall training progress.
Let $p=\frac{\text{current_step}+1}{\text{total_steps}}$ represent the fraction of training completion, where current_step is the current training iteration (e.g., batch or epoch, 0-indexed) and total_steps is the total number of such iterations for the run.

The weight for the next-token prediction loss, $w_{n}$, is designed to smoothly increase, following the formula:

$$ $w_{n}(p)=1-\cos\left(\frac{\pi}{2}\cdot p\right)$ $$

This schedule ensures that the next-token prediction (NTP) objective becomes increasingly dominant as training nears completion, aligning with the goal for NTP to be the primary objective by the end of the run.
Concurrently, the weight for the contrastive loss, $w_{c}$, is determined by a separate schedule:

$$ $w_{c}(p)=\cos\left(\frac{\pi}{2}\cdot p\right)\cdot\min\left(2p,1\right)\cdot C$ $$

Here, $C$ represents a constant scaling factor (0.4 for Phi and Gemma3 and 1.6 for Pixtral).
This formulation for $w_{c}$ incorporates two dynamic components based on the training progress $p$.
A cosine decay term, $\cos\left(\frac{\pi}{2}\cdot p\right)$, which smoothly transitions the base weight from $1$ down to 0.
A linear ramp-up term, $\min\left(2\cdot p,1\right)$, which effectively scales the influence of the cosine term.
This ramp-up is active during the first half of the training duration, remaining at $1$ for the second half.
The product of these terms, further scaled by $C$, allows the contrastive loss to have an initial warm-up period and then a gradually diminishing influence as the training increasingly prioritizes the NTP objective towards its conclusion.
Both prompt tuning and LoRA training was conducted using cosinel schedule.
For prompt tuning progress was calculated using epochs and for LoRA, progress was calculated using batches.

## Appendix G LoRA on vision encoder

To further investigate the claim that the primary performance bottleneck resides within the LLM’s reasoning pathways, we conducted an ablation study.
We compare the final generative accuracy of our standard approach, where LoRA is applied to both the vision encoder and language model,
against a configuration where same LoRA is applied only to the language model’s attention layers.
This analysis was performed across all three models, both datasets, and both training objectives to ensure the robustness of our findings.

**Table A.3: Ablation of Vision Encoder LoRA. This table compares the final generative accuracy (%) when LoRA is applied to both the vision encoder and LLM versus only the LLM. The comparison is shown for both the standard next-token prediction ($\mathcal{L}_{\text{NT}}$) and the combined contrastive ($\mathcal{L}_{\text{combined}}$) objectives. The results show no statistically significant performance difference across these configurations.**
| Model | Dataset | Method | Both<br>(VE + LLM) | LLM-only |
| --- | --- | --- | --- | --- |
| Phi | OpenWorld | Direct baseline | $59.0\pm 4.3$ | $59.0\pm 4.3$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.2\pm 2.4$ | $92.4\pm 2.3$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $95.6\pm 1.8$ | $95.6\pm 1.8$ |
|  | HOI | Direct baseline | $52.1\pm 3.5$ | $52.1\pm 3.5$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $78.6\pm 2.8$ | $77.5\pm 2.9$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $79.2\pm 2.8$ | $77.9\pm 2.9$ |
| Pixtral | OpenWorld | Direct baseline | $72.4\pm 3.9$ | $72.4\pm 3.9$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $93.4\pm 2.2$ | $93.6\pm 2.1$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $95.0\pm 1.9$ | $94.8\pm 1.9$ |
|  | HOI | Direct baseline | $57.8\pm 3.4$ | $57.8\pm 3.4$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $78.0\pm 2.9$ | $78.9\pm 2.8$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $79.6\pm 2.8$ | $79.5\pm 2.8$ |
| Gemma3 4B | OpenWorld | Direct baseline | $76.0\pm 3.7$ | $76.0\pm 3.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.4\pm 2.3$ | $92.4\pm 2.3$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $95.6\pm 1.8$ | $95.6\pm 1.8$ |
|  | HOI | Direct baseline | $56.5\pm 3.4$ | $56.5\pm 3.4$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $84.2\pm 2.5$ | $84.2\pm 2.5$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $84.2\pm 2.5$ | $84.2\pm 2.5$ |

The results presented in Table [A.3](https://arxiv.org/html/2507.07574v2#A7.T3) are remarkably consistent.
In every tested scenario, there is no statistically significant performance difference between applying LoRA adapters to the LLM alone versus applying them to both the vision encoder and the LLM.

This provides evidence that the LoRA adapters on the vision encoder are redundant for improving final task performance.
It confirms that the gains from fine-tuning are not derived from adapting the initial feature extraction process,
but rather from unlocking and refining the latent reasoning capabilities within the language model.
This holds true even for the $\mathcal{L}_{\text{combined}}$ objective, demonstrating that co-adapting the vision encoder is unnecessary for the LLM to learn how to structure its final representations more effectively.

## Appendix H PEFT comparison

To understand how best to resolve the alignment gap, we compared two primary PEFT methods: prompt tuning,
which seeks to activate latent abilities by finding an optimal input, and LoRA, which adapts the model’s core weights.
We evaluated these methods using two distinct training objectives:
the standard next-token prediction loss ($\mathcal{L}_{\text{NT}}$) and our combined objective ($\mathcal{L}_{\text{combined}}$),
which adds a contrastive loss to explicitly improve representation quality.
To further probe the limits of these methods, we tested both default and aggressive hyperparameter configurations for prompt-based tuning.

Our evaluation also measured structural robustness.
The in-distribution (ID) test uses the standard interleaved prompt structure seen during training,
while the out-of-distribution (OOD) test uses a labeled prompt structure that groups images by category,
challenging the model’s ability to generalize beyond the training syntax.

**Table A.4: Comparison of PEFT methods. In the generative columns, bold indicates the the LSC is surpassed. Superscripts on embedding classification scores indicate a statistically significant ($p~<~0.05$) dependence between the item-level predictions of the similarity-based classifier and a generative method: I for the ID method and O for the OOD method. My default we use batch size of 25 and learning rate of $1e^{-4}$. Postfix tuning uses an aggressive batch size of 2 and learning rate of 0.01.**
| Model | Dataset | Method | Generative<br>(ID, %) | Generative<br>(OOD, %) | LSC<br>(%) | Final rep.<br>(ID, %) | Final rep.<br>(OOD, %) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Phi | OpenWorld | Direct baseline | $59.0\pm 4.3$ | $79.4\pm 3.5$ | $84.0\pm 3.2^{I}$ | $76.4\pm 3.7^{-I}$ | $78.2\pm 3.6$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{94.2\pm 2.0}$ | $\mathbf{90.2\pm 2.6}$ | $84.2\pm 3.2$ | $77.4\pm 3.7^{I}$ | $78.8\pm 3.6^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{90.4\pm 2.6}$ | $83.4\pm 3.3$ | $84.2\pm 3.2$ | $74.6\pm 3.8^{I}$ | $76.6\pm 3.7^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $\mathbf{94.4\pm 2.0}$ | $86.0\pm 3.0$ | $84.2\pm 3.2^{I}$ | $94.2\pm 2.0^{I}$ | $93.2\pm 2.2$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{92.2\pm 2.4}$ | $\mathbf{89.8\pm 2.7}$ | $84.4\pm 3.2^{I}$ | $74.8\pm 3.8$ | $78.2\pm 3.6^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.6\pm 1.8}$ | $\mathbf{94.4\pm 2.0}$ | $84.4\pm 3.2^{I}$ | $93.8\pm 2.1^{I}$ | $91.2\pm 2.5^{O}$ |
|  | HOI | Direct baseline | $52.1\pm 3.5$ | $58.8\pm 3.4$ | $71.9\pm 3.1$ | $60.5\pm 3.4^{-I}$ | $60.6\pm 3.4$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $63.2\pm 3.3$ | $59.1\pm 3.4$ | $71.9\pm 3.1^{I,O}$ | $60.9\pm 3.4^{I}$ | $61.8\pm 3.4^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $58.5\pm 3.4$ | $52.6\pm 3.5$ | $71.9\pm 3.1^{I}$ | $59.9\pm 3.4^{I}$ | $61.9\pm 3.4^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $65.4\pm 3.3$ | $62.1\pm 3.4$ | $71.9\pm 3.1^{I}$ | $68.8\pm 3.2^{I}$ | $62.9\pm 3.3^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{78.6\pm 2.8}$ | $72.8\pm 3.1$ | $71.9\pm 3.1^{I,O}$ | $63.6\pm 3.3^{I}$ | $64.4\pm 3.3$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{79.2\pm 2.8}$ | $73.5\pm 3.1$ | $71.9\pm 3.1^{I,O}$ | $82.0\pm 2.7^{I}$ | $82.9\pm 2.6^{O}$ |
| Pixtral | OpenWorld | Direct baseline | $72.4\pm 3.9$ | $79.4\pm 3.5$ | $76.0\pm 3.7^{I,O}$ | $87.2\pm 2.9^{I}$ | $88.0\pm 2.8^{O}$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{93.6\pm 2.1}$ | $\mathbf{93.6\pm 2.1}$ | $76.6\pm 3.7$ | $87.6\pm 2.9$ | $87.6\pm 2.9$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $\mathbf{93.6\pm 2.1}$ | $79.2\pm 3.8$ | $76.6\pm 3.7^{I}$ | $78.4\pm 3.6$ | $83.0\pm 3.3^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $\mathbf{94.4\pm 2.0}$ | $\mathbf{88.2\pm 2.8}$ | $76.6\pm 3.7$ | $95.0\pm 1.9^{I}$ | $93.4\pm 2.2^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{93.4\pm 2.2}$ | $\mathbf{90.0\pm 2.6}$ | $76.6\pm 3.7$ | $87.2\pm 2.9^{I}$ | $89.6\pm 2.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.0\pm 1.9}$ | $\mathbf{94.6\pm 2.0}$ | $74.4\pm 3.8$ | $96.2\pm 1.7^{I}$ | $95.4\pm 1.8^{O}$ |
|  | HOI | Direct baseline | $57.8\pm 3.4$ | $60.6\pm 3.4$ | $62.7\pm 3.4^{I,O}$ | $70.2\pm 3.2^{I}$ | $70.5\pm 3.2$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $66.4\pm 3.3$ | $66.4\pm 3.3$ | $62.7\pm 3.4^{I,O}$ | $71.0\pm 3.1^{I}$ | $71.0\pm 3.1^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $58.6\pm 3.4$ | $54.0\pm 3.5$ | $62.7\pm 3.4^{I}$ | $62.6\pm 3.4^{I}$ | $63.1\pm 3.3^{-O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $\mathbf{70.8\pm 3.2}$ | $58.9\pm 3.4$ | $62.7\pm 3.4^{I}$ | $73.5\pm 3.1^{I}$ | $72.2\pm 3.1^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{78.0\pm 2.9}$ | $\mathbf{74.4\pm 3.0}$ | $61.6\pm 3.4^{I,O}$ | $74.9\pm 3.0^{I}$ | $73.2\pm 3.1^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{79.6\pm 2.8}$ | $62.0\pm 3.4$ | $63.1\pm 3.3^{I}$ | $77.8\pm 2.9^{I}$ | $77.6\pm 2.9^{O}$ |
| Gemma3 4B | OpenWorld | Direct baseline | $76.0\pm 3.7$ | $68.6\pm 4.1$ | $89.8\pm 2.7$ | $50.0^{I}$ | $50.0^{O}$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $86.0\pm 3.0$ | $86.0\pm 3.0$ | $89.8\pm 2.7$ | $50.0$ | $50.0$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $86.0\pm 3.0$ | $71.2\pm 4.0$ | $89.8\pm 2.7^{I}$ | $50.0$ | $50.2\pm 4.4^{O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $82.2\pm 3.4$ | $64.6\pm 4.2$ | $89.8\pm 2.7^{I}$ | $50.8\pm 4.4^{-I}$ | $50.4\pm 4.4^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.4\pm 2.3$ | $\mathbf{94.0\pm 2.1}$ | $89.8\pm 2.7^{I}$ | $50.0$ | $50.0$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{95.6\pm 1.8}$ | $\mathbf{96.2\pm 1.7}$ | $89.8\pm 2.7^{O}$ | $96.6\pm 1.6^{I}$ | $96.8\pm 1.5^{O}$ |
|  | HOI | Direct baseline | $56.5\pm 3.4$ | $54.2\pm 3.5$ | $74.1\pm 3.0$ | $50.0^{I}$ | $50.0^{O}$ |
|  |  | Postfix tuning ($\mathcal{L}_{\text{NT}}$) | $56.1\pm 3.4$ | $56.1\pm 3.4$ | $74.1\pm 3.0$ | $50.0^{-I}$ | $50.0^{-O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{NT}}$) | $54.1\pm 3.5$ | $51.6\pm 3.5$ | $74.1\pm 3.0^{O}$ | $50.0^{I}$ | $50.0^{-O}$ |
|  |  | Prompt tuning ($\mathcal{L}_{\text{combined}}$) | $58.9\pm 3.4$ | $54.0\pm 3.5$ | $74.1\pm 3.0^{I}$ | $50.0^{-I}$ | $50.0^{O}$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $\mathbf{84.2\pm 2.5}$ | $\mathbf{81.8\pm 2.7}$ | $74.1\pm 3.0^{I,O}$ | $50.0$ | $50.0$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $\mathbf{84.2\pm 2.5}$ | $67.4\pm 3.2$ | $74.1\pm 3.0^{I}$ | $83.2\pm 2.6^{I}$ | $82.1\pm 2.7^{O}$ |

The results in Table [A.4](https://arxiv.org/html/2507.07574v2#A8.T4) reveal two key insights regarding the methods for resolving the alignment gap.

Mechanistic divergence towards a common goal.
A primary finding is that multiple strategies exist to achieve high generative accuracy.
For instance, postfix tuning with aggressive hyperparameters can yield generative performance comparable to methods using the $\mathcal{L}_{\text{combined}}$ objective.
However, this equivalence in outcome should not be mistaken for an equivalence in mechanism.
The aggressive $\mathcal{L}_{\text{NT}}$ tuning likely discovers a more effective non-linear decision logic,
enhancing the model’s computational pathway without explicitly improving the linear structure of its representations.
In contrast, the $\mathcal{L}_{\text{combined}}$ objective provides a more principled method for enhancing the linear separability of the final embeddings themselves.
Thus, while both can surpass the LSC, they do so via different strategies: one by optimizing the downstream computation,
the other by directly structuring the representations for that computation.

A trade-off between representation quality and structural robustness.
The second key insight is a trade-off between representation quality and generalization to new prompt formats.
Methods that are less invasive or only intervene late in the sequence, such as LoRA with $\mathcal{L}_{\text{NT}}$ and especially postfix tuning, demonstrate strong structural robustness, maintaining high performance on OOD prompts. Conversely, methods that either explicitly restructure representations ($\mathcal{L}_{\text{combined}}$) or introduce learnable tokens at the start of the sequence (standard prompt tuning) can exhibit brittleness. For example, on the HOI dataset, LoRA trained with $\mathcal{L}_{\text{combined}}$ shows a significant performance drop on the OOD prompt, despite its final representations remaining highly separable. This suggests that forcing a specific geometric structure can lead to syntactic overfitting, where the model’s generative pathway learns to depend on the training prompt’s structure, compromising its ability to reason flexibly.

## Appendix I Results across HOI splits

Table [A.5](https://arxiv.org/html/2507.07574v2#A9.T5) provides a detailed breakdown of model performance across the four distinct test splits of the Bongard-HOI dataset.

**Table A.5: Performance on HOI - detailed breakdown**
| Model | Method | Objective | Seen obj.<br>Seen act. (%) | Seen obj.<br>Unseen act. (%) | Unseen obj.<br>Seen act. (%) | Unseen obj.<br>Unseen act. (%) |
| --- | --- | --- | --- | --- | --- | --- |
| Phi | Baseline gen. |  | $51.5\pm 6.9$ | $55.0\pm 6.9$ | $51.0\pm 6.9$ | $51.0\pm 6.9$ |
|  | Baseline sim. |  | $61.5\pm 6.7$ | $60.0\pm 6.8$ | $57.5\pm 6.9$ | $63.0\pm 6.7$ |
|  | Prompt tuning gen. |  | $60.5\pm 6.8$ | $59.0\pm 6.8$ | $52.5\pm 6.9$ | $62.0\pm 6.7$ |
|  | Prompt tuning sim. |  | $61.5\pm 6.7$ | $59.0\pm 6.8$ | $57.0\pm 6.9$ | $62.0\pm 6.7$ |
|  | Prompt tuning gen. | $\mathcal{L}_{\text{sim}}$ | $66.5\pm 6.5$ | $69.0\pm 6.4$ | $62.5\pm 6.7$ | $63.5\pm 6.7$ |
|  | Prompt tuning sim. | $\mathcal{L}_{\text{sim}}$ | $63.5\pm 6.7$ | $73.5\pm 6.1$ | $70.5\pm 6.3$ | $67.5\pm 6.5$ |
|  | LoRA gen. | $\mathcal{L}_{\text{NT}}$ | $79.5\pm 5.6$ | $79.5\pm 5.6$ | $75.5\pm 6.0$ | $80.0\pm 5.5$ |
|  | LoRA sim. | $\mathcal{L}_{\text{NT}}$ | $65.0\pm 6.6$ | $61.5\pm 6.7$ | $61.5\pm 6.7$ | $66.5\pm 6.5$ |
|  | LoRA gen. | $\mathcal{L}_{\text{combined}}$ | $77.0\pm 5.8$ | $77.5\pm 5.8$ | $81.0\pm 5.4$ | $81.5\pm 5.4$ |
|  | LoRA sim. | $\mathcal{L}_{\text{combined}}$ | $82.0\pm 5.3$ | $81.0\pm 5.4$ | $77.5\pm 5.8$ | $87.5\pm 4.6$ |
| Pixtral | Baseline gen. |  | $60.0\pm 6.8$ | $61.5\pm 6.7$ | $51.5\pm 6.9$ | $58.0\pm 6.8$ |
|  | Baseline sim. |  | $66.5\pm 6.5$ | $73.5\pm 6.1$ | $69.5\pm 6.4$ | $71.5\pm 6.3$ |
|  | Prompt tuning gen. |  | $60.5\pm 6.8$ | $58.5\pm 6.8$ | $56.5\pm 6.9$ | $59.0\pm 6.8$ |
|  | Prompt tuning sim. |  | $63.0\pm 6.7$ | $62.0\pm 6.7$ | $62.5\pm 6.7$ | $63.0\pm 6.7$ |
|  | Prompt tuning gen. | $\mathcal{L}_{\text{sim}}$ | $73.0\pm 6.2$ | $74.0\pm 6.1$ | $64.0\pm 6.7$ | $72.0\pm 6.2$ |
|  | Prompt tuning sim. | $\mathcal{L}_{\text{sim}}$ | $76.0\pm 5.9$ | $73.5\pm 6.1$ | $70.0\pm 6.4$ | $74.5\pm 6.0$ |
|  | LoRA gen. | $\mathcal{L}_{\text{NT}}$ | $77.0\pm 5.8$ | $79.5\pm 5.6$ | $75.5\pm 6.0$ | $80.0\pm 5.5$ |
|  | LoRA sim. | $\mathcal{L}_{\text{NT}}$ | $73.0\pm 6.2$ | $77.5\pm 5.8$ | $71.0\pm 6.3$ | $78.0\pm 5.7$ |
|  | LoRA gen. | $\mathcal{L}_{\text{combined}}$ | $78.5\pm 5.7$ | $81.5\pm 5.4$ | $74.5\pm 6.0$ | $84.0\pm 5.1$ |
|  | LoRA sim. | $\mathcal{L}_{\text{combined}}$ | $79.5\pm 5.6$ | $78.5\pm 5.7$ | $74.5\pm 6.0$ | $78.5\pm 5.7$ |
| Gemma3 4B | Baseline gen. |  | $58.5\pm 6.8$ | $56.0\pm 6.9$ | $53.0\pm 6.9$ | $58.5\pm 6.8$ |
|  | Baseline sim. |  | $50.0$ | $50.0$ | $50.0$ | $50.0$ |
|  | Prompt tuning gen. |  | $53.0\pm 6.9$ | $56.5\pm 6.9$ | $53.0\pm 6.9$ | $54.0\pm 6.9$ |
|  | Prompt tuning sim. |  | $50.0$ | $50.0$ | $50.0$ | $50.0$ |
|  | Prompt tuning gen. | $\mathcal{L}_{\text{sim}}$ | $56.5\pm 6.9$ | $61.5\pm 6.7$ | $59.5\pm 6.8$ | $58.0\pm 6.8$ |
|  | Prompt tuning sim. | $\mathcal{L}_{\text{sim}}$ | $50.0$ | $50.0$ | $50.0$ | $50.0$ |
|  | LoRA gen. | $\mathcal{L}_{\text{NT}}$ | $83.5\pm 5.1$ | $84.5\pm 5.0$ | $85.0\pm 4.9$ | $84.0\pm 5.1$ |
|  | LoRA sim. | $\mathcal{L}_{\text{NT}}$ | $50.0$ | $50.0$ | $50.0$ | $50.0$ |
|  | LoRA gen. | $\mathcal{L}_{\text{combined}}$ | $83.0\pm 5.2$ | $84.0\pm 5.1$ | $85.0\pm 4.9$ | $85.0\pm 4.9$ |
|  | LoRA sim. | $\mathcal{L}_{\text{combined}}$ | $84.0\pm 5.1$ | $86.5\pm 4.7$ | $82.5\pm 5.3$ | $80.0\pm 5.5$ |

There is no significant accuracy degradation when models are tested on unseen objects, unseen actions, or both simultaneously.
In fact, for both Phi and Pixtral, the highest generative accuracy with the combined loss is achieved on the most difficult split (unseen obj. / unseen act.),
providing strong evidence that the models are learning the abstract relational concept rather than memorizing training examples.

## Appendix J Domain generalization

To rigorously assess whether the reasoning skills learned during fine-tuning are truly generalizable, we conducted two distinct cross-domain evaluations.
The first tests for generalization across different Bongard-style tasks (OpenWorld and HOI).
The second, more challenging evaluation tests whether the learned skills transfer to a task with a completely different structure:
the text-image retrieval task of the Winoground dataset.

### J.1 Cross-Bongard generalization

In this evaluation, models were fine-tuned on one Bongard dataset (e.g., OpenWorld) and evaluated on the other (e.g., HOI)
to assess their ability to generalize beyond the training domain’s specific concepts and structures.
The dataset column in Table [A.6](https://arxiv.org/html/2507.07574v2#A10.T6) indicates the evaluation dataset (i.e., the model was trained on the other dataset).

**Table A.6: Cross-Bongard generalization performance. Models were trained on one dataset and evaluated on the other. bold indicates surpassing of LSC. Superscripts on similarity scores denote a significant dependence (p < 0.05) with the predictions of the corresponding generative method (G).**
| Model | Dataset | Method | Generative<br>(OOD, %) | LSC<br>(%) | Final rep.<br>(OOD, %) |
| --- | --- | --- | --- | --- | --- |
| Phi | OpenWorld | Direct baseline | $79.4\pm 3.5$ | $84.0\pm 3.2$ | $78.2\pm 3.6$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{NT}}$) | $\mathbf{90.0\pm 2.6}$ | $84.4\pm 3.2^{G}$ | $78.4\pm 3.6^{G}$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{combined}}$) | $\mathbf{92.8\pm 2.3}$ | $84.4\pm 3.2^{G}$ | $89.2\pm 2.7^{G}$ |
|  | HOI | Direct baseline | $58.8\pm 3.4$ | $71.9\pm 3.1$ | $60.6\pm 3.4$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{NT}}$) | $64.6\pm 3.3$ | $71.9\pm 3.1^{G}$ | $60.6\pm 3.4$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{combined}}$) | $67.4\pm 3.2$ | $71.9\pm 3.1^{G}$ | $67.2\pm 3.3^{G}$ |
| Pixtral | OpenWorld | Direct baseline | $79.4\pm 3.5$ | $76.0\pm 3.7^{G}$ | $88.0\pm 2.8^{G}$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{NT}}$) | $\mathbf{88.8\pm 2.8}$ | $76.6\pm 3.7^{G}$ | $89.0\pm 2.7^{G}$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{combined}}$) | $\mathbf{83.4\pm 3.3}$ | $74.4\pm 3.8^{G}$ | $87.4\pm 2.9^{G}$ |
|  | HOI | Direct baseline | $60.6\pm 3.4$ | $62.7\pm 3.4^{G}$ | $70.5\pm 3.2$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{NT}}$) | $64.9\pm 3.3$ | $61.6\pm 3.4^{G}$ | $72.5\pm 3.1^{G}$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{combined}}$) | $\mathbf{71.0\pm 3.1}$ | $63.1\pm 3.3^{G}$ | $68.4\pm 3.2^{G}$ |
| Gemma3 4B | OpenWorld | Direct baseline | $68.6\pm 4.1$ | $89.8\pm 2.7$ | $50.0^{G}$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{NT}}$) | $87.2\pm 2.9$ | $89.8\pm 2.7^{G}$ | $50.0$ |
|  |  | LoRA (HOI, $\mathcal{L}_{\text{combined}}$) | $83.4\pm 3.3$ | $89.8\pm 2.7$ | $83.2\pm 3.3^{G}$ |
|  | HOI | Direct baseline | $54.2\pm 3.5$ | $74.1\pm 3.0$ | $50.0^{G}$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{NT}}$) | $69.0\pm 3.2$ | $74.1\pm 3.0^{G}$ | $50.0$ |
|  |  | LoRA (OpenWorld, $\mathcal{L}_{\text{combined}}$) | $65.8\pm 3.3$ | $74.1\pm 3.0^{G}$ | $67.4\pm 3.2^{G}$ |

The results indicate that models can generalize their reasoning strategies to novel concepts within the same task paradigm.
Pixtral demonstrates the most robust generalization, surpassing its LSC in both transfer directions.
Phi shows strong generalization when trained on the broader relational concepts of HOI and tested on OpenWorld, but this success is not bidirectional.
Across all models, a key observation is that generative accuracy remains tightly coupled with the final representation classification accuracy
when trained with the combined loss ($\mathcal{L}_{\text{combined}}$),
suggesting the learned alignment is a general property, not specific to the training data distribution

### J.2 Generalization to Winoground

To test if fine-tuning imparts a more fundamental reasoning capability, we evaluated the models on the Winoground benchmark without task-specific adaptation.
This benchmark provides a compelling test case, as its text-retrieval component directly evaluates the inter-image comparison skills honed during our training.
The results in Table [A.7](https://arxiv.org/html/2507.07574v2#A10.T7) reveal how the nature of the training task and the fine-tuning objective dictates the success of cross-task generalization.

**Table A.7: Zero-shot cross-task generalization to the Winoground dataset. The CLIP Baseline is a linear probe on the vision encoder of the Phi model (and it’s respective text encoder) serves as an LSC-like benchmark for that model.**
| Model | Train dataset | Objective | Text retrieval acc. (%) | Image retrieval acc. (%) |
| --- | --- | --- | --- | --- |
| Phi | Baseline | 16.75 | 2.50 |  |
|  | CLIP Baseline (ViT-L/14) | 27.75 | 11.75 |  |
|  | OpenWorld | $\mathcal{L}_{\text{NT}}$ | 24.00 | 14.75 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 18.75 | 1.75 |
|  | HOI | $\mathcal{L}_{\text{NT}}$ | 18.25 | 13.50 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 29.25 | 14.00 |
| Pixtral | Baseline | 28.75 | 4.25 |  |
|  | OpenWorld | $\mathcal{L}_{\text{NT}}$ | 32.75 | 19.75 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 33.25 | 8.50 |
|  | HOI | $\mathcal{L}_{\text{NT}}$ | 43.50 | 24.75 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 54.75 | 12.25 |
| Gemma3 4B | Baseline | 5.25 | 0.75 |  |
|  | OpenWorld | $\mathcal{L}_{\text{NT}}$ | 5.75 | 1.00 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 12.25 | 2.75 |
|  | HOI | $\mathcal{L}_{\text{NT}}$ | 5.75 | 3.25 |
|  |  | $\mathcal{L}_{\text{combined}}$ | 12.25 | 13.25 |

Task-specific skill transfer.
Generalization is highly dependent on the alignment between the skills cultivated during training and those required by the evaluation task.
Fine-tuning on OpenWorld, which focuses on atomic semantic concepts, resulted in poor transfer to the compositional reasoning required by Winoground.
This mirrors our findings in the cross-Bongard evaluation,
suggesting that the model learned a task-specific strategy that was not broadly applicable to compositional challenges.

Objective-driven alignment.
Training on the relational HOI dataset proved far more effective for transferring to Winoground.
Within this setting, the $\mathcal{L}_{\text{combined}}$ objective consistently improved text retrieval accuracy
over the standard $\mathcal{L}_{\text{NT}}$ loss across all models.
This is most pronounced for Pixtral, whose accuracy jumps from 43.50% to 54.75%.
This demonstrates that our combined objective successfully aligns the model’s structured visual representations and its generative pathway.

Re-emergence of the LSC.
The Winoground evaluation of the Phi model offers a compelling illustration of our framework in a new domain.
The untuned model’s generative performance (16.75%) exhibits an alignment gap,
falling well below the LSC (27.75%) established by a linear probe on its vision and respective text encoder.
Fine-tuning on the relational HOI task with our $\mathcal{L}_{\text{combined}}$ objective closes this deficit,
elevating the end-to-end performance to 29.25%.
However, this improvement is not statistically significant over the LSC.
Consequently, by our framework’s definition, an alignment gap persists.
This indicates that the model has learned to exploit the linearly decodable information within its representations
but has not yet developed a sophisticated (non-linear) reasoning capability that provides a performance advantage beyond this ceiling on a new task.
We hypothesize that achieving such an improvement would require fine-tuning on a dataset that more comprehensively covers the specific compositional and relational concepts tested by the Winoground benchmark.

## Appendix K Class imbalance investigation

Evaluating with unbalanced classes can lead to a misleading estimate of the LSC.
This can be most easily seen when we split the accuracy into two, positive and negative only ground labels, as done in Table [A.8](https://arxiv.org/html/2507.07574v2#A11.T8).

**Table A.8: Class imbalances across datasets and models**
| Model | Dataset | Method | gen. acc.<br>pos. (%) | gen. acc.<br>neg. (%) | gen. acc.<br>avg. (%) | LSC<br>(pos., %) | LSC<br>(neg., %) | LSC<br>(avg., %) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phi | OpenWorld | Direct (interleaved) | $75.2\pm 5.4$ | $42.8\pm 6.1$ | $59.0\pm 4.3$ | $86.8\pm 4.2$ | $81.2\pm 4.8$ | $84.0\pm 3.2$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.8\pm 3.2$ | $91.6\pm 3.4$ | $92.2\pm 2.4$ | $87.2\pm 4.1$ | $81.6\pm 4.8$ | $84.4\pm 3.2$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $94.8\pm 2.8$ | $96.4\pm 2.3$ | $95.6\pm 1.8$ | $87.2\pm 4.1$ | $81.6\pm 4.8$ | $84.4\pm 3.2$ |
|  | HOI | Direct (interleaved) | $97.5\pm 1.5$ | $6.8\pm 2.5$ | $52.1\pm 3.5$ | $75.0\pm 4.2$ | $68.8\pm 4.5$ | $71.9\pm 3.1$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $85.2\pm 3.5$ | $72.0\pm 4.4$ | $78.6\pm 2.8$ | $75.0\pm 4.2$ | $68.8\pm 4.5$ | $71.9\pm 3.1$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $83.2\pm 3.7$ | $75.2\pm 4.2$ | $79.2\pm 2.8$ | $75.2\pm 4.2$ | $68.5\pm 4.6$ | $71.9\pm 3.1$ |
| Pixtral | OpenWorld | Direct (interleaved) | $80.8\pm 4.9$ | $64.0\pm 6.0$ | $72.4\pm 3.9$ | $78.4\pm 5.1$ | $73.6\pm 5.5$ | $76.0\pm 3.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.8\pm 3.2$ | $94.0\pm 2.9$ | $93.4\pm 2.2$ | $79.2\pm 5.0$ | $74.0\pm 5.4$ | $76.6\pm 3.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $95.6\pm 2.5$ | $94.4\pm 2.9$ | $95.0\pm 1.9$ | $76.4\pm 5.3$ | $72.4\pm 5.5$ | $74.4\pm 3.8$ |
|  | HOI | Direct (interleaved) | $67.8\pm 4.6$ | $47.8\pm 4.9$ | $57.8\pm 3.4$ | $64.8\pm 4.7$ | $60.8\pm 4.8$ | $62.7\pm 3.4$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $80.5\pm 3.9$ | $75.5\pm 4.2$ | $78.0\pm 2.9$ | $62.5\pm 4.7$ | $60.8\pm 4.8$ | $61.6\pm 3.4$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $80.5\pm 3.9$ | $78.8\pm 4.0$ | $79.6\pm 2.8$ | $67.5\pm 4.6$ | $58.8\pm 4.8$ | $63.1\pm 3.3$ |
| Gemma3 4B | OpenWorld | Direct (interleaved) | $69.6\pm 5.7$ | $82.4\pm 4.7$ | $76.0\pm 3.7$ | $92.4\pm 3.3$ | $87.2\pm 4.1$ | $89.8\pm 2.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $92.0\pm 3.4$ | $92.8\pm 3.2$ | $92.4\pm 2.3$ | $92.4\pm 3.3$ | $87.2\pm 4.1$ | $89.8\pm 2.7$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $95.6\pm 2.5$ | $95.6\pm 2.5$ | $95.6\pm 1.8$ | $92.4\pm 3.3$ | $87.2\pm 4.1$ | $89.8\pm 2.7$ |
|  | HOI | Direct (interleaved) | $24.5\pm 4.2$ | $88.5\pm 3.1$ | $56.5\pm 3.4$ | $77.5\pm 4.1$ | $70.8\pm 4.5$ | $74.1\pm 3.0$ |
|  |  | LoRA ($\mathcal{L}_{\text{NT}}$) | $86.5\pm 3.3$ | $82.0\pm 3.8$ | $84.2\pm 2.5$ | $77.5\pm 4.1$ | $70.8\pm 4.5$ | $74.1\pm 3.0$ |
|  |  | LoRA ($\mathcal{L}_{\text{combined}}$) | $81.5\pm 3.8$ | $87.0\pm 3.3$ | $84.2\pm 2.5$ | $77.5\pm 4.1$ | $70.8\pm 4.5$ | $74.1\pm 3.0$ |

The data in Table [A.8](https://arxiv.org/html/2507.07574v2#A11.T8) demonstrates how average accuracies, whether generative or similarity-based,
can mask significant performance disparities between classes.
An aggregated LSC is composed of differing per-class separability scores (e.g., for Gemma3 on HOI, 77.5% for positive vs. 70.8% for negative).
This class-dependent variability means the overall LSC may not fully capture the model’s representational quality for all distinct classes.
Therefore, this detailed per-class separability analysis serves as a valuable diagnostic tool.
It effectively uncovers such representational biases that overall average metrics might otherwise conceal.

## Appendix L Example semantic concepts

The Bongard OpenWorld dataset primarily utilizes semantic concepts based on objects, scenes, actions, or attributes:

- -
Elderly person using a cell phone.
- -
Evening sunset on the desert dunes.
- -
People playing water polo in the swimming pool.
- -
People are in a hurry.
- -
A boat tied with a rope in the water.
- -
A player shooting on the hockey field.
- -
A cute dog wearing a cozy sweater.
- -
Solar panels on the house roof.
- -
The shepherd herds flocks of sheep.
- -
A closeup of a dandelion.
- -
Vehicle tires (e.g., steel frame for car tires, motorcycle tires).
- -
Chimney on the house roof.

The HOI dataset utilizes semantic concepts based on human-object interactions:

- -
A person carrying a surfboard.
- -
A person riding a surfboard.
- -
A person holding an apple.
- -
A person swinging a tennis racket.
- -
Multiple people sitting on a bench.
- -
A person brushing with a toothbrush.
- -
A person adjusting or tying a tie.
- -
A person using a mouse.
- -
A person throwing a frisbee.
- -
A person holding and about to eat an apple.
- -
A person peeling or cutting an apple.
- -
A person lying on a bench.

## Appendix M Visualizing the Mechanism of PEFT via Attention Maps

The quantitative results in the main paper demonstrate that PEFT can effectively bridge and surpass the LSC.
To understand *how* these interventions alter the model’s internal computations,
this appendix provides a qualitative, mechanistic explanation by visualizing the models’ attention patterns before and after fine-tuning.

Attention maps reveal the core mechanism of the transformer architecture,
showing which parts of the input sequence (text and image tokens) are attended to when building updated representations.
A higher attention score (brighter color) indicates a stronger influence.
By comparing these maps, we can directly observe how fine-tuning reshapes the information flow within the model to solve the reasoning task,
providing visual evidence that complements the quantitative analysis.

### M.1 Distinct architectural signatures and tuning mechanisms

Our analysis reveals that each model family possesses a unique baseline attention strategy,
which dictates the mechanism of improvement unlocked by fine-tuning.

Gemma3: overcoming representational degradation.
As established in the main paper, Gemma3 models exhibit a high LSC but suffer from severe representational degradation,
where the final-layer embeddings ($acc_{final}$) become poorly separated.
The attention maps in Figure [A.2](https://arxiv.org/html/2507.07574v2#A13.F2) reveal the architectural reason and the solution.
Gemma3 uses an efficient sliding window attention for the most part (narrow diagonal band) (layer 24 behaved differently for some reason).

- -
With $\mathcal{L}_{\text{NT}}$ tuning, the model learns the task but the attention mechanism is not visibly altered;
it refines computations within its existing pathways.
- -
With $\mathcal{L}_{\text{combined}}$ tuning, the contrastive loss induces a change that makes the model’s reasoning process visible.
In the final layers, the attention map develops more structured, bright vertical stripes.
Each well-defined stripe represents a targeted “read” operation, where the model globally accesses the compressed representation of visual features.
This structured, cross-image comparison provides a clear mechanistic explanation for how the model overcomes its baseline shortcoming.
The explicit contrastive pressure forces it to develop and execute this comparison strategy, repairing its degraded final representations
and aligning with the recovery of $acc_{final}$ seen in Table [6](https://arxiv.org/html/2507.07574v2#S7.T6).

Figure: (a) Gemma3 4B after LoRA tuning ($\mathcal{L}_{\text{NT}}$)
Refer to caption: bongard_images/gemma3_4b_attn_map_lora_sim=False.png

Phi and Pixtral: representation refinement and feature enhancement.
Phi and Pixtral employ a standard causal attention mask across all layers (Figures [A.3](https://arxiv.org/html/2507.07574v2#A13.F3) and [A.4](https://arxiv.org/html/2507.07574v2#A13.F4)).
Their path to success is not about fixing a shortcoming but refining existing processes.

Phi possessing a higher-quality LSC from the outset, Phi’s fine-tuning is primarily a process of refinement and noise reduction.
The attention maps visualize this as the patterns *within* each image block becoming more structured and less diffuse.
Instead of a scattered focus across all visual patches, the model learns to consistently attend to the most salient features while ignoring irrelevant ones.
This refined intra-image feature extraction allows for a cleaner signal to be passed to the final layers, enabling the model to better leverage its already-strong representations and surpass its LSC.

Figure: (a) Phi after LoRA tuning ($\mathcal{L}_{\text{NT}}$)
Refer to caption: bongard_images/phi_attn_map_lora_sim=False.png

Pixtral’s success, even at baseline, stems from its inherent ability to perform vision representation refinement.
The attention maps reveal the mechanism: an intensified focus on intra-image processing in later layers, visible as more structured diagonal blocks.
Crucially, this visual signature is nearly identical when fine-tuning with either the standard $\mathcal{L}_{\text{NT}}$ or the $\mathcal{L}_{\text{combined}}$ objective.
This equivalence demonstrates that standard next-token prediction is sufficient to fully engage Pixtral’s innate pathway for extracting and enhancing salient features.

Figure: (a) Pixtral after LoRA tuning ($\mathcal{L}_{\text{NT}}$)
Refer to caption: bongard_images/pixtral_attn_map_lora_sim=False.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How sensitive are your results to pooling strategy (mean pooling vs. attention pooling vs. CLS token)?
2. How stable is the LSC metric across different random seeds or mini-batch samplings?
3. How exactly is the statistical comparison between generative accuracy and LSC performed in Fig. 2—are these paired comparisons over test trials or aggregated accuracies? For models that pass LSC, the advantage of generative accuracy is low.
4. Could this framework be applied to tasks requiring fine-grained perceptual reasoning (e.g., gaze direction or social interaction), rather than Bongard tasks that suit better for linear separation?
soundness: 2
strengths: 1. The paper is well motivated and identifies an important problem: diagnosing perceptual–reasoning misalignment in VLMs.
2. The proposed LSC is a clear, interpretable metric that operationalizes representational quality in a meaningful way.
3. The paper is well written and organized, with 2 dataset on 8 models coupled with various promptings.
4. The paper introduced a contrastive fine-tuning objective that simultaneously improves generative accuracy and final-layer separability.
summary: This paper introduces the LSC to diagnose whether VLMs fail on abstract reasoning tasks due to perception or reasoning. The authors identify an alignment gap where reasoning fails to surpass the model’s own representational limit. They further proposed a contrastive fine-tuning objective that can close this gap on Bongard-style reasoning tasks.

Overall, the paper is thoughtfully motivated but overinterprets linear-probe diagnostics as mechanistic evidence for reasoning non-linearity. The statistical evidence is somewhat weak, and the causal link between representational geometry and reasoning behavior remains speculative. I will consider moving up the score, depending on how the authors clarify statistical robustness and theoretical grounding during discussion.
weaknesses: 1. The paper defines “non-linearity” as “cases where linear probes fail.”  The non-linearity can be an artifact of your measurement of cosine similarity of euclidean averaged embeddings, not a measured representational property. The claim would be stronger with direct evidence of curvature or manifold structure.
2. Important evaluation details are underspecified—for instance, how generative accuracy is computed relative to the probe-based classification accuracy. 
3. Some of the results (e.g., Section 7.2) appear cherry-picked without consistent statistical treatment. A group-level comparison across models or prompt conditions would increase confidence in the claims.
4. The causal interpretation—that contrastive fine-tuning resolves misalignment—remains speculative. The improvements may simply reflect more linearly organized feature geometry rather than deeper mechanistic reasoning.
5. The paper notes catastrophic forgetting as a limitation but does not fully analyze why L_{combined} causes this — an area that could benefit from ablation or regularization experiments.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: Please refer to the weak points.
soundness: 2
strengths: 1)	The paper introduces the Linear Separability Ceiling framework to disentangle perception and reasoning in VLMs.
2)	Through large-scale analysis, the authors reveal a pervasive alignment gap: most VLMs fail to outperform their own LSC, highlighting a fundamental but previously unmeasured bottleneck in vision–language reasoning.
3)	The approach attains or surpasses human-level accuracy on OpenWorld and narrows the gap on HOI reasoning, demonstrating that the limitation in current VLMs stems from misalignment, not innate capacity.
summary: This paper investigates why Visual–Language Models (VLMs) often fail on abstract visual reasoning tasks and whether such failures arise from perception or reasoning deficits. To diagnose this, the authors propose the Linear Separability Ceiling (LSC) framework, which quantifies the performance achievable by a linear probe on a model’s visual embeddings. Analyses across state-of-the-art VLMs reveal a pervasive alignment gap, where models rarely exceed their own LSC. To address this, the study augments next-token prediction with a contrastive objective that enhances the linear structure of visual representations. Fine-tuned models consistently surpass the LSC, achieving human-level accuracy on Bongard-OpenWorld and significant gains on relational reasoning benchmarks, demonstrating that reasoning limitations stem from misalignment rather than intrinsic capacity.
weaknesses: 1)	While the Linear Separability Ceiling is intuitively defined, the paper lacks a rigorous theoretical justification for why linear separability should represent the upper bound of perceptual quality. A more formal link between LSC and model capacity or information-theoretic limits is missing.
2)	The claim that failures arise from “alignment gaps” rather than perception deficits is mostly correlational. The experiments show association but not causal evidence that reasoning misalignment causes underperformance.
3)	The evaluation focuses mainly on Bongard-style reasoning and a single compositional benchmark. Broader validation on diverse abstract reasoning or real-world multimodal tasks would strengthen the generality of conclusions.
4)	The “nonlinear decision logic” mechanism is described conceptually but not visualized or quantitatively analyzed. Without feature attribution or attention-map evidence, the interpretation remains speculative.
5)	This paper is lengthy and conceptually dense. Core ideas like “alignment gap” and “surpassing the ceiling” could be more precisely illustrated. Figures (e.g., Fig. 2) lack clear axis descriptions, and some tables overflow with statistical detail without clear takeaway messages.
6)	The paper’s structure is somewhat diffuse, with diagnostic and intervention sections interleaving and key concepts repeated across sections. This weakens the logical flow from problem to solution and makes the main argument harder to follow.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Can the fine-tuning approach be successfully applied to improve VLM performance on other challenging reasoning domains beyond Bongard problems, such as VQA or complex instruction following?
soundness: 3
strengths: 1. The graphs and tables are clear and easy to understand.
2. Experiments are thorough, covering multiple VLMs, datasets, PEFT methods, objectives, and generalization scenarios.
summary: This paper investigates failures of VLMs on abstract visual reasoning tasks like Bongard problems, questioning whether the bottleneck lies in visual perception or higher-level reasoning. The authors introduce a diagnostic framework centered on the Linear Separability Ceiling (LSC): the maximum performance achievable by a linear classifier on the VLM's initial visual embeddings. Applying this framework, they discover an alignment gap: most state-of-the-art VLMs fail to generatively outperform their own LSC, suggesting their reasoning capabilities are poorly aligned with their visual representations. They propose a fine-tuning method using LoRA with a combined objective, adding a contrastive loss to the standard next-token prediction loss. This method improves the linear separability of final embeddings, successfully allowing models to systematically surpass the LSC and achieve higher performance on abstract reasoning tasks.
weaknesses: 1. While effective, the LSC relies solely on linear separability. It's possible that representations hold complex non-linear structures useful for reasoning that the LSC metric fails to capture.
2. The core observation that VLM generative performance often fails to surpass a linear probe on its visual features is not entirely new. Similar gaps between representation quality and end-to-end performance have been previously studied, showing VLMs can underperform linear probes on classification or generally overlook information in their visual representations.
Some related works like:
[Why are Visually-Grounded Language Models Bad at Image Classification?](https://arxiv.org/pdf/2405.18415.pdf)
[Hidden in plain sight: VLMs overlook their visual representations](https://arxiv.org/pdf/2506.08008.pdf)

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1) The paper mentions LSC could be used as a “live diagnostic” during model training. Could the authors outline how such an online LSC metric might be integrated into training pipelines (e.g., as a stopping criterion or auxiliary signal)?
2) How sensitive are the results to the relative weights w_m and w_c in L_combined? Does a small contrastive component already improve alignment, or is a strong contrastive signal necessary?
3) Have the authors evaluated whether the LSC framework predicts reasoning performance on tasks that are less abstract (e.g., visual question answering or commonsense reasoning)? How consistent is the alignment gap in these settings?
soundness: 2
strengths: 1) The concept of an “alignment gap” between perception and reasoning reframes common VLM failures through a geometric and statistical lens
2) The paper’s decomposition of reasoning into linear vs. non-linear computational mechanisms (perceptual refinement vs. reasoning) provides conceptual clarity that could generalize beyond VLMs
summary: The paper investigates why VLMs fail on abstract visual reasoning tasks such as the Bongard problems. The paper proposes the Linear Separability Ceiling (LSC), a diagnostic measure of how well a simple linear classifier performs on a VLM’s visual embeddings. It serves as a baseline to test whether a model’s reasoning pipeline adds non-linear reasoning beyond what its visual features already provide.
The authors evaluate several vLMS (e.g. Gemma, Pixtral, Phi, Qwen, InternVL) and realize that most models fail to exceed their own LSC, implying that their reasoning components are not effectively leveraging visual representations.
They also identify two strategies by which the VLMs can surpass the LSC
1) Enhanced linear separability: improving internal representations to be more discriminative
2) Non-linear decision logic: leveraging deeper reasoning pathways beyond linear readouts 

The paper also proposes a finetuning method that combines next-token prediction and a contrastive loss to improve representation alignment. They show that this object allows models to surpass LSC and improve in-domain and cross-domain reasoning.
weaknesses: 1) The experimental validation is limited to binary image-to-text retrieval variants of Bongard-style tasks (Bongard OpenWorld and Bongard HOI), where the model must choose between two options given a query image. While these are well-defined tests for abstract reasoning, they represent a highly specific and simplified evaluation setup. It remains unclear whether the proposed framework and fine-tuning method would generalize to broader vision-language reasoning tasks such as open-ended visual question answering, caption generation. The improvements demonstrated may partially stem from optimizing for the same binary discriminative signal introduced by the contrastive objective, rather than from a genuinely general enhancement of multimodal reasoning.
2) Because the proposed contrastive objective directly optimizes for improved separability between positive and negative examples, part of the observed performance gain could be attributed to alignment with the evaluation metric itself (linear separability), rather than improved reasoning. Additional experiments on independent tasks not directly linked to the contrastive loss would help verify generalization.
3) The paper notes that the combined objective (L_combined) can induce catastrophic forgetting and prompt-format overfitting. It is unclear whether the performance of the finetuned model on general vision-language benchmarks (e.g. VQA benchmarks) is preserved after finetuning.
Furthermore, the paper provides little quantitative analysis of how the weighting between the next-token prediction and contrastive loss terms affects this trade-off. A sensitivity study or learning dynamics analysis would strengthen the claims.


Decision: Reject