Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models
Authors: Luca M. Schulze Buschoff
Konstantinos Voudouris
Elif Akata
Matthias Bethge
Joshua B. Tenenbaum
Eric Schulz
Venue: ICML 2025 poster (2025)
Keywords: Cognitive Science
Machine Learning
Fine-tuning
Vision language models
Intuitive phyics
Causal reasoning
Counterfactual reasoning
Primary area: applications->neuroscience_cognitive_science
TL;DR: Fine-tuning can make models better at visual cognition tasks, but it does not lead to robust human-like generalization to other tasks.

Abstract: Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains.

Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models
Luca M. Schulze Buschoff" ! Konstantinos Voudouris" *1 Elif Akata 12
Matthias Bethge² Joshua B. Tenenbaum² Eric Schulz
Abstract
(Battaglia et al., 2012; Piloto et al., 2022), an ability that is present very early in development and does not require extensive learning or experience (Baillargeon et al., 1995; Spelke, 1990; Spelke & Kinzler, 2007). Causal reason- ing describes the ability to infer cause-effect relationships (Waldmann, 2017; Pearl, 2009). There is growing evidence that humans possess intuitive capacity to infer and predict causal relationships (Griffiths & Tenenbaum, 2009), and that this ability emerges early in development (Kuhn, 2012; So- bel & Kirkham, 2006). In the psychology literature, intuitive physics and causal reasoning have been studied most promi- nently in their relation to visual cognition- -nvestigatinng how humans and other animals reason about their physi- cal environment and its causal structure through the visual inputs they receive.
Pre-trained vision language models still fall short of human visual cognition. In an effort to im- prove visual cogmition and align models with hu- man behavior, we introduce visual stimuli and human judgments visual cognition tasks, al- lowing us to systematically evaluate performance across cogmitive domains under a consistent en- vironment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance the respective fine-tuning domain. Furthermore. it can improve model alignment with human be- havior. However, we find that task-specific fine- tuning does not contribute to robust human- like generalization to data with other visual character- istics or to tasks in other cognitive domains.
Vision language models (VLMs), which receive visual and textual linguistic input and produce textual output, have received recent attention for their apparently sophisticated reasoning in visual and linguistic tasks (Liu al., 2025). However, recent work has established that VLMs are still limited in their understanding of the physical world and its causal structure (Jin et al., 2023; Balazadeh et al., 2024), suggesting that they lack human-like intuitive physics and causal reasoning. While VLMs perform reasonably well on intuitive physics problems, such as predicting the stability of block towers, they do not show a good fit with human behavioral data. On tests of causal reasoning, such pre- dicting whether removing block would cause tower to fall, VLMs perform poorly and again do not fit well with human behavior (Schulze Buschoff et al., 2025). Beyond the domains of intuitive physics and causal reasoning. VL have also been shown to have a number of visual deficien- cies, and they often struggle with simple visual tasks that would be trivial for human observer (Rabmanzaaehgerrv et al., 2024; Schulze Buschoff et al., 2025; Balazadeh et al., 2024). VLMs are prone to hallucinations, where the corre- sponding output does not sensibly correspond to the input image (Li et al., 2023; Liu et al., 2024). Ullman (2024) shows that VLMs hallucinate visual illusions where there are none, if the visual stimuli resemble canonical illusions that were likely their training data. Similarly, Zhang et al. (2023) show that while the general alignment human per-
1. Introduction
One of the main goals of machine learning research is to build machines that think and behave like humans. To meet this goal, Lake et al. (2017) proposed that human-like ma- chine learning models must be capable of reasoning about their physical and social environment and its causal struc- ture. These capabilities sometimes summarized in- tuitive theories-the cognitive expectations humans and other animals have about their environment from early on in development that they use to behave adaptively.
In this paper, we focus on two classes of intuitive theory. In- tuitive physics relates to the ability to predict and understand the physical properties and interactions of inanimate objects
Equal contribution 'Institute for Human-Centerd AL Helmholtz Munich, OberschleiBheim, Germany University of Tübingen, Tübingen, Germany 'Department of Brain and Cognitive Sciences, MIT, Massachusetts, USA. Correspondence to: Luca M. Schulze Buschoff <lucaschulzebuscchfffi gmai..com>..
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Limits of Fine-Tuning in Vision Language Models
ception is low, larger models are somewhat susceptible the same visual illusions as humans. Additionally, VLMs are not adversarially robust and are therefore subject to manipu- lation of both textual and visual inputs (Zhao et al., 2024). Campbell et al. (2024) suggest that the failures of VI.Ms on tasks containing multiple objects can be explained by a binding problem, in which VLMs, like humans (Frankland et al., 2021). struggle to attend to, represent, and distinguish between multiple objects at the same time, because they share the same representational resources.
Ming & Li (2024) explore VLMs' ability to generalize to out-of-distribution labels in image classification task, pre- senting evidence that fine-tuning noticeably improves per- formance. However, they do not investigate generalization in more complex, psychologgcallly-nnpiree domains like intuitive physics or causal reasoning. Chen et al. (2021) find that a neurosymbolic moo-trransforerr-bseell model can ro- bustly reason causally about visual scenes the CLEVRER dataset (Johnson et al., 2017: Yi et al., 2020), and generalize to new causal reasoning tasks. Generalization and causal reasoning has also been studied extensively outside of the visual cognition domain, such as mathematics (Zhou et al., 2023; 2024) and compositional reasoning (Dziri et al., 2023; Li et al., 2024). Binz et al. (2024) find that fine-tuning on diverse human behavioral data can confer an advantage on wide range of tasks relevant to human psychology.
In pursuit of improving the performance of language mod- els. fine-tuning is quickly distinguishing itself as the gold standard, enabling researchers to efficiently steer models towards better capabilities (Han et al., 2024) as well as towards more human-aligned outputs (Binz et al., 2024; Hussain al., 2024). In this paper, we explore whether fine- tuning VLMS on single tasks can improve their performance on intuitive physics and causal reasoning tasks in the visual domain, as well as steer them towards more human-aligned outputs.
1.2. This Work
In this work, we fine-tune VLMs on single tasks from two cognitive domains inspired by research in cognitive sci- ence, intuitive physics and causal reasoning (Baillargeon & Hanko-Summerss, 1990; Baillargeon et al., 1992; Battaglia et al., 2012; Lake et al., 2017; Lerer et al., 2016; Piloto et al., 2022; Spelke et al., 1992). In particular, we focus on model intuitions about the factual and counterfactual stability of stacks of coloured, uniformly dense blocks. We design these tasks in ThreeDWorld (Gan et al., 2020, TDW), a virtual environment with realistic physics engine built in Unity (Unity Technologies, 2023). We refer to our dataset of block towers built in TDW as Cubeworld. We then evaluate the fine-tuned models' ability to generalize to four different conditions (see Figure 1):
However, a hallmark of human cognition is not just the abil- ity to reason about the physical environment and its causal structure, but also to robustly generalize from limited ex- perience to solve new tasks (Collins et al., 2022; Geirhos et al., 2018; Griffiths & Tenenbaum, 2009). Therefore, we seek to evaluate whether task-specific fine-tuning not only improves performance on visual cognition tasks sampled from an identical distribution, but also whether it produces models that can generalize to new, but related, tasks in new domains. For example, we ask whether model fine-tuned to accurately judge the stability of short tower blocks can generalize this knowledge to judge the stability of tall tower blocks, of tower blocks with different visual characteris- tics (from a different environment), or to causal reasoning problems about tower blocks. Our results allow us to ap- praise the limits of task task-specific fine-tuning for building performant, human-like machine learning models that can generalize beyond the kinds of data on which they have been trained. Across a range of datasets and models, we do not find evidence that fine-tuning alone can achieve all these objectives.
A held-out test set randomly sampled from the same distribution as the fine-tuning data. Example: A model fine-tuned to judge the stability of towers consisting of 2-4 blocks is then tested on new unseen towers consisting of 2-4 blocks.
2. A test set of new block stacks, on the same task and domain as the fine-tuning data (c.g., tower stability). Example: A model fine-tuned on 2-4 block towers tested on 5-7 block towers.
3. A test set of block stacks from the same task and do- main but with different visual characteristics. Example: A model fine-tuned on 2-4 block towers from the Cube- world environment is tested on real block towers with 2-4 blocks from Lerer et al. (2016).
1.1. Related Work
Closest to our work is Balazadeh et al. (2024). who fine- tune the VLM PaliGemma-3B (Beyer et al., 2024) on a series of intuitive physies and visual reasoning tasks, asking questions about the height, color, and shape of tower blocks in an image, as well as whether the towers stable or certain blocks are likely to move. They find that smaller fine-tuned VLMS can outperform larger non fine-tuned mod- els on the fine-tuning task. However, they do not investigate whether fine-tuned VLMs can generalize to new problems.
4. A test set of block stacks from a new cognitive domain that shares the same visual characteristics. Example: A model fine-tuned to make stability judgments (intuitive physics) is tested on its ability to make counterfactual stability judgments (causal reasoning).
2

---

Limits of Fine-Tuning in Vision Language Models
A
Intuitive physics
Causal reasoning
Cubeworld
Lerer at al. (2016)
Cubeworid
Test (same distribution)
New towers (opposite spiit)
Natural images
New domain
Fine-tune
Hunnaa all Appmmnnt
Evaluate
Intuitive
Causal
physics
reasoning
Human judgments
Figure I. Methodology overview. A: We study causal reasoning and intuitive physics using our Cubeworld fine-tuning and evaluation datasets and the Lerer et al. (2016) block tower evaluation dataset. B: Models are fine-tuned on the ground truth human judgments for domain of the Cubeworld dataset. C: We test whether fine-tuning improves model performance in four socnarios: new towers of the same heights training; new lowers different heights compared to training; naturalistic images from Lerer et al. (2016); and block towers from the other domain. We also test the alignment of these models to human responses.
In each form in each context, but also how well their performance aligns with human data on identical tasks. We conduct coun- terbalanced evaluations, testing the interactions between tower sizes, visual characteristics, and cognitive domain (in- tuitive physics vs. causal reasoning). Finally, we fine-tune models on human judgments to if this leads better human alignment.
not only measure how well the models per-
towers such these have been used extensively to inves- tigate intuitive physics in humans and machines (Battaglia et al., 2013; Lerer et al., 2016).
The towers consist of to 7 blocks and their rotation, size, color, and offset are sampled randomly. Offset distributions become more constrained as the number of blocks increases, that randomly sampling offsets leads to roughly 50/50 split between stable and unstable configurations for all tower sizes. This is to ensure that the distributions of stable and unstable towers have comparable difficulties: both contain easy canonical configurations as well as configurations that are harder to judge. The models are presented with an image of block tower and they must judge it stable or wnstable.
2. Methods
2.1. Fine-Tuning & Evaluation Data
We generated four new data sets, two data sets for intuitive physics, and two for causal reasoning. For each cognitive domain, we used one set for fine-tuning and one for evaluation. All four data sets, which we collectively call Cubeworld, consist of different configurations of colored blocks.
Causal reasoning For causal reasoning, generated colored block pyramids in a minimal gray room, inspired by the stimuli in Zhou et al. (2022) (see section A.2 in the Appendix). The pyramids are made up of to rows with bottom row consisting as many blocks as there are rows in total, and each consecutive row featuring one less block than the row below (resulting to 15 blocks in total). The color of each block is sampled randomly and the offset and sizes are sampled within ranges that still allow for a stable pyramid.
We generated similar stimuli for both domains to ensure that models can, in theory, transfer knowledge between them. This allows us to test generalization within domains (such as fine-tuning models on physical stability judgments of small towers and testing on bigger towers) and between domains (such fine-tuning models on physical stability judgments and testing them on counterfactual stability judgments).
range of
Each pyramid features a red block. The models are asked if any other blocks would fall if the red block were not there, similar the the protocol of Zhou al. (2022). This question requires the models perform the counterfactual
Intuitive physics For intuitive physics, we generated block towers that consist of stacks of single colored blocks a minimal gray room (see section A. in the Appendix). Block
3

---

Limits of Fine-Tuning in Vision Language Models
simulation of computing the stability of the tower without the red block. We randomly sample the position of the red block so that it never on the top of the pyramid and so that it has an equal chance of being in every row of a pyramid.
2.4. Fine-Tuning Procedure
We used Parameter Efficient Fine-Tuning (PEFT; Han et al., 2024), focusing on training low-rank adapters for quan- tized models (QLoRA; Dettmers et al., 2024; Hu et al., 2021). PEFT is quickly becoming the dominant fine-tuning paradigm, blending high performance with computational and memory efficiency. PEFT selectively adjusts only a small number of model parameters during training, which not only reduces computational overhead but also minimizes overfitting and the prospect of existing knowledge being washed out by subsequent training (catastrophic forgetting; French, 1999). QLoRA is an approach to PEFT where the model is first quantized, reducing its memory footprint by reducing the precision of the models weights and activations, and then injecting small adapter layers into the transformer blocks of the VLM, both for the vision encoder and the au- toregressive text decoder. For the weight matrix, W, of any layer, an accompanying adapter layer, Wa, is injected. W. is the product of two low-rank matrices Li and L2 where L1 E Rdxr and L2 E Rr*k where is much smaller than d and kt the dimensionality of the input and output and respec- tively. Given some input æ; it is transformed by both W and Wa independently and then summed (subject to scaling a), to produce the output. In QLoRA, only the values of Li and L2 are altered; W remains fixed. The weights of L1 and L2 are altered by backpropagation under the supervision of the next token in a document, using a cross-entropy loss. Given the relatively small r, models can be trained much more quickly than through training the full-rank W matrix. We chose r == 16 for all experiments and a scaling of where a = 16, thus balancing the effect of W and W. on the outputs. We fine-tuned layers in the ViT vision encoder, and attention and MLP layers in the language decoder, this has been shown to be effective in prior work on fine-tuning for intuitive physics understanding (Balazadeh et al., 2024). We used the ADAM optimizer and an initial learning rate of 0.0002. We fine-tuned all models for 10 epochs on 10,000 text-image pairs on 80GB NVIDIA A100 GPUs. To ensure the robustness of our results, we repeated every experiment with three different seeds, leading to different samples of training data and different Shapter-weight initializations, and report all results as averages across the three repeats.
To study whether models could gen- eralize to data with other visual characteristics, we used a sample of 100 intuitive physics tower block images from Lerer al. (2016) (see section A.3 in the Appendix). This dataset consists of pictures of real block towers with 2, 3, and 4 blocks that are either stable or unstable. The images look different to Cubeworld, but the underlying cognitive task is the same as in the intuitive physics data set. Human data for this task was taken from Schulze Buschoff et al. (2025), who collected 107 participants on 100 randomly selected images from the experiment by Lerer al. (2016).
Naturalistic Data
2.2. Models
We fine-tune the 7B parameter version of the Qwen2-VI. model (Wang et al., 2024) and the 11B and 90B versions of Llama 3.2 (Grattafiori et al., 2024) using the unsloth library (Han et al., 2023). We used pre-trained models quantized to 4-bit precision.
We evaluate the models by sampling the log probabilities of the "Yes" and "No" tokens conditional on the input and normalizing them by exponentiating and then using the softmax function. This gives a measure of the relative probability of the model answering "Yes" or "No" to each question. We then evaluate whether the model is correct by examining which token is assigned the higher probability and comparing this to the ground truth (see section C in the Appendix for information on the packages used for analysis). We also elicited free text responses from the model and found that these aligned with the normalized token probabilities anyway.
2.3. Prompts
For intuitive physies, we prompt the models with the fol- lowing pre-prompt: "You are now viewing a tower of blocks. Will the tower fall? Answer Yes if you think this tower is unstable and will fall. Answer No if you think this tower is stable and will not fall."
For the causal reasoning pyramids, we prompt the models with this pre-promphl "You are now viewing a pyramid of blocks. If the red block was not there, would any other blocks fall? Answer Yes if you think that other blocks would fall if the red block was not there, Answer No if you think that no other blocks would fall if the red block was not there."
2.5. Human experiments
We performed three separate human experiments to obtain fine-tuning and evaluation data. All participants agreed to take part in the study and were informed about the general purpose of the experiment. Experiments were conducted on Prolific in accordance with the relevant guidelines and regu- lations approved by the ethics committee of the University of Tübingen. For information on the samples, durations, and payout of the experiments, see Section B in the Appendix.
'softmax(p)= > 2s where p the vector of probabilities
of length K and is the exponential function.
4

---

Limits of Fine-Tuning in Vision Language Models
Llama-3.2-11B
Intuitive physics
Causal reasoning
Lerer et (2016)
4
10
15
3
Intuitive physics
2-4
0.83
0.81
0.78
0.71
0.67
0.6
0.48
0.52
0.52
0.52
0.52
0.46
0.64
0.9
5-7
0.61
0.69
0.71
0.69
0.67
0.63
0.5
0.51
0.51
0.52
0.44
0.48
0.7
3 & 6
0.45
0.49
0.49
0.51
0.51
0.5
0.88
0.9
0.89
0.84
0.38
0.7
0.57
0.8
Causal reasoning
10 & 15
0.36
0.43
0.47
0.48
0.49
0.49
0.75
0.89
0.92
0.92
0.43
0.52
0.49
0.7
2-4, 3 &
0.79
0.78
0.74
0.69
0.65
0.59
0.86
0.89
0.88
0.83
0.48
0.56
0.68
Accuracy
2-4, 10 15
0.79
0.78
0.74
0.68
0.64
0.59
0.77
0.88
0.91
0.9
0.52
0.63
0.71
0.66
Joint
5-7, 3.& & 6
0.61
0.66
0.67
0.67
0.65
0.6
0.86
0.89
0.87
0.82
0.49
0.52
0.75
0.5
5-7, 10
0.54
0.63
0.68
0.67
0.66
0.61
0.76
0.88
0.91
0.91
0.52
0.48
0.7
0.4
Base Model
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.67
0.57
0.3
GPT-4o
0.66
0.57
0.56
0.54
0.55
0.55
0.51
0.55
0.51
0.52
0.71
0.52
0.61
0.3
Human Average
0.62
0.71
0.68
0.65
0.58
0.56
0.68
0.71
0.75
0.76
0.69
0.69
0.59
Figure 2. Heat map showing accuracies for the 11B model on all combinations of ground truth fine-tuned models and evaluations. Each contains model fine-tuned the ground truth for specific data split. Each column contains the results for specific block number in each evaluation data set. Models fine-tuned on single cogmitive domain do not generalize to the other cognitive domain. Models fine-tuned both cognitive domains perform well both domains well as the naturalistic Lerer al. (2016) dataset.
Intuitive physics For the majority of results reported here, the model is fine-tuned on the ground- truth of the generated block configuration. However, we also fine-tune the models on human responses. For this purpose, we collected individ- ual human responses for each image in the 2-4 block tower intuitive physics fine-tuning data set. We collected 100 re- sponses on average from 100 human participants to cover the 10,000 images in the fine-tuning data set. In this exper- iment, all participants received different images and were given the same pre-prompt as the models in the intuitive physics experiment (see Section 2.3).
pyramid sizes x stable unstable x 10 images per condi- tion). Participants received the images in a random order and were given the same pre-prompt as the models in the causal reasoning experiment (see Section 2.3).
3. Results
First we fine-tune on the ground truth. We evaluate whether this leads to improved performance (3.1), different types of generalization (3.2-3.4) and alignment to human judgments (3.5). We then fine-tune model on human responses on the same task (3.6), which leads to better human alignment. All results reported here are averaged over three seeds. The random seed changes the random initialisation of the fine- tuning weights and subset of the fine-tuning data.
We also collected the responses of 100 separate participants on the same subset of 120 images from all conditions in the evaluation set for the intuitive physics tower task (6 tower sizes x stable unstable x 10 images per condition). This allows us to compute similarities between human and model judgments. Participants received the images in random order and were given the same prompt in the experiment above.
3.1. Fine-tuning performance improvement
Fine-tuning substantially improves the performance of most models compared to the zero-shot case. Fully fine-tuned IIB models achieve accuracies between 0.6 and 0.92 on single block sizes from the split they were fine-tuned on (see Fig. 2), compared the rero-shot base models, which
Causal reasoning For the causal reasoning pyramids, we also collected a human evaluation data set of 100 separate participants on the same 80 images in the evaluation set (4
5

---

Limits of Fine-Tuning in Vision Language Models
Fine-tuned model
78. 2-4
7B, 5-7
11B, 2-
11B, 5-7
90B, 2-4
90B, 5-7
Accuracy
Fine-tuned model
7B, 3 &
7B, 10 15
11B, 3&6
11B, 10 15
908, 3
908, 10 15
0.5
2
3
15
Number blocks
Number of blocks
Figure 3. A: Models fine-tuned on two splits of intuitive physics towers (2 or 5-7 blocks) and evaluated on all tower sizes. Performance for models fine-tuned 2-4 black towers decreases with tower size. Models fine-tuned on 5-7 block towers show similar performance over all tower sizes. B: Models fine-tuned two splits of causal reasoning pyramids (3 & 6 or 10 & 15 blocks) and evaluated on all pyramids sizes. All models perform better on pyramid sizes they have been trained on.
perform at around chance for all tower and pyramid sizes (see Figs. 11B and 12B in Appendix E).
formance decreases as the number of blocks increases, it is still above that of the base model even for bigger towers.
For the intuitive physics fine-tuned models, we find that the 7B, 1IB and 90B models fine-tuned on 2-4 block towers achieve accuracies between 0.78 and 0.83 on towers from their fine-tuning distribution (see Fig. 3A). This picture is not clear for the models fine-tuned on 5-7 block towers, with all models showing more or less similar performance improvements over all tower sizes.
In contrast, the models fine-tuned on 5-7 towers do not show a strong difference in performance between towers that were and out-of their fine-tuning distribution. Crucially, they only performs well on the 5-7 block towers the 2-4 fine-tuned models, even though these latter models have to generalize from their fine-tuning distribution to bigger towers.
The models might have difficulty learning from the 5-7 block towers because judging the stability of a tower be- comes harder as it increases size. This is mirrored in human performance the evaluation data set, with human average accuracies of 0.62, 0.71, and 0.68 for towers of size 2, 3, and 4, and average accuracies of 0.65, 0.58, and 0.56 for towers of size 5, 6, and 7. The mean human accuracy over all towers was 0.63.
The performance of the causal reasoning fine-tuned models is more constant over different pyramid sizes (see Fig. 3B), Still, models fine-tuned on 10 & 15 block pyramids perform slightly worse block pyramids (see Fig. 11A).
3.3. Generalizing different visual quality
Models fine-tuned on artificial block towers do not gener- alize well to realistic block towers. To ascertain to what extent fine-tuned models can generalize to the same task with different visual characteristics, we tested them on real images depicting block towers from Lerer et al. (2016). We find that models fine-tuned on a single domain do not gener- alize well to all tower sizes in the Lerer al. (2016) dataset (see Fig. 2). For example the 1 1BB model fine-tuned on block towers in Cubeworld, which identical the Lerer data in task and the number of blocks, only performs above chance on towers with 4 blocks from the Lerer dataset (see also Fig. 13 in Appendix F).
find that the models fine-tuned on causal reasoning im- prove in performance on all pyramid sizes regardless of their fine-tuning split (see Fig. 3B). This is likely because the causal reasoning data set is easier to learn. Human par- ticipants had an average accuracy of 0.72, with accuracies of 0.68 and 72 for and block pyramids, and accuracies of 0.75 and 0.76 for 10 and 15 block pyramids.
3.2. Generalizing to taller and shorter towers
Models are able to generalize to taller and shorter towers to some degree. For models fine-tuned on 2-4 block intu- itive physies towers, we see that they are able to somewhat generalize to bigger towers (see Fig. 3A). While their per-
6

---

Limits of Fine-Tuning in Vision Language Models
"If the red block was not there, would any other blocks fali?"
B
"The red block has been removed. Will any of the blocks fall?"
Fine-tuned on causality (A), tested on unseen causality stimuli
D
Fine-tured on causality, tested on matching physics stimuli (B)
1.0
0.9
Fine-tuned model
Accursoy
0.8
7B, &
11B, 3
0.7
90B, &
0.6
0.5-
0.4
Number of blocks
Number of blocks
Figure 4. A: Models are fine-tuned on the counterfactual reasoning task with pyramids of 3 & 6 blocks. B: Models are given the corresponding images their training data with the red block removed, and must judge whether is stable. C: Models can generalize unseen pyramids on the causal reasoning task. D: Models cannot generalize to judging the factual stability of the pyramids they have been trained on, only now without the red block.
We also fine-tune joint models on combined halves of 5,000 data points from each domain. We again find that these models do not perform well on Lerer towers (see Joint rows in Fig. 2). Indeed, there appears to be trade-off where models that perform well on 2-4 block towers in Cubeworld perform poorly 2 block towers from the Lerer dataset.
only slight performance decrements in either domain. This confirms that the models have the capacity of solving both tasks at the same time. Still, models fine-tuned on a sin- gle cognitive domain are unable to generalize to the other domain.
To establish whether these failures to generalize to other tasks are due to small differences between tasks, or if the models struggle with learning intuitive theories through task-specific fine-tuning, we added another dataset where differences between the tasks are kept as minimal as pos- sible. We generate paired images of pyramids, in which the causal reasoning image contains red block which is removed to generate the intuitive physics image (sce Fig, 4),
3.4. Generalizing to a new task
We find that model fine-tuned on single cognitive do- main performs well on the other cognitive domain (see Fig. 2 and Figs. 14-16 in Appendix G). Models were fine-tuned on intuitive physics towers or causal reasoning pyramids from Cubeworld. To test how well models generalize to another task in another cognitive domain, we evaluate them on the task they were not fine-tuned on.
In principle, being able to reason about the counterfactual stability of a pyramid ought to predispose models to reason about the factual stability of pyramids. Thus, we expected transfer from causal reasoning to intuitive physics, espe- cially since we test models using the corresponding images from the pairs they were fine-tuned on. Furthermore, ex- plicitly tell the models that the red block has been removed. Nevertheless, we do not find evidence of this transfer, sug- gesting that task-specific fine-tuning does not lead to models learning intuitive theories. Instead, they appear to be learn- ing task-specific superficial shortcuts that do not generalize (Geirhos et al., 2020a; llyas al., 2019).
Reasoning about tower stability is prerequisite for coun- terfactual judgments on tower stability. This is especially obvious for the 3 block tower pyramids, where computing the counterfactual requires tower stability judgment on a two block tower. Therefore, we would expect an improve- ment in causal reasoning to carry with improvement on intuitive physics as well.
However, models fine-tuned mixture of data from both tasks can achieve good performance in both domains, with
7

---

Limits of Fine-Tuning in Vision Language Models
A
Intuitive physics: Cubeworld
Intuitive physics: Lerer el. (2016)
C
Causal reasoning: Cubeworld
0.8
Human judgement tuned
Humanss
Joint ground truth luned
Physics ground truth luned
0.6
0.4
0.2
0.0
to ground Iruth
to humans
ground truth
humans
to ground Irulh
to humans
Figure 5. Error consistency to ground truth and human raters on three evaluation datasets (A-C) for the 1IB model separately fine-tuned on three different datascts: (1) human judgments on intuitive physies, (2) the ground truth on intuitive physics, (3) the ground truth on causal reasoning and intuitive physics. Human error consistency is provided as a comparison. A: Results for the Cubeworld intuitive physics evaluation dataset. Results for naturalistic dataset (Lerer et al., 2016). Results for the Cubeworld causal reasoning evaluation dataset.
3.5. Alignment with human judgments
We find that fine-tuning on human responses leads to a con- siderable performance improvement on judging the ground truth stability of intuitive physics block towers compared to the base model. Even though the model is fine-tuned on human judgments on towers of size 2-4, it learns predict the ground truth stability of bigger towers well. We see the same pattern emerge as before, where model accuracy decreases as the number of blocks increases, albeit with overall slightly lower accuracies compared to the model fine-tuned on the ground truth.
We see that fine-tuning on the ground truth leads to some alignment with human judgments on the fine-tuning task. However, this does not transfer well to human judgments on the same task with other visual characteristics, and not at all to human judgments another cognitive domain.
To analyze the alignment of model judgments with human judgments, we use boostrapped Cohen's к arithmetic means (Geirhos et al., 2020b), a single behavioral score that mea- sures the agreement between two observers from their re- sponses (see Appendix D).
Furthermore, fine-tuning the model human judgments in- creases the mean k with human judgments the same data set to 0.37 (see Fig. 5A). Additionally, it leads to higher mean K of 0.37 with human judgments on the same task with different visual characteristics and a better transfer on the ground truth performance (see Fig. 5B). however does not transfer to causal reasoning domain (see Fig. 5C). Here, the mean to humans is -0.02.
The 11B model fine-tuned on the ground truth intuitive physics 2-4 block towers have mean of 0.23 with hu- mans on the same task, but only 0.15 on the Lerer task, and 0.09 on the causal reasoning pyramids (see Fig. 5). In con- trast, the IIB model fine-tuned on the ground truth causal reasoning 3 & block pyramids has mean & of 0.4 with humans on the same task, but only -0.02 on the Lerer task, and -0.08 the intuitive physics block towers.
4. Discussion
3.6. Fine-tuning on human data
Previous work has shown that pre-trained VLMs strug- gle with several aspects visual cognition, particularly in causal reasoning and intuitive physics (Schulze Buschoff et al., 2025). We find that fine-tuning on intuitive physics and causal reasoning tasks can improve the performance of VLMs in these cognitive domains, and that it improves alignment with human judgments. However, there is also evidence that fine-tuned VLMs' physical and causal reason- ing is brittle. On the naturalistic intuitive physics data from
We find that fine-tuning the model on human judgments makes model behavior more similar to human behavior. We collected single human responses for each of the intuitive physics 2-4 block towers in the fine-tuning data set (see section 2.5), allowing us to fine-tune models on the human responses instead of the ground truth. The correlation be- tween the human fine-tuning data and the ground truth was 0.27, with an overlap in labels of 63.5%.
8

---

Limits of Fine-Tuning in Vision Language Models
Lerer al. (2016), models fine-tuned on Cubeworld data in the same domain and task perform below chance level for some tower sizes.
improve outcomes. While we do not find evidence of over- fitting per se, it is possible that the models have overfitted in a more general, task-level sense. The models may have been sensitive to non-robust predictive features of the fine- tuning data in a particular domain that led to good perfor- mance there but not on new domains or with naturalistic data (Ilyas et al., 2019; Geirhos et al., 2020a). Parametrizing the QLoRA procedure with different and a may improve generalization performance by modulating knowledge distil- lation and the relative effects of model weights and adapter weights. Similarly, introducing greater variance into the fine-tuning datasets, fine-tuning on broader distributions of tasks, and fine-tuning on larger volumes of data might im- prove model performance. Finally, the visual and cognitive demands of these tasks are entangled. Models require visual abilities to detect and distinguish blocks, appraise distances, and judge dimenssinai volumes from two-dimensional images. They also need an understanding of gravity, mass, inertia, and friction. Future work should explore whether providing information about these properties in symbolic or schematic form can lead to improved performance and better generalization in these tasks.
Similarly. neither models fine-tuned on intuitive physics nor models fine-tuned on causal reasoning successfully general- ize to the other cognitive domain. This result is particularly notable for the models fine-tuned on causal reasoning, as the ability to make physical stability judgments is a necessary precursor capability for judging the counterfactual stability of tower. Models' inability to generalize to another cogni- tive domain is not due to them being limited in parameters or potential ability- - models fine-tuned on a mixture of intuitive physics and causal reasoning data performed well in both domains. It is important to note that we primarily showcase the limits of models fine-tuned on specific task. While we cannot evaluate how the joint models would gen- eralise to a third cognitive task in Cubeworld, it is possible models on broader distributions of tasks could lead to more robust improvements. Indeed, models fine-tuned on data from both tasks also somewhat gener- alize to the realistic block towers from Lerer et al. (2016), suggesting that data diversity is beneficial for generalization performance. These models also show higher agreement with human judgments both the artificial and realistic datasets.
that
Our findings underscore the limits of task-specific parameter-efficient fine-tuning in achieving robust gener- alization in vision- laionnlanguaae models. PEFT noticeably im- proves performance on tasks closely similar to the fine- tuning data, enabling generalization not just to new data sampled from the fine-tuning distribution but also to, for example, taller and shorter towers and pyramids- an out- of-distribution problem. Moreover, PEFT can align models more closely with human behavior in these contexts. How- ever, task-specific fine-tuning does not lead to the broad, flexible reasoning abilities that characterize human cogni- tion. Models fine-tuned on one cognitive task (e.g.. intuitive physics) fail to generalize to another (e.g., causal reason- ing), despite clear conceptual overlap, and models struggle to reliably extrapolate their knowledge to real-world images with different visual properties.
One account for these results is that fine-tuning does not reduce the effect of the so-called binding problem (Camp- bell al., 2024; Frankland et al., 2021). In human visual cognition, participants placed under significant cognitive load by having to process multiple multi-feature objects very quickly make more mistakes than usual. In our tasks, models had to process multiple blocks simultaneously, judging their colours and relative positions. While fine-tuning improves performance on specific tasks, perhaps facilitating a better division of labour between specific feature detectors, these strategies are brittle and appear to fail on novel domains. An alternative account is that supervised fine-tuning leads to data memorization, whereas post-training method would better facilitate generalization (Chu et al., 2025). We leave exploring these hypotheses to future work.
Together, these results suggest that current approaches to fine-tuning are limited in the ways that they can improve models, and remain insufficient for developing models that reason about the physical and causal structure of the world in way that mirrors human cognition. Achieving this level of generalization may require different training and fine- fine-tuning paradigms that go beyond parameter-efffiiient adaptation.
We present a first investigation on the limitations of fine- tuning for visual cognition. There are several avenues for future research improve our understanding of fine-tuning and how well fine-tuned models can generalize:
First, it is possible that robust generalization from the fine- tuning domain to another can only emerge with even larger models. We studied the effects of fine-tuning on mod- els up to 90 billion parameters, which are relatively small compared some closed-source alternatives. Future work should therefore cxplore fine-tuning to improve even larger models. Second, alternative fine-tuning procedures may
Acknowledgements
We thank Marcel Binz, Can Demircan, and Julian Coda-Forno for helpful discussions and feedback on the manuscript.
9

---

Limits of Fine-Tuning in Vision Language Models
Impact Statement
Chen, Z., Mao, J., Wu, J., Wong, K.-Y. K., Tenenbaum, B., and Gan, C. Grounding physical concepts of objects and events through dynamic visual reasoning. arXiv preprint arXiv:2103. 16564, 2021.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chu, T., Zhai, Y., Yang, J., Tong, S., Xie, S., Schuurmans, D., Le. Q. V., Levine, S., and Ma, Y. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501. 17161, 2025.
References
Baillargeon, R. and Laino-Summers, S. Is the top object ad- equately supported by the bottom object? young infants' understanding of support relations. Cognitive Develop- ment, 5(1):29-53, 1990.
Collins, K. M., Wong, C., Feng, J., Wei, M., and Tenenbaum, J. B. Structured, flexible, and robust: benchmarking and improving large language models towards more human- like behavior in -fitt--distiituttinn reasoning tasks, arXiv preprint 2rXiv:2205.05718 2022.
Baillargeon, R., Needham, A., and Vos, J. The develop- ment of young infants' intuitions about support. Early development and parenting. 1(2):69 78,17 1992.
Deumees, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized Ilms. Advances in Neural Information Processing Systems, 36, 2024.
Baillargeon, R., Kotovsky, L., and Needham, A. The ac- quisition of physical knowledge in infancy. Clarendon Press/Oxford University Press, 1995.
Dziri, N., Lu, X.,, Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36: 70293-70332, 2023.
Balazadeh, V., Ataei, M.. Cheong, H., Khasahmadi, A. H., and Krishnan, R. G. Synthetic vision: Training vision- language models to understand physics. arXiv preprint arXiv:2412.086// 02.66999 2024.
Frankland, S. M., Webb, T., Lewis, R., and Cohen, J. D. No coincidence, george: Processing limits in cognitive function reflect the curse of generalization. 2021.
Battaglia, P., Ullman, T., Tenenbaum, J., Sanborn, A., For- bus, K., Gerstenberg. T., and Lagnado, D. Computational models of intuitive physics. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2012.
French, R. M. Catastrophic forgetting in connectionist net- works. Trends in cognitive sciences, 3(4): 128-155, 1999.
Battaglia, W., Hamrick, B., and Tenenbaum, B. Sim- ulation as engine of physical scene understanding. Proceedings of the National Academy of Sciences, 2013.
Gan, C., Schwartz, J., Alter, S., Mrowca, D., Schrimpf, M., Traer, J., De Freitas, J., Kubilius, J., Bhandwaldar, A., Haber, N., et al. Threedworld: A platform for inter- active mudti physical simulation. arXiv preprint arXiv: 2007.0400,. 2020.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I., Tschan- nen, M., Bugliarello, E., Unterthiner, T., Keysers, D., Koppula, S., Liu, F., Gryener, A., Gritsenko, A., Houlsby, N., Kumar, M.. Rong, K., Eisenschlos, J., Kabra, R., Bauer, M., Bošnjak, M., Chen, X., Minderer, M., Voigt- laender, P., Bica, I., Balazevic, I., Puigcerver, J., Papalam- pidi, P., Henaff, O., Xiong, X., Soricut, R., Harmsen, J., and Zhai, X. PaliGemma: A versatile 3B VLM for transfer. arXiv preprint arXiv:2407.07776, 2024.
Geirhos, R., Temme, C. R., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
Geirhos, R., Jacobsen. J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. Shorteut learn- ing in deep neural networks. Nature Machine Intelligence, 2(1 (1666666677777 2020a.
Binz, M.. Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éitettō N., et al. Centaur: foundation model of human cognition. arXiv preprint arXiv:2410.200222 2024.
Geirhos, R., Meding, K., and Wichmann, F, A. Beyond accuracy: quantifying trial-by-triaa behaviour of enns and humans by measuring emoor consistency. Advances in Neural Information Processing Systems, 33:13890- 13902, 2020b.
Campbell, D., Rane, S., Giallanza, T. De Sabbata, C. N., Ghods, K., Joshi, A., Ku, A., Frankland, S., Griffiths, T., Cohen, J. D., et al. Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37: 11 113436-113 113460, 2024.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
10

---

Limits of Fine-Tuning in Vision Language Models
Griffiths, T. L.. and Tenenbaum, J. B. Theory-based causal induction. Psychological review, 11 116(4))661, 2009. 2009.
Li, C., Jing, C., Li, Z., Zhai, M., Wu, Y., and Jia, Y. In- context compositional generalization for large vision- language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17954-17966, 2024.
Han, D., Han, M., and Unsloth team. Unsloth. Unsloth, 2023. URL http::/ cwm/ooo///////o comm/unslothail unsloth.
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao. W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.1053 2023.
Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q- Parameter-efffcient fine-tuning for large models: A com- prehensive survey. arXiv preprint .rXiv22433144008 2024.
Liu, H., Xue, W., Chen, Y., Chen, D., Zhao, X., Wang, K., Hou, L., Li, R., and Peng, W. A survey on halluci- nation in large vision- lasgual models. arXiv preprint 2ww22402.005.. 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 100.0965, 2021.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2025.
Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puting in Science & Engineering, 9(3):90-95, 2007. doi: N0.1109/MCSE2000
Hussain, Z., Binz, M., Mata, R., and Wulff, D. U. A tutorial on open-source large language models for behavioral science. Behavior Research Methods, 56(8):8214-8277, 2024.
Ming. Y. and Li, Y. How does fine-tuning impact out-of- distribution detection for vision- languaee models? In- ternational Journal of Computer Vision, 132(2):596-609, 2024.
Hyas, A., Santurkar, S., Tsipras, D.,, Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
pandas Development Team. pandas-dev/pandas: Pandas. pandas, 2020.
Pearl, J. Causality, Cambridge university press, 2009.
Piloto, L. S., Weinstein, A., Battaglia, P., and Botvinick, M. Intuitive physies learning in deep-learning model inspired by developmental psychology. Nature human behaviour, 6(9): 2022.
Jin, 7., Chen, Y., Leeb, F., Gresele, I.., Kamal, O., Zhiheng, L., Blin, K., Adauto, F. G., Kleiman- Weiner, M., Sachan, M., et al. Cladder: Assessing causal reasoning in lan- guage models. In Shirty-eeentth conference on neural information processing systems, 2023.
Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vi- sion, pp. 18-34, 2024.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2911111 2017.
Schulze Buschoff, L. M., Akata, E., Bethge, M., and Schulz, E. Visual cognition in multimodal large language models. Nature Machine Intelligence, pp. 1-11, 2025.
Kuhn, D. The development of causal reasoning. Wiley Interdisciplinary Reviews: Cognitive Science, 3(3):327- 335, 2012.
Seabold, S. and Perktold, J. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
Sobel, D. M. and Kirkham, N. Blickets and babies: the development of causal reasoning in toddlers and infants. Developmental psychology, 42(6): 103, 2006.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Spelke, E. S. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
Lerer, A., Gross, S., and Fergus, R. Learning physical intuition of block towers by example. In International conference on machine learning, pp. PMLR, 2016.
Spelke, E. S. and Kinzler, K. D. Core knowledge. Develop- mental science, 10(1):89-96, 2007.

---

Limits of Fine-Tuning in Vision Language Models
Spelke, E. S., Breinlinger, K., Macomber, J., and Jacobson, K. Origins of knowledge. Psychological review, 99(4): 605, 1992.
Ullman, T. The illusion-illlsione Vision language models see illusions where there are none. OSF, 2024.
Unity Technologies. Unity. Unity, 2023. URL unity. com.
Waldmann, M. The Oxford handbook of causal reasoning. Oxford University Press, 2017.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprini vrXiv:2409.12199 2024.
Waskom, M. L. seaborn: statistical data visualization. Jour- nal Open Source Software, 6(60):3021, 2021. doi: 10.21 10.21105/fos..3002.. URL https://doi.com
21 21105/joss, 03021.
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, Clevrer: collision events for video representation and reasoning, arxiv. arXiv preprint arXiv:1910.014442,P 2020.
Zhang, Y., Pan, J., Zhou, Y., Pan, R., and Chai, Grounding visual illusions in language: Do vision-language models perceive illusions like humans? In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5718-5777 5728,, 2023.
Zhao, Y. Pang, Du, C., Yang, X., Li, C., Cheung, N.- M. M., and Lin, M. On evaluating adversarial robustness of large vasion-language models. Advances in Neural Information Processing Systems, 36, 2024.
Zhou, H., Bradley, A., Littwin, E.. Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, What algo- rithms can transformers learn? study in length general- ization. arXiv preprini arXiv:23 10. 16028, 2023.
Zhou, L., Smith, K., Tenenbaum, J., and Gerstenberg, T. Mental jenga: A counterfactual simulation model of causal judgments about physical support. PsyArXiv, 2022.
Zhou, R., Xu, M., Chen, Liu, J., Li, Y., Lin, X., Chen, Z., and He, Math for ai: On the generalization of leamning mathematical problem solving. In The 4th Workshop Mathematical Reasoning and Al at NeurIPS 24, 2024.
12

---

Limits of Fine- Fine-Tuning in Vision Language Models
A. Data examples
A.I. Intuitive physics
Figure 6. Stable examples from the Cubeworld intuitive physies block tower evaluation set.
Figure 7. Unstable examples from the Cubeworld intuitive physics block tower evaluation set.
13

---

Limits of Fine-Tuning in Vision Language Models
A.2. Causal reasoning
Figure 8. Stable examples from the Cubeworld causal reasoning pyramid evaluation set.
Figure 9. Unstable examples from the Cubeworld causal reasoning pyramid evaluation
14

---

Limits of Fine-Tuning in Vision Language Models
A.3. Lerer
Figure 10. Examples from the Lerer al. (2016) evaluation set.
B. Human experiment information and demographics
Intuitive physies: human fine-tuning data Participants received a base pay of 1.5$ and an additional bonus of 0.01$ for each correct answer, bringing the maximum payout to 2.5$. Completing the experiment took participants 09:54 minutes on average. All participants were native English speakers from the UK and the US with mean age of 30.34 and a split of 55 females to 57 males.
Intuitive physics: human evaluation data Participants received base pay of 1.5$ for their participation and an additional bonus of 0.01$ for each correct answer, bringing the maximum payout 2.7$. Completing the experiment took participants 11:37 minutes average. All participants were native english speakers from the UK and the US and had mean age of 30.46 and split of 50 females to 50 males.
Causal reasoning: human evaluation data Participants received base pay of 1$ for their participation and an additional bonus of 0.01$ for each correct answer, bringing the maximum payout to L.8$. Completing the experiment took participants 09:07 minutes on average. All participants were native English speakers from the UK and the US with a mean age of 31.32 and split of 50 females to 50 males.
C. Analysis Tools
We analyze all data using Python 3.12.7 using pandas 2.2.2 (pandas Development Team, 2020), seaborn 0.13.2 (Waskom, 2021), matplotlib 3.9.2 (Hunter, 2007), and statsmodels 0. 14.2 (Seabold & Perktold, 2010).
D. Analysis Methods
We Cohen's 's & to analyze the alignment of models to human judgments and the ground truth in Figure 5. Cohen's k is define as:
Cobss.j Cexp...
Ki.j 1 Cexp..5
where Cots is the observed error overlap defined Cobss,, P.P, + (1 (1- P;),, the expected overlap that two observers and j with accuracies p: and p, will have by chance.
with Citj the number of equal responses and Geup
To arrive at single mean & between models and humans, we calculate k for each combination of models and humans and take the mean over the values. We produce bootstrapped 95% confidence intervals by computing the central 95 percentiles over 10,000 random subsamples the x distribution. For comparisons to the ground truth, we use the central 95 percentiles of the distribution over mean i for 10,000 random subsamples the item level judgments. We note that models fine-tuned on human judgments have higher between the model and humans than between the humans themselves (see Figure 5). This is likely because the latter calculation has many degrees of freedom than the former since every human is compared to every other, whereas each human is compared only to model time.
15

---

Limits of Fine-Tuning in Vision Language Models
E. Performance improvement over time
A
90B, 2-4
Blocks
2
3
7B, 2-4
7B, 5-7
11B, 11B,
0.8
90B, 2-4
90B,
Accuracy
0.7
0.6
0.5
3
Base 1000
2000
3000
4000
5000
6000
7000
8000
9000 10000 11000 Final
Number blocks
Fine-tuning steps
Figure 11. A: Models evaluated on intuitive physics tower stacks that either have the same number of blocks as the fine-tuning data or different numbers of blocks compared to the fine-tuning data. Note: bars jittered slightly for better readability. B: Performance of the 90B model fine-tuned on 2-4 blocks on the test set for all number of blocks over the process of fine-tuning. The model performs best on towers is fine-tuned but it generalize to bigger towers somewhat. Generalization decreases the block tower size moves away from the fine-tuning distribution. Both subplots show the proportion of correct answers (accuracy) with Wilson score intervals as error hars.
90B, 3
0.9
0.8
Accuracy
0.7
Fine-tuned model
- 7B, 3 &
7B, 10 & 15
11B, 3
11B, 10 15
908, 3
90B, 10 15
0.6
Blocks
10
15
Base 1000
2000
3000
5000
6000
7000
8000
9000
10000
11000
Final
Number of blocks
Fine-luning steps
Figure 12. A: Models evaluated causal reasoning pyramids that either have the number of blocks the fine-tuning data or different numbers of blocks compared to the fine-tuning data. Models performance is roughly the same for in-distribution and out-of of-distribution pyramid sizes. Note: bars are jittered slightly better readability. B: Performance the 90B model fine-tuned on the test set for all number of blocks over the process of fine-tuning. The model performs well the pyramids it is fine-tuned on and can generalize bigger towers. Both subplots show proportion correct answers (accuracy) with Wilson score intervals error bars.
& block pyramids
16

---

Limits of Fine-Tuning in Vision Language Models
F. Generalization to Lerer
A 1.0
90
B
90B, &
0.
08-
07-
Arturayy
06
05
Blocks
1000 2000
9000
8000
70000
5000
5000
10000
3000
4000
5000
6000
7000
10000 11000
Finel
Fine-tuning steps
Fine-tuning
Figure 13. A: 90B model fine-tuned on intuitive physics towers with 2-4 blocks but evaluated on the Lerer stimuli showing real images of 2-4 block towers. B: 90B model fine-tuned 3 6 block pyramids, evaluated the Lerer stimuli, which have different visual quality and are in another cognitive domain.
G. Generalization to new domains over time
0.65
Fine-tuned model
Fine-tuned model
78, 2-4
78, 5-7
7B, 3
78, 10 15
11B, 2-4
11B, 5-7
11B, 3
11B, 10 15
0.6
90B, 2-4
908, 5-7
908, 3
90B, 10 & 15
0.55
Accuracy
0.5
0.4
0.35
10
15
2
3
4
Number blocks
Number blocks
Figure 14. A: Models fine-tuned on intuitive physics towers but evaluated on causal reasoning pyramids. B: Models fine-tuned on causal reasoning pyramids but evaluated on intuitive physics towers. Models do not generalize to the other domain, though shares the same visual features.
17

---

Limits of Fine-Tuning in Vision Language Models
A
B
90B,
Fine-tuned model
7B,
7B,
0,6
Accuracy
0.5
0.45
Blocks
0.4
0.35
10
15
Base 1000
2000
3000
4000
5000
6000
7000
8000
9000
10000 11000
Final
Number blocks
Fine-tuningg steps
Figure 15. A: Models fine-tuned on intuitive physics towers but evaluated on causal reasoning pyramids. Models again do not generalize this other task, though shares the visual features. B: Performance of the 90B model fine-tuned 2-4 black towers, tested on causal reasoning pyramids. The model does not generalize to any tower size. Both subplots show the proportion of correct answers (accuracy) with Wilson score intervals as error bars.
90B, 3&6
Blocks
Fine-luned model
7B, 3 &
78, 10 15
11B, 3
11B, 10 15
90B, 3 8
90B, & 15
0.6
Accuracy
0.3
Base 1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
11000
Final
Number of blocks
Fine-tuning steps
Figure Models fine-tuned on causal pyramids evaluated on intuitive physies tower stacks. Models not generalize to this other task, even though shares the same visual features and can be seen as pre-requisite for the fine-tuning task. This is especially true for the fine-tuning on block tower pyramids, where computing two block tower. B: Performance of the 90B model fine-tuned on 3 & 6 block pyramids, tested intuitive physics block towers. The model does not generalize to any tower size. Both subplots show the proportion correct answers (accuracy) with Wilson score intervals error bars.
counterfactual question requires solving binary tower stability
18


Reviews:
[{"review_id": "DlGdjYUQhX", "reviewer": "Reviewer_fjkm", "summary": "This paper explores the limitations of Vision-Language Models (VLMs) in causal understanding of the physical world — a problem that is quite interesting to the community. The authors examine the potential of fine-tuning (FT) as a method to improve performance on intuitive physics and causal reasoning tasks. They fine-tune a VLM on tasks from the cognitive domain, specifically intuitive physics and causal reasoning in CubeWorld and a real-world environment. However, the results indicate that fine-tuning alone does not achieve the desired abilities. The model is evaluated on tasks such as IID stability of towers, OOD generalization with different numbers of blocks, OOD domain transfer from CubeWorld to real-world scenes, and counterfactual tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer fjkm, thank you very much for your comments. We appreciate that you think the findings are “very interesting” and that you agree that our “method and evaluation criteria make sense”. In the following we discuss the specific concerns that you raised and how we have sought to remedy them. Your comments have led to our paper becoming considerably stronger, particularly with the inclusion of larger models and the clarification of key methodological points. \n\\\n\\\n*The 2B and 7B might be too small and having at least one other class of VLM in the evaluation would provide a better perspective. Fine-tuning and how weights encode sub-routines for solving the task can be affected by the number of parameters.*\n\\\n\\\nThank you for raising this point. We agree that testing only the 2B and 7B Qwen models restricts the generality of the claims we can make based on our experiments. To remedy this, we have added two bigger models from another model family: Llama-3.2 Vision 11B and 90B. We find the same pattern across both model classes and all model sizes - models are capable of generalising to new tower sizes, particularly when trained on shorter towers, but they fail to generalise to naturalistic data or to a new cognitive task (see here for updated versions of Figures 3 and 13 with 7B as reference: [Figure 3](http://postimg.cc/HVcmqHCm), [Figure 13](http://postimg.cc/HjzJNvx9)).\n\\\n\\\n*The related work seems too small and could be expanded to include studies that discuss (1) causal understanding/generalization on visual scenes in the pre-VLM era, and (2) Any other relevant VLM works and how they leave a gap with regards to evaluating causal understanding ability. (3) Studies on VLMs focusing on other kinds of reasoning abilities (not necessarily causal understanding/reasoning or perhaps using text only).*\n\\\n\\\nThank you for this comment, which also echoes the comments of reviewers uVR9 and 3bXm. Regarding point (1), we have referenced the CLEVRER dataset and a prominent model that is trained on it [1-3], which are key works studying causal reasoning and generalization in computer vision systems. Regarding points (2) and (3), we have referenced several papers suggested by the other reviewers. In particular, we have included references to other papers discussing causal reasoning and generalisation on other cognitive tasks in LLMs and VLMs [4-7]. We have also included references to works proposing explanations for why VLMs may struggle to generalise to tasks such as intuitive physics and causal reasoning [8-9], as well as work employing more varied fine-tuning datasets to improve performance [10].\n\\\n\\\n*What is the rationale behind seeking alignment with human behavior?*\n\\\n\\\nThank you for this question. We realize that we were not explicit enough about the hypotheses behind seeking alignment with human behavior. We have added further explanation to the related works section, including [10-11], which were also pointed out by reviewer 3bXm. Binz et al. show that fine-tuning on human choices can lead to models that predict human behavior in previously unseen tasks. On our tasks, human choices and the ground truth are not perfectly aligned, and we sought to explore whether fine-tuning could (a) align models with human choices, and (b) whether training on human choices would lead to improved generalisation performance. We were interested in exploring (b) because fine-tuning on human choices could lead to models learning human intuitions, which might be more robust and generalizable. We confirmed (a) but found only limited evidence for (b): fine-tuning on human behaviour only confers a slight advantage at transferring to the naturalistic Lerer et al. (2016) tower blocks, but no detectable advantage for transferring to a different cognitive task.\n\\\n\\\n\\\n[1] Johnson, J.,  et al. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. \n\n[2] Yi, Kexin, et al. (2020). Clevrer: Collision events for video representation and reasoning.\n\n[3] Chen, Zhenfang, et al. (2021). Grounding physical concepts of objects and events through dynamic visual reasoning.\n\n[4] Dziri, et al.  (2023). Faith and fate: Limits of transformers on compositionality.\n\n[5] Zhou, H., et al. (2023). What algorithms can transformers learn? A study in length generalization. \n\n[6] Zhou, R., et al. (2024). Math for AI: On the Generalization of Learning Mathematical Problem Solving.\n\n[7] Li, C., et al.  (2024). In-context compositional generalization for large vision-language models. \n\n[8] Campbell, D., et al. (2024). Understanding the limits of vision language models through the lens of the binding problem. \n\n[9] Frankland, S. M. et al. (2025). No Coincidence, George: Processing Limits in Cognitive Function Reflect the Curse of Generalization.\n\n[10] Binz, M. et al.  (2024). Centaur: a foundation model of human cognition. \n\n[11] Binz, M. & Schulz, E. (2023). Turning large language models into cognitive models.", "claims_and_evidence": "Yes, but a more rigorous study could be done as pointed out in the \"Cons\" below.", "methods_and_evaluation_criteria": "Yes, the benchmark, method, and evaluation criteria make sense. I am a bit unclear about why the alignment with human behavior is of importance that the paper reports.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes. I don't have any issues with the design/analyses currently in the paper -- except that there could be a more complete study as discussed in the Cons below.", "relation_to_broader_scientific_literature": "To me, it seems that evaluation studies about the limitations of VLMs are only gradually emerging. Previous studies have not explored the intersection of visual causal understanding, fine-tuning, and VLMs.", "essential_references_not_discussed": "Essential references appear to be cited.", "other_strengths_and_weaknesses": "## Pros\n\nVery interesting findings:\n\n1. Fine-tuning on CubeWorld does not lead to good performance on real world data suggesting the model / fine-tuning process is lacking the ability to work with abstractions.\n2. The ability to solve one task does not lend it the ability to solve a different yet related task. \n\n## Cons\n\n1. The 2B and 7B might be too small. I don’t mean to say that a larger model would necessarily resolve all problems, just that 2B and 7B may be too small to claim it as the “limits of fine-tuning”.\n2. Also, having only Qwen in the evaluation is a limitation — having at least one other class of VLM in the evaluation would provide a better perspective. Else, the results might be too specific to Qwen and don’t provide a general message to the community.\n3. The related work seems too small and could be expanded to include studies that discuss (1) causal understanding/generalization on visual scenes in the pre-VLM era, and (2) Any other relevant VLM works and how they leave a gap with regards to evaluating causal understanding ability. (3) Studies on VLMs focusing on other kinds of reasoning abilities (not necessarily causal understanding/reasoning or perhaps using text only).", "other_comments_or_suggestions": "See Cons listed above.", "questions_for_authors": "> Alignment with human behavior\n> \n\nWhat is the rationale behind seeking alignment with human behavior? (I don’t mean to argue against it, just seeking more clarification.)\n\n> L407 (right): Models’ inability to generalize to another cognitive domain is not due to them being limited in parameters or potential ability — models fine-tuned on a mixture of intuitive physics and causal reasoning data performed well in both domains.\n> \n\nYes, but fine-tuning and how weights encode sub-routines for solving the task can be affected by the number of parameters. Wonder if the authors have thoughts about this and if it would be informative to show whether scaling model size has any effect on the generalization / causal understanding performance. The authors later point out in the discussion that this is a future work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tm5KpMcP7E", "reviewer": "Reviewer_mM6z", "summary": "This is an interesting piece of work that seems to be among the first to investigate the following: fine-tuning is a widespread approach to improving LLM performance in the text domain, but for domains such as intuitive physics and causal reasoning, which are not text-related and not really purely visual capabilties either, how can can finetuning go to develop or improve such fundamental capabilities in VLMs?\n\nOne would not really expect this to work extremely well (unlike text), and I doube the author expected it either, but the paper does shed some light on the extent to which it works. In short, it works somewhat well within the domain (not that surprisingly), but does not generalize to the other domain.\n\n## Update after rebuttal:\n\nI find that the work was/is generally interesting, provides a useful contribution, and perhaps worth publicizing (i.e. acceptance).\n\nMy main issues were with the lack of robustness in the evaluation, and the over-generality of the claims, esp. the title. At the very least, the work is not about \"reasoning\" in general.\n\nCritically contingent on the new expt results (more model, repeats, etc.) claimed by the authors, I upgrade my review score from 2 to 3.\n\nHowever, I would strongly, strongly recommend that the title needs to revised and scoped down. (Whether the AC/SAC can enforce that, I'm not sure...)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer mM6z, thank you very much for your feedback. We appreciate that you think our work is interesting and that the “designs and analyses were generally sound”. Furthermore, we are glad to hear that you agree that our work “does shed some light on the extent to which [vision finetuning] works”. Below, we discuss the specific concerns you raised and the revisions we have made, which we think have strengthened our paper.\n\\\n\\\n*One would not really expect [fine-tuning to improve fundamental capabilities in VLMs] to work extremely well (unlike text), and I doubt the authors expected it either.*\n\\\n\\\nThank you for this comment. As you suggest, finetuning has proven to be an effective method for producing generalisable behaviour in text-only models. However, as also highlighted by reviewer fjkm, we are not aware of prior work that establishes the link between fine-tuning and improved visual cognition in VLMs. Therefore, our work here seeks to investigate whether finetuning can confer an advantage to the visual domain too. \n\\\n\\\nWhile previous work has established that pre-trained VLMs do not perform well in visual cognition tasks, we did not know whether fine-tuning could sufficiently improve their performance on these tasks. We were surprised to find that fine-tuning can make models perform well on the specific task they are fine-tuned on. Furthermore, we found that the models can even generalize robustly to unseen towers of sizes that they had not seen during training. Both of these results show that fine-tuning works surprisingly well in improving VLM performance on specific tasks and domains. \n\\\n\\\nHowever, the ways in which models can generalize from their fine-tuning data is severely limited. We show that they have trouble generalizing to visually distinct stimuli as well as to visually similar stimuli in another domain. The take-away from this investigation therefore should not be that fine-tuning does not improve VLM capabilities, but rather that it leads to very task-specific improvements that do not generalize well. \n\\\n\\\n*Whatever claims the paper makes, they are severely limited by a number of factors, as already pointed out by the authors themselves in the Discussion.*\n\\\n\\\nThe main limitations we outline in the discussion were that we: (1) investigate only smaller models of a single model family, and (2) use a single parameterisation for fine-tuning and only one dataset distribution per domain. We have now fixed these limitations by: (1) adding bigger models of another class with 11B and 90B parameter versions of Llama-3.2 Vision; (2) conducting three repetitions of every model on every dataset, using a different adapter weight initialisation and a different sample of fine-tuning data. Our new results follow the same pattern as our previous results. We think that these extensions have significantly improved the generality and robustness of our claims.\n\\\n\\\n*I'm not sure any claim about intuitive physics whatsoever is supported. Rather, any claims should be minimally scoped down to the very specific thing being investigated.*\n\\\n\\\nWe agree that intuitive physics refers to a broad set of capabilities, of which we only investigate a subset. The tower stacking task is a canonical task that has long been used as a testbed for intuitive physical capabilities in machine learning systems [1-4], drawing on an even longer history in cognitive science using the same task [5-7]. However, we also agree that it is important to be specific about the scope of the investigated capabilities. We have therefore outlined the scope of our experiments more specifically in the discussion section, namely, that we focus on model intuitions for stability judgements involving solid, uniformly dense blocks.  \n\\\n\\\n[1] Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. \n\n[2] Battaglia, P., Pascanu, R., Lai, M., & Jimenez Rezende, D. (2016). Interaction networks for learning about objects, relations and physics. \n\n[3] Lerer, A., Gross, S., & Fergus, R. (2016). Learning physical intuition of block towers by example.\n\n[4] Piloto, L. S., Weinstein, A., Battaglia, P., & Botvinick, M. (2022). Intuitive physics learning in a deep-learning model inspired by developmental psychology.\n\n[5] Baillargeon, R., & Hanko-Summers, S. (1990). Is the top object adequately supported by the bottom object? Young infants' understanding of support relations. \n\n[6] Spelke, E. S., Breinlinger, K., Macomber, J., & Jacobson, K. (1992). Origins of knowledge.\n\n[7] Baillargeon, R., Needham, A., & DeVos, J. (1992). The development of young infants' intuitions about support.", "claims_and_evidence": "1) Whatever claims the paper makes, they are severely limited by a number of factors, as already pointed out by the authors themselves in the Discussion.\n\n2) More importantly though, intuitive physics and causal reasoning are very broad capabilities, and the blocksworld-like datasets used here are a very small sliver of these domains/capabilities. I'm not sure any claim about intuitive physics whatsoever is supported. Rather, any claims should be minimally be scoped down to the very specific thing being investigated, e.g. stability, etc. In addition, these need to be caveated that only solid/rigid cubes of uniform density are used, etc. etc.", "methods_and_evaluation_criteria": "Methods and evaluation criteria were generally sound.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Experimental designs and analyses were generally sound, but subject to a number of limitations, some of which the authors themselves point out (e.g. size of models finetuned, alternative finetuning procedures, etc. etc.)", "relation_to_broader_scientific_literature": "This paper is generally somewaht useful in terms of contributing some knowledge about what works (or not) to what extent for improving intuitive physics and causal reasoning through fine-tuning of VLMs. However, as the authors themselves point out in the Discussion section, there are a number of limitations, which in my view quite significantly limit the usefull of this paper.", "essential_references_not_discussed": "Nil.", "other_strengths_and_weaknesses": "Nil.", "other_comments_or_suggestions": "Nil.", "questions_for_authors": "Nil.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kukKkGGfRZ", "reviewer": "Reviewer_uVR9", "summary": "This paper investigates whether fine-tuning models on intuitive physics and causal reasoning improves performance within these specific domains. The authors conclude that such fine-tuning does not enhance performance on other visual characteristics or tasks in different cognitive domains.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer uVR9, thank you very much for your thorough review. We appreciate that you find our topic “intriguing” and our experimental designs “sound and valid”. In the following we discuss how we have remedied the concerns that you have raised. Your comments have, in our view, considerably strengthened the paper, particularly regarding the inclusion of larger models and the development of a second, more targeted dataset for evaluating between-task generalisation.\n\\\n\\\n*Testing a broader range of model architectures and including larger models would strengthen the conclusions. Will larger models exhibit different patterns regarding this phenomenon?*\n\\\n\\\nThank you for highlighting this. To remedy this weakness, we added a new family consisting of two larger models: Llama-3.2 Vision 11B and 90B. We find the same pattern across both model families and all model sizes - models are capable of generalising to new tower sizes, particularly when trained on shorter towers, but they fail to generalise to naturalistic data or to a new cognitive task (see here for updated versions of Figures 3 and 13 with 7B as reference: [Figure 3](http://postimg.cc/HVcmqHCm), [Figure 13](http://postimg.cc/HjzJNvx9)).\n\\\n\\\n*It appears that all experiments are conducted on the cubes dataset; I suggest that incorporating other types of physical understanding tasks could also help verify the findings.*\n\\\n\\\nThank you for this suggestion. We chose to design the cubeworld dataset to ensure that models have a fair chance of generalizing given that they have become used to the visual characteristics of the environment.  This allows us to infer to some degree whether models’ failure to generalize is due to small differences between the tasks or due to their inability to learn intuitive theories through task-specific fine-tuning. We also designed the tasks in cubeworld based on a long history in cognitive science of using block towers to study physical understanding in humans [1-3]. However, we agree that it is important to be clear about the limitations of our work and have therefore added a sentence to the limitations outlining that we only investigate a subset of intuitive physics here, also taking into account the comments of reviewer mM6z. \n\\\n\\\n*The experiments do not provide sufficient analysis on the reasons behind the failures in generalization, which limits the insights offered by the study. What are the underlying reasons for the failures in generalization?*\n\\\n\\\nThank you for raising this question. To establish whether failures in generalization are due to small differences between tasks, or if the models struggle with learning intuitive theories through task-specific fine-tuning, we added another dataset where differences between the two cognitive domains are kept as minimal as possible. We generate paired images of pyramids, in which the causal reasoning image contains a red block which is removed to generate the intuitive physics image (see this [Figure](http://postimg.cc/3k5Czc3k)). \n\\\n\\\nIn principle, being able to reason about the counterfactual stability of a pyramid ought to predispose models to reason about the factual stability of pyramids. Thus, we expected a transfer from causal reasoning to intuitive physics, especially since we test models using the corresponding images from the pairs they were fine-tuned on. Furthermore, we explicitly tell the models that the red block has been removed. Nevertheless, we do not find evidence of this transfer, suggesting that task-specific fine-tuning does not lead to models learning intuitive theories. Instead, they appear to be learning task-specific superficial shortcuts that do not generalize [4-5].\n\\\n\\\n*I find that the paper is poorly written and lacks organization. I recommend that the authors reorganize the experimental sections and clarify the conclusions. Figures 3 and 4 are not clearly conveyed, making them difficult to understand.*\n\\\n\\\nWe are sorry to hear this. Conveying our ideas and conclusions clearly is very important to us, so in order to improve this, we added a conclusion for every section of the results. We have also updated the captions of Figures 2, 3, and 4 to make them easier to understand. We hope that these changes improve the readability of our paper.\n\\\n\\\n*Essential References Not Discussed.*\n\\\n\\\nThank you for bringing these references to our attention. We have extended the related works section based on your comments and those of reviewers fjkm and 3bXm.\n\\\n\\\n\\\n[1] Baillargeon, R., & Hanko-Summers, S. (1990). Is the top object adequately supported by the bottom object? Young infants' understanding of support relations.\n\n[2] Spelke, E. S., et al. (1992). Origins of knowledge.\n\n[3] Baillargeon, et al. (1992). The development of young infants' intuitions about support.\n\n[4] Ilyas, A., et al. (2019). Adversarial examples are not bugs, they are features.\n\n[5] Geirhos, R., et al. (2020). Shortcut learning in deep neural networks.", "claims_and_evidence": "The submission is supported by some evidence. As demonstrated in Figure 2, fine-grained control variable experiments were conducted across various settings, including different fine-tuning datasets and evaluation sets. However, the experiments only utilize Qwen2-VL models with 2B and 7B parameters. I believe that testing a broader range of model architectures, such as LLaVA, and including larger models, like those with 30B or 70B parameters, would strengthen the conclusions.", "methods_and_evaluation_criteria": "It appears that all experiments are conducted on the cubes dataset; I suggest that incorporating other types of physical understanding tasks could also help verify the findings.", "theoretical_claims": "There are no equation in this paper.", "experimental_designs_or_analyses": "I have reviewed the experimental designs presented in Figures 2, 3, and 4, and they appear to be fundamentally sound and valid. However, the key points in Figures 3 and 4 are not clearly conveyed, making them difficult to understand. I recommend that the authors add annotations to these figures to clarify the key points.", "relation_to_broader_scientific_literature": "See the below session.", "essential_references_not_discussed": "This observation is linked to the discussion on the generalization of reasoning in large language models (LLMs) and vision-language models (VLMs). In the context of LLMs, it relates to references [1][2][3], which explore whether these models, when fine-tuned on specific tasks, can generalize to in-domain/out-of-domain or simpler/more complex tasks. I believe these discussions are also relevant to this paper's ideas, as the understanding and intelligence of VLMs are fundamentally derived from LLMs. Consequently, grasping the generalization capabilities of LLMs is essential for comprehending the generalization of VLMs. For vision-language models, this is connected to reference [4].\n\n[1] Faith and Fate: Limits of Transformers on Compositionality. Nouha Dziri et al. \n[2] Algorithms Can Transformers Learn? A Study in Length Generalization. Hattie Zhou et al. \n[3] Math for AI: On the Generalization of Learning Mathematical Problem Solving. Ruochen Zhou et al.\n[4] In-Context Compositional Generalization for Large Vision-Language Models. Chuanhao Li et al.", "other_strengths_and_weaknesses": "This work addresses the generalization problem of vision-language models (VLMs), which is an intriguing topic. However, the experiments do not provide sufficient analysis on the reasons behind the failures in generalization, which limits the insights offered by the study. Furthermore, I find that the paper is poorly written and lacks organization. I recommend that the authors reorganize the experimental sections and clarify the conclusions. For example, Figures 3 and 4 are challenging to interpret, and the descriptions associated with these figures could be placed in separate sections for enhanced clarity. Additionally, restructuring the discussion section could help present the key points more effectively.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. What are the underlying reasons for the failures in generalization? Could the authors provide a more detailed analysis of these factors?\n2. Will larger models exhibit different patterns regarding this phenomenon? Like 13B/30B/70B models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jbqz5Drp9V", "reviewer": "Reviewer_3bXm", "summary": "This work investigates the extent to which task-specific fine-tuning can help VLMs to overcome limitations in two psychologically inspired domains, with a special emphasis on the extent to which benefits of fine-tuning generalize between tasks and across variations within a task.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 3bXm, thank you very much for your comments. We are happy to hear that our experiments “very clearly demonstrate that task-specific fine-tuning leads to brittle improvements”, this was the main point we were trying to convey. We also appreciate your assessment that our “methods and evaluation criteria are all sensible” and that you found our discussion of related work “generally strong”. In the following we discuss the specific concerns that you raised and how we have sought to remedy them. We think addressing them has significantly strengthened our paper, particularly with respect to improving the clarity and including a GPT baseline. \n\\\n\\\n*The experiments are described as establishing the limits of fine-tuning in general, but it seems more accurate to say that they establish the limits of task-specific fine-tuning.*\n\\\n\\\nThank you for raising this point. We agree that our experiments first and foremost establish the limits of fine-tuning on specific tasks, namely, reasoning about the stability of solid block towers. However, we would like to emphasise the long history of using such tasks for investigating visual cognition in humans. We think that they are sufficiently representative for us to make some more general claims about visual cognition in vision language models. Our results present some evidence that fine-tuning models on one cognitive task does not lead to generalization on a related but distinct cognitive task. In contrast to this, human learning is characterised by a robust ability to generalise between distinct but related tasks. \n\\\n\\\nYou are correct to point out that models trained on both tasks do generalize slightly better than models trained on only one (Fig. 2). However, with our current datasets, we cannot evaluate how these models would generalise to a third cognitive task in cubeworld, and we suspect they would generalize poorly, given our current results. \n\\\n\\\n*Second, it seems that the fine-tuned models actually outperform humans within the specific domain that they are fine-tuned in (although this is difficult to tell because the data are not presented in the same plot). In other words, the results clearly demonstrate that fine-tuning yields only very brittle benefits, which does not seem very human-like, but it seems worth mentioning that the performance is actually quite strong relative to humans within these domains.*\n\\\n\\\nThank you for these comments. First, we have addressed the problem of model-human comparison by including a human baseline in Figure 2, and all heatmaps in the appendices. Second, we have emphasised in the discussion that fine-tuned models outperform humans on the domains they have been directly trained on, while being outperformed by them on out-of-distribution domains.\n\\\n\\\n*In Figure 2, it would be helpful to include additional rows illustrating 1) zero-shot performance of the base model, 2) human performance, 3) zero-shot performance for a state-of-the-art model (e.g. GPT-4o or Claude).*\n\\\n\\\nThank you for this suggestion. We agree that this makes it easier to interpret the results. We have therefore added rows showing the zero-shot performance of the base model, human performance and zero-shot performance for GPT-4o. You can see the updated figure here: [Figure 2](http://postimg.cc/8JwwDwYv) (please note this shows Llama-3.2-11B, as other reviewers highlighted the need for bigger models from other families).\n\\\n\\\n*Relation to broader scientific literature could be expanded.*\n\\\n\\\nThank you for the very relevant pointers. We have updated our related works section according to the feedback by you and by reviewers fjkm and uVR9. We were aware of [2] but had not made the connection to the difficulties that models have with generalization. This is a very interesting direction and we thank you for bringing it to our attention. We have also added [3] to better motivate why fine-tuning on human choices is of interest and might lead to more robust generalization. \n\\\n\\\n*In Figure 4, why is the model-to-human alignment apparently higher than the human-to-human alignment in some cases (i.e. the bars labelled 'humans')?*\n\\\n\\\nThis is an intriguing outcome from our initial results. It indicates that the average agreement between the model and each human rater is higher than the average agreement between every pair of human raters. We speculate that this arises from the fact that there is a sizable variance in human ratings with heavy tails; the fine-tuned model appears to accurately capture the average human rating, resulting in a higher average agreement than the human case, where average agreement is pulled down by raters with different ratings due to the way Cohen’s kappa is computed, since the observed agreement will be low relative to the expected agreement. We should note that our plots have changed slightly, since we now average over three fine-tuning seeds (see the updated [Figure 4](http://postimg.cc/47qjMqfW) with the 7B Qwen model).", "claims_and_evidence": "The experiments very clearly demonstrate that task-specific fine-tuning leads to brittle improvements, in the sense that these improvements do not generalize between tasks, and they do not even clearly generalize to slightly different versions of the same task. I have a few comments about the interpretation of these results. First, the experiments are described as establishing the limits of fine-tuning in general, but it seems more accurate to say that they establish the limits of *task-specific* fine-tuning. It would be good to clearly state that the experiments do not necessarily rule out more robust gains from fine-tuning across a wider distribution of tasks (and they even seem to suggest that this might work, given the relatively improved generalization in the model fine-tuned on both tasks). Second, it seems that the fine-tuned models actually outperform humans within the specific domain that they are fine-tuned in (although this is difficult to tell because the data are not presented in the same plot). In other words, the results clearly demonstrate that fine-tuning yields only very brittle benefits, which does not seem very human-like, but it seems worth mentioning that the performance is actually quite strong relative to humans within these domains.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are all sensible. In Figure 2, it would be helpful to include additional rows illustrating 1) zero-shot performance of the base model, 2) human performance, 3) zero-shot performance for a state-of-the-art model (e.g. GPT-4o or Claude). This will help to assess not only how fine-tuning generalizes between tasks, but also how the performance of the fine-tuned models compares in absolute terms with humans and other models.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "All experiment designs and analyses appear reasonable.", "relation_to_broader_scientific_literature": "The discussion of the relationship to the broader literature is generally strong. There are two points where it may be expanded. First, some recent work [1] has found that failures of VLMs can be related to the classic 'binding problem' in cognitive science, in the sense that the failures are similar to those observed for human participants under time pressure. This is also related to theoretical work [2] which suggests that sequential processing (i.e. inference-time compute) is needed to overcome these binding errors. One prediction of this perspective is that, so long as the model is required to generate a response in a single feedforward pass, fine-tuning will only lead to very task-specific benefits, whereas inference-time compute will be needed for more generalizable benefits. The results in this paper seem consistent with that line of reasoning, which may be interesting to discuss.\n\nAdditionally, it may be interesting to discuss whether approaches like [3] may be lead to more generalizable benefits, i.e. via fine-tuning on  a broader distribution of tasks.\n\n[1] Campbell, D., Rane, S., Giallanza, T., De Sabbata, C. N., Ghods, K., Joshi, A., ... & Webb, T. (2024). Understanding the limits of vision language models through the lens of the binding problem. Advances in Neural Information Processing Systems, 37, 113436-113460.\n\n[2] Frankland, S. M., Webb, T. W., Lewis, R. L., & Cohen, J. D. (2025). No Coincidence, George: Processing Limits in Cognitive Function Reflect the Curse of Generalization.\n\n[3] Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., ... & Schulz, E. (2024). Centaur: a foundation model of human cognition. arXiv preprint arXiv:2410.20268.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "In Figure 4, why is the model-to-human alignment apparently higher than the human-to-human alignment in some cases (i.e. the bars labelled 'humans')?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This study aims to enhance the visual cognition of VLMs by incorporating visual stimuli and human feedback. Specifically, the authors fine-tune the model on intuitive physics and causal reasoning tasks. While all reviewers agree that the study addresses an interesting research question, they also raise concerns regarding its applicability and experimental validity. Overall, the strengths outweigh the weaknesses, and I am inclined to accept this paper.
Decision: Accept (poster)