Title: Vision Language Models Cannot Reason About Physical Transformation
Primary area: datasets and benchmarks
Keywords: vision language models, physical transformation, multi-image understanding, spurious correlation

Abstract: The ability to comprehend physical transformations is essential for reasoning in dynamic, real-world environments, yet it remains unclear whether vision–language models (VLMs) possess this capacity. To address this gap, we evaluate whether VLMs exhibit conservation—the understanding that physical quantities remain invariant despite changes in appearance. Inspired by Piaget’s framework, we design a systematic benchmark for conservation reasoning across four quantitative domains: number, length, volume, and size. Each task requires models to integrate visual evidence across time to identify invariant properties under transformation. To guard against spurious success, we also curated a set of negative control tasks paired one-to-one with the benchmark tasks, in which the targeted quantities are not conserved. Both benchmarks and controls incorporate four prompt types, three frame extraction methods, and 48 task variations per domain, yielding 384 baseline tasks and 13,824 total questions. Evaluating 34 VLMs, we find that none achieve systematic success: across conserving and non-conserving tasks, models consistently perform only marginally above chance, with those excelling on conservation tasks performing inversely on controls, indicating rigid biases in reasoning about physical processes. Moreover, models show no benefit from higher temporal resolution or prompt design, suggesting a reliance on static visual cues. Together, these findings indicate that current VLMs fail to internalize the structured representations necessary for systematic physical inference.