Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models
Authors: Yinglun Zhu
Jiancheng Zhang
Fuzhi Tang
Venue: ICLR 2026 Poster (2026)
Keywords: Compositional reasoning
multimodal learning
test-time adaptation
evaluation metrics
vision-language models
Primary area: foundation or frontier models, including LLMs

Abstract: Frontier AI models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at or below random chance on established benchmarks. We revisit this problem and show that widely used evaluation metrics systematically underestimate model capability. To correct this artifact, we introduce a group matching score that more faithfully evaluates model capability. Moreover, correctness under the new metric can be translated into correctness under existing metrics via a simple overfitting step. This adjustment enables SigLIP-B16 to surpass all previous results and GPT-4.1 to yield the first result surpassing estimated human performance on Winoground. Building on this insight, we propose Test-Time Matching (TTM), an iterative, self-improving algorithm that further bootstraps model performance without any external supervision. TTM delivers additional, non-trivial improvements: for example, TTM enables SigLIP-B16 to surpass GPT-4.1 on MMVP-VLM, establishing a new state of the art. Importantly, TTM remains broadly effective even on benchmarks without metric-induced effects or group structures, achieving relative gains up to 85.7% on challenging datasets such as WhatsUp. Across 16 dataset variants spanning diverse setups, our experiments demonstrate that TTM consistently improves model performance and advances the frontier of compositional reasoning.

Published as a conference paper at ICLR 2026
TEST-TIME MATCHING: UNLOCKING COMPOSITIONAL
REASONING IN MULTIMODAL MODELS
Yinglun Zhu' Jiancheng Zhang University of California, Riverside {yzhu, jzhan745, fuzhit] edu
Fuzhi Tang
ABSTRACT
Frontier Al models have achieved remarkable progress, yet recent studies suggest they struggle with compositional reasoning, often performing at below random chance on established benchmarks. We revisit this problem and show that widely used evaluation metrics systematically underestimate model capability. To correct this artifact, we introduce a group matching score that more faithfully evaluates model capability. Moreover, correctness under the new metric be translated into correctness under existing metrics via a simple overfitting step. This adjustment enables SigLIP-B16 surpass all previous results and GPT-4. yield the first result surpassing estimated human performance on Winoground. Building on this insight, we propose Test- Time Matching (TTM), an iterative, self-improving algo- rithm that further bootstraps model performance without any external supervision. TTM delivers additional, non-trivial improvements: for example, TTM enables SigLIP-B16 to surpass GPT-4.11 on MMVP-VLM. establishing a new state of the art. Importantly, TTM remains broadly effective even on benchmarks without metric-induced effects or group structures, achieving relative gains up to 85.7 % on challenging datasets such as WpatsUp. Across 16 dataset variants spanning di- verse setups, experiments demonstrate that TTM consistently improves model performance and advances the frontier of compositional reasoning.
INTRODUCTION
Compositional reasoning provides stringent test of frontier AI models, assessing their ability to systematically combine primitive elements- such as objects, attributes, and relations- interpret or reason about novel configurations (Lake et al., 2017; Bahdanau et al., 2019). Recent benchmarks evaluate this capability by organizing examples into small groups of images and captions that differ in subtle yet systematic ways (Thrush et al., 2022; Hsich et al., 2023; Kamath et al., 2023; Tong et al., 2024; B Burapacheep et al., 2024). For example, Winoground consists of x 2 groups where both captions contain the same words but in different orders, such that each caption correctly describes only one of the two images.
Despite the impressive practicality of modern multimodal systems, both contrastive vision-language models (VLMs) and multimodal large language models (MLLMs) have been reported to perform at or below random guessing these benchmarks (Thrush al., 2022; Diwan al., 2022: Tong et al., 2024; Burapacheep et al., 2024; Li et al., 2025). On Winoground, even frontier AI models still fall far short of the estimated human performance of 85.5 (Thrush al., 2022), with the previous state of the art reaching only 58.75, achieved through scaffolding and prompt tuning GPT-4V (Wu et al., 2023; Vaishnav & Tammet, 2025).
We revisit this conclusion and show that the widely used evaluation metric GroupScore (Thrush et al., 2022; Tong al., 2024; Burapacheep et al., 2024) systematically underestimate model capability. We introduce a group matching score (GroupMatch) that more faithfully evaluates model capability. Importantly, correctness under GroupMatch can translated into correctness under GroupScore by overfitting to the matchings induced by GroupMatch, an approach we refer to SimpleMatch (see Section 3.1). SimpleMatch alone reveals substantial hidden capability: as shown in Fig. 1,
PProectt lead and corresponding author, Code: htttps:/ggitii http:///ihuuuu.coooiiglluuut http://ooo/yygggn///ssttttttttte teme--imeeeee time-matching

---

Published as a conference paper at ICLR 2026
Winoground
MMVP-VLM
ColorSwap
100
91.4
88.0
94.3
91.1
97.4
61.5
67.9
69.8
68.2
Accuracy
SimpleMatch (Ours)
40
TTM (Ours)
20
SigLIP-B16
GPT-4.1
SigLIP-816
GPT-4.1
SigLIP-B16
GPT4411
Figure SimpleMatch and TTM substantially improve VLM and MILM performance on com- positional reasoning benchmarks Winoground, MMVP-VLM, and ColorSwap. achieving new per- formance records. We highlight: (1) SimpleMatch enables GPT-4.1 to surpass human performance on Winoground (leff), and (2) TTM enables SigLIP-B16 surpass GPT-4.1 MMVP-VLM, establishing a new state of the art (middle).
SigLIP-B16 improves from 10.25 > on Winoground, 22.96 + 81.48 on MMVP-VLM, and 30.33 88 on ColorSwap. surpassing all previous results without access to additional data (Wu et al., 2023; Vaishmav Tammet, 2025; Zhang et al., 2024c; Burapacheep et al., 2024). GPT-4. also improves dramatically, from 69.75 91.38 Winoground, 68.15 88.52 on MMVP-VLM, and 91.08 + 97.42 on 85.5 Winoground (Thrush et al., 2022).
the first result to surpass the estimated human performance
Building on this insight, we introduce Test-Time Matching (TTM), an iterative, self-improving algorithm that further bootstraps model performance without any external supervision. TTM selects matching-inducee pseudo-labels for self-training and progressively relaxes the selection threshold expand coverage over the test set. This yields additional, non-trivial gains top of SimpleMatch: SigLIP-B16 reaches 72.5 on Winoground, 89.44 on MMVP-VLM, and 94.25 on ColorSwap. Remarkably, TTM elevates SigLIP-LI6 the level of GPT-4.1 on ColorSwap (Table 1) and enables SigLIP-B16 to surpass GPT-4.11 on MMMP--LLN, establishing a new state of the art. See Fig. 1 and Table for details. Crucially, TTM broadly effective even where metric changes cannot help- on 1 x & benchmarks such as SugarCrepe (Hsieh et al., 2023) and WhatsUp (Kamath et al., 2023), where GroupScore and GroupMatch coincide, TTM still delivers substantial test-time improvements, including up to 85.7 % relative gains challenging datasets such as WpatsUp (Fig. 3).
Finally, we extend TTM beyond group-structured datasets by formulating single global matching across all images and captions. Even one-shot global matching outperforms GroupScore, and applying the global variant of TTM yields further improvements, demonstrating that the test-time matching principle generalizes robustly beyond benchmarks with group structures.
Contributions. We summarize our main contributions below:
1. Correcting evaluation metries. We show that the widely used evaluation metric GroupScore can systematically underestimate model capability, and we introduce GroupMatch as a more faithful measure. We further show that simple overfitting step translates correctness under GroupMatch into correctness under GroupScore; with this adjustment, GPT-4.1 achieves the first Winoground result surpassing estimated human performance.
Test-time matching for self-improvements. We propose Test-Time Matching (TTM), an iterative, self-immree iel--improiing algorithm that selects matching-indus induced pseudo-labels for self-training and pro- gressively relaxes the selection threshold to expand coverage. TTM delivers additional, non-trivial gains on top of SimpleMatch, enabling SigL.P-BB to surpass GPT-4.1 on MMVP-VLM and establishing new state of the art.
14, the most recent GPT model available to us at submission time that provides log-probability outputs, enabling more accurate computation of similarity scores (Lin et al., 2024). At the time of submission, GPT-5 did support log probability outputs.
'We use
2

---

Published as a conference paper at ICLR 2026
3. Broad applicability of TTM. We conduct extensive experiments actoss 16 dataset variants spanning 2 2, x k, and non-grown non-grouped settings, demonstrating that TTM consistently improves model performance across diverse scenarios, including those without metric-induced effects or predefined group structures.
Paper organization. Section 2, we review group-structured evaluation for compositional reason- ing. In Section 3, revisit evaluation metries, introduce new group matching score (GroupMatch), present our test-time matching (TTM) algorithm, and extend to global mnon-grouped) settings. In Section 4, we report results benchmarks with 2 x 2 groups, 1 k groups, and non-grouped structures, together with ablations and analysis. We conclude in Section 5. Related work, formal proofs, additional experimental details, and extended results are provided in the Appendix.
PRELIMINARIES
We study compositional reasoning in multimodal models. Benchmarks for this task are typically organized into groups of images and captions, often of shape x k or x k. Within each group, the images and captions differ in subtle yet systematic ways. For example, the widely used Winoground dataset consists of groups with two images and two captions, where both captions contain the same set of words but different orders, such that each caption correctly describes only one of the two images (Thrush et al., 2022).
To succeed on these benchmarks, model must correctly align images and captions within each group. Let 84j := s(,,, Cj) denote the similarity score between image 1; and caption Cj. For contrastive vision-language models such as CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), sij is typically computed the inner product of image and text embeddings. For multimodal large language models, similarity can instead be estimated using metrics such as VQAScore (Lin et al., 2024). We collect all scores into similarity matrix which shares the same shape the group.
The GroupScore metric for A groups. Consider group of k images and k captions with ground--rruti pouun pairings {(4, C:) i= hidden from the learner. The most widely used evaluation metric the GroupScore (Thrush et al., 2022: Tong et al., 2024; Burapacheep al., 2024). The GroupScore equals if the model's similarity scores admit a bijection such that (i) each image is assigned its correct caption and (ii) each caption is assigned to its correct image; otherwise it equals 0. Mathematically, we have
Vi: Sü >> maxjşi $tj and S max Sja.
GroupScore(s)
otherwise.
Evaluation metrics for groups. Without loss of generality, we assume each group consists of 1 image and & captions (Kamath et al., 2023; Hsieh al., 2023). In this case, the GroupScore reduces to the TextScore, which equals if the model selects the correct caption and 0 otherwise.
Scope and extensions. this paper, we primarily focus on l x k and 1 x k groups as they are the most common configurations in compositional reasoning benchmarks. We defer discussion of general rectangular groups of shape m x & to Appendix B.2.
3 METHODS
In Section 3.1, we show that the standard evaluation metric can systematically undercount model capability group-structured benchmarks, and we introduce group matching score (GroupMatch) that corrects this artifact. Building on this, we develop an iterative, self-improving Teme Matching (TTM) algorithm that bootstraps model performance without external supervision (Section 3.2).
Finally, we extend TTM beyond group-structured datasets to a global matching formulation applicable general settings (Section 3.2.1).
3

---

Published as a conference paper at ICLR 2026
3.1 REVISITING EVALUATION METRICS: FROM RANDOM GUESSING TO GROUP MATCHING
Most compositional reasoning benchmarks use the GroupScore metric described in Section 2. Despite the broad practical success of frontier Al models, reported results on established benchmarks particularly those with k x k groups-oaa are often at or below random guessing (Thrush et al., 2022: Diwan et al., 2022; Tong et al., 2024; Burapacheep et al., 2024; Li al., 2025).
Revisiting evaluation metrics. Such counter-intuitive results motivate us to re-examine evaluation metrics for k x & groups. To calibrate their behavior, we analyze random guessing model under each metric. Consider a group of & images {4,)==1 and k captions {C:} with ground-truth pairings C.), hidden from the learner (Thrush et al., 2022; Tong et al., 2024; Burapacheep et al., 2024). For each pair (Ts; Cý), the random guessing model assigns similarity score sim [T, C2) (n0, 1)), producing similarity matrix R with entries 8ig == sim(I, Cý).
Under the widely used GroupScore metric, achieving a score of requires the similarity matrix to satisfy 2A 2k constraints (see Eq. (1)). Equivalently, each diagonal entry sa must be the largest element in both its row and column-a highly restrictive condition. The probability of achieving a group score of under random guessing is given below (see Appendix B.1 for proofs).
Proposition For random similarity scores & E R P(GroupScore(s)) 1)
(k- 1)! (2k 1)!
new group matching score. We introduce a new evaluation metric that evaluates the best overall matching rather than isolated pairwise comparisons. We consider bijective matchings cone-to-one and onto) from images to captions. Let π denote such a matching, where (i) is the caption assigned to image We define the GroupMatch
GroupMatch(s) := {
[ Ei (t,m"(1) > Ei= (t,(()) VTTT"
0 otherwise,
where n² i denotes the ground-truth matching. Intuitively, the GroupMatch equals 1 if the total similarity of the ground-truth matching exceeds that of all other possible matchings. For k = 2. this reduces the simple condition $11 + $22 > 812 + $21. Since there are k! distinct matchings (permutations) and, under random guessing, each is equally likely to maximize the total score, we obtain the following result.
Proposition 2. For random similarity scores s E Rixk P(GroupMatch(()) = 1) z.
Remark The GroupMatch naturally extends to general rectangular groups of shape m k (with m < k) by considering all injective matchings (one-to-one). In these cases, it also improves over the GroupScore, increasing the expected random guessing score from 1/k² to (k- [mmmm m)//Ka /k! (see Appendix B.2 for details). In the special case of 1 x l groups, the GroupMatch and the GroupScore coincide.
SimpleMatch: translating GroupMatch to GroupScore. Two key observations emerge:
(GrouupMatch(s) 1) P(GroupScore(s) 1) for all integers k 1.
If the correct matching x² is selected, overfitting to x* at test time guarantees a GroupScore of 1. In other words, correctness under GroupMatch can be translated into correctness under GroupScore
Taken together, these observations imply that GroupScore systematically underestimates model capability: it imposes unnecessary constraints and can miscount correct matchings as wrong. In fact, one can easily improve model performance under GroupScore by (i) selecting the most likely matching under GroupMatch and (ii) overfitting to that matching test time transfer gains. We refer to this approach as SimpleMatch with GroupMatch. In the commonly studied case with i: = 2, the expected group score of random guessing model increases from 1/6 to 1/2.
"These benchmarks are widely adopted; for example, as of October 2025, Winoground (Thrush et al., 2022)
has over 500 citations and MMVP-VLM (Tong et al., 2024) has nearly 500 citations.
'Since overfitting to matchings induced by GroupMatch achieves the same level of performance under GroupScore, throughout the paper, we report raw model performance under GroupScore and our algorithms' performance under GroupMatch. The latter can always he converted to equivalent GroupScore performance with additional overfitting step.
4

---

Published as a conference paper at ICLR 2026
Algorithm Test-Time Matching (TTM)
Input:
Pretrained fo: test set groups {G:) number of iterations T; thresholds (T.
1: for iteration / = to T do
2: Initialize pseudo-labeled set S +- 0.
3: for each group G; D do
4: Induce matching T ft (G;) t arg max Gi,
5:
Compute margin AGG;; fi-1) as
AGG; fi-1) + (( (G.); Go, fi-1)
max
SiT; ft-1).
**%
6:
S f S U fi Tt then
7:
(G,)))
8: Finetune model on to obtain ft. // Self-improving with no external supervision.
Output: Test-time adapted model fr.
Empirical validation. We evaluate SimpleMatch on SigLIP (Zhai et al., 2023) and GPT-4.1 across three established compositional reasoning benchmarks with k i group structures: Winoground (Thrush et al., 2022), MMVP-VLM (Tong et al., 2024), and Colorswap (Burapacheep et al., 2024). Results are presented in Fig. 1. SimpleMatch reveals substantial hidden capability: SigLIP-BI6 improves from 10.25 > 67 Winoground, 22.96 81.48 on IMVP-VLM, and 30.33 + 88 on ColorSwap, surpassing all previous results without access additional data (Wu et al., 2023; Vaishnav & Tammet, 2025; Zhang et al., 2024c; Burapacheep et al., 2024). GPT-4.1 also improves dramatically, from 69.75 - 91.38 on Winoground, 68.15 15 + 88.52 on MMVP-VLM, and 91.08 - 97.42 on ColorSwap-yyeldinng the first result to surpass the estimated human performance of 85.5 Winoground (Thrush et al., 2022).
3.2 TEST-TIME MATCHING: ITERATIVE BOOTSTRAPPING OF MODEL PERFORMANCE
The GroupMatch metric introduced in Section 3.1 reveals substantial model capability masked by previous metrics. To push performance further, we introduce test-time matching algorithm that iteratively bootstraps model performance, yielding new state-of-the-ae art results. Our method applies to groups of general shapes: we consider bijective matchings for square groups and injective matchings for rectangular groups.
High-level idea. Our test-time matching algorithm (Algorithm 1) proceeds iteratively for iter- ations. At each round E [T'. the current model fe-1 induces candidate matchings for all groups, which serve as pseudo-labels. The algorithm then retains only those matchings it is most confident about, and finetunes on them obtain the next model ft. By repeating this process, the model progressively self-improves directly at test time, without any external supervision.
The core of Algorithm lies in two design choices: (1) how pseudo-labels are induced within each group, and (2) how the confidence thresholds are scheduled across iterations. We discuss both below.
Group matching and pseudo-labeling. For a group G and model ft-1, we define the induced matching ft- (G) arg max, s(x; G, ft-1, where s(n; G, fi-1) En Su,r(a) (G; f(-1) 1) de- notes the total similarity of matching
on G under fi-1- For example, in 2 x 2 group, Ti- (G) (1 + 2 + 2) if $n1 + 822 $12 + 821, and (1
+ 1) otherwise. For 1 x lo group, the induced matching is (1+ arg maxje[K] $1z). We convert se-n (G) into a pseudo-
label (G, (G,TT,_,(C (G)) and add it to the training set S, only when its margin
A(G;fi-)) == s(T,,,, (G),
s( G(T:
is greater than or equal to a threshold тG- By controlling the threshold, we ensure that the model retains pseudo-labels is sufficiently confident about.
A decaying selection threshold schedule. Any pseudo- based method (including TTM) faces
two types of pseudo-labeling errors: (i) false positives incorrect pseudo-labels that are mistakenly included in the training set, and (ii) false negatives- correct predictions that excluded due to
5

---

Published as a conference paper at ICLR 2026
Matching Minegruund (2x2
Matching SugarSrepe [1x2 Grups)
TTM wet Effferent Thrashold Schodales
1400
72
Notthed Cruup
x Notched Groups
Aucuresy 1%
-90
Accursy Phi
MECAPAY
70.1
200-
Selected Groups
Matching Accuracy
-80
1.00
200
4
5
Basetnn
Constantt
Ascand
Threhoodd
Thrwshodd
Figure 2: Left and middle: Matching results across different thresholds Winoground and SugarCrepe (the Replace Relation subset) with SigLIP-B16 Right: Performance of TTM under different threshold schedules on Winoground with SigLIP-B16. Baseline denotes model performance without TTM (under GroupMatch). Constant applies TTM with a fixed threshold = 2.0. Ascend applies TTM with linearly increasing schedule from 5 0 to 7/ 2.0, but yields no gains as the model quickly overfits to all pseudo-labels in the first iteration. Decay applies TTM with a linearly decreasing schedule from т 2.0 7r = 0, yielding the best performance.
overly strict selection criteria. In matching- based aseudo-labeling, a lower threshold increases the number of selected pseudo-labels but typically reduces precision (more false positives), whereas a higher threshold yields cleaner but fewer pseudo-labels (more false negatives). This trade-off is illustrated the left and middle plots of Fig. 2, which report the number of matched groups (blue) and the accuracy among matched groups (orange) as function of T.
TTM employs a decaying threshold schedule (Tr+1 TT) to balance these errors over iterations: begins with a high threshold to ensure high-precision pseudo- (few false positives), and then gradually lowers the threshold increase coverage and reduce false negatives as the model improves. In contrast, an ascending schedule can admit substantial false positives in early rounds, which derails learning and limits subsequent improvement; a constant schedule avoids early false positives by keeping the threshold high, but induces many false negatives in later iterations, leading to limited gains and early plateauing. The right plot of Fig. supports this intuition: the decaying schedule consistently outperforms the alternatives.
In practice, we find it effective to set the initial threshold ti such that roughly 15%-30% of the groups matched, and the final threshold Tr such that more than 90% of the test set is covered. Both cosine and linear decay schedules perform well. Further analyses and ablations are provided in Section 4.5.
Runtime analysis. The runtime of TTM scales O(T Cn), where T is the number of iterations and Cn denotes the per-iteration model finetuning cost. Empirically, TTM demonstrates strong improvements even with small number of iterations (e.g., T is comparable to standard test-time training methods in the literature (Sun et al., 2020; Akyürek et al., 2025). See Appendix C.3 for detailed analysis and discussion.
10). Thus, the runtime of TTM
3.2.1 TEST- TIME MATCHING WITHOUT GROUP STRUCTURES
While Algorithm 1 is designed for datasets organized into local groups, the same principle extends naturally to settings without any predefined group structure. In this case, we treat the entire dataset as single global matching problem between all images and all captions.
Let S, denote the set of images and Se: the set of captions. We assume |Sy S [Sol and each image has a unique corresponding caption (one-to-one assignment). Let E R ( be the similarity matrix produced by a model f. We consider all injective matchings क : S + So from images to captions. The model-induced global matching is then defined as
Σ $i,m(()- iES;
/ arg max
Sc
which maximizes the total similarity assignment problem. which can be efficiently solved by strongly-polyyone time algorithms such as the Hungarian algorithm (Kuhn, 1955).
pairs. Eq. (2) corresponds the classical

---

Published as a conference paper at ICLR 2026
Analogous to Algorithm adopt an iterative schedule with pseudo-labeling. At iteration t, let f be the global matching induced by model fi-1- Because the entire dataset is treated as a single group, group-level margin thresholding loses granularity: the model would either accept all matches or none. To address this, we apply thresholding at the level of individual pairs. Specifically, the pseudo-label at iteration is
S = (f,, - (0) 2 Tz):
where T is the threshold iteration t. The threshold can be set either as an absolute value or relative to the distribution of similarity scores (i.e., the p-th percentile). Following the same principle as in Algorithm I, we begin with a relatively high threshold to ensure high-precision pseudo-labels and gradually decay it over iterations to expand coverage and bootstrap performance over the test set.
4 EXPERIMENTS
We describe experimental setups in Section 4.1, present main results in Sections 4.2 to 4.4, and provide analyses and ablations in Section 4.5. Additional experimental details and results are deferred to Appendix C.
4.1 EXPERIMENTAL SETUPS
Datasets. We evaluate five challenging compositional reasoning benchmarks: Winoground (Thrush et al., 2022), MMVP-VLM (Tong et al., 2024), Colorswap (Burapacheep et al., 2024), SugarCrepe (Hsieh al., 2023), and WhatsUp (Kamath al., 2023). Winoground, MMVP-VLM. and Colorswap consist of 2 x 2 groups; we also construct their non- non-grouped variants by discarding group structures (Section 3.2.1). SugarCrepe consists of 2 groups and Wh Wp consists of x groups; we evaluate on 4 different subsets of SugarCrepe and all 2 subsets of WpatsUp. Following Li al. (2025), we further convert WpatsUp into 4 different variants with 2 x groups. In total, our evaluation spans 16 dataset variations covering diverse structures and evaluation settings.
Models. We test both contrastive vision-language models and multimodal large language models. For contrastive models, we use SigLIP (Zhai et al., 2023) and CLIP (Radford et al., 2021) at multiple scales, including SigLIP-B16, SigI.IP-L.16, CLIP-B16 and CLIP-B32, For multimodal large language models, G.T-4.1, where image-text similarity is computed based on VQAScore (Lin et al., 2024).
Evaluation metrics. For GPT-4.1, we report raw GroupScore and GroupMatch-induced perfor- mance via SimpleMatch (Section 3.1). For CLIPs and SigLIPs, we additionally include results with TTM (Algorithm 1). Specifically: on 2 x 2 datasets report (i) raw GroupScore, (ii) GroupMatch- induced performance, and (iii) TTM-boosted performance; on 1 x k datasets we report (i) raw GroupScore and (ii) TTM-boosted performance, since GroupScore and GroupMatch coincide in this case; and on datasets without group structures we report (i) raw GroupScore (with known groups), (ii) global assignment accuracy under Eq. (2), and (ii) TTM-bbosted performance via the global variant introduced in Section 3. In all cases, we highlight performance gains from TTM over over Grtch for 2 x datasets, over GroupScore for x & datasets, and over global assignment accuracy under Eq. (2) for datasets without group structures. All results are averaged over four random runs, with standard deviations reported.
4.2 TTM ACHIEVES NEW SOTAS
We evaluate on three established compositional reasoning benchmarks- Winnuurousd,dddddddd MMVP- VLM, and ColorSwap- all consisting of 2 x 2 groups and considered challenging for frontier AI models. Previous state-of-the-an results include 58.75 on Winoground (GPT-4V with prompt tuning (Wu et al., 2023; Vaishnav & Tammet, 2025)), 70.7 on MMVP (via a GPT-40 multi-agent system with tool use (Zhang et al., 2024c)), 4 and 87.33 on ColorSwap without training-set access (95.33 with finetuning on the training set (Burapacheep et al., 2024)).
'This result is on MMVP. variant of MMVP-VLLL formulated as binary-choice question answering. In this paper, we focus MMVP-VIM, which is better suited for contrastive models. Prior work has shown that model performance on the two variants positively correlated (Tong et al., 2024; Li et al., 2025).
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance on Winoground, MMVP-VLM, and ColorSwap. Raw model performance is reported under GroupScore. SimpleMatch corresponds to the performance under GroupMatch (Section 3.1), and TTM corresponds the performance of Algorithm 1. We report absolute gains (A), relative gains, and relative error reductions of TTM over SimpleMatch. Cells highlighted in indicate results obtained with TTM, while cells
denote the SOTA performance for each dataset.
Dataset Model
Raw
SimpleMatch
TTM
A
Error Red.
Winoground
GPT-4.1
69. 75 0.56
91.38 0.80
CLIP-B16
7.25
60.00
65.44 : 1.1
+3.4
(9.1% 1)
13.6% +
SigLIP-B16
10.25
67.00
72.50 1 0.64
+5.5
(8.2% 1)
16.7 %
SigL.I--L16
13.00
69.50
72.75 1 0.64
+3.3
(4.7% %
10.7 %
MMVP-VLM
GPT-4.1
68. 0.00
88.52 1: 0.83 72.59
CLIP-B16
5.19
80.19 0.81
+ +7.6
(10.5% % f) (9.8 1)
27.7 % + 43.0% +
SigLIP-B16
22.96
81.48
89.44 = 0.96
+ +8.0
ColorSwap
GPT-4.1
91.08 0.28
97.42 0.14
CLIP-B16
12.00
77.67
85.75 : 0.64
+8.1
(10.4% f)
36.2 % +
SigLIP-B16
30.33
88.00
94.25 1 0.43
6.3
(7.1 % 1)
52.1 %
SigLIP-L16
37.00
91.33
96.08 + 0.43
+ 4.8
(5.2% 1)
54.8%
Simple matching reveals hidden capabilities. Applying SimpleMatch (Section 3.1) to CLIP, SigLIP, and GPT-4.1 already yields striking improvements (Table 1). SimpleMatch enables SigLIP-B16 to surpass all prior state-of-the-arn results without access to additional data, and enables GPT-4.1 to set new records across all three benchmarks. Notably, GPT-4.1 improves from 69.75 to 91.38 on Winoground, yielding the first result surpass the estimated human performance of 85.5 (Thrush et al., 2022). These findings confirm that the GroupMatch metric can reveal substantial hidden compositional reasoning capabilities.
Test-time matching further boosts performance. We next apply TTM (Algorithm 1) to CLIP and SigLIP, enabling additional performance gains without external supervision. As shown in Table 1, TTM consistently improves over SimpleMatch across datasets and model scales, with relative gains up to 10.5% and relative error reduction up to 54.8%.² Crucially, TTM elevates SigLIP-L16 to the level of GPT-4.1 on ColorSwap and enables SigLIP-B16 to surpass GPT-4.1 on MMVP-VLN, establishing a new state of the art. These results demonstrate that TTM is powerful and practical approach for enhancing model performance through self-improvement at test time.
4.3 TTM IMPROVES MODELS WITHOUT METRIC-INDUCER BOOSTS
To evaluate the effectiveness of Algorithm beyond cases where alternative metries can inflate per- formance, we consider benchmarks with 1 x k group structure, where GroupScore and GroupMatch coincide and thus provide no metric-induced boost.
We experiment on 4 SugarCrepe subsets (1 2 groups) and all 2 WhatsUp subsets (1 x 4 groups), reporting results in Fig. 3. Even without metric-induced gains, Algorithm I consistently delivers substantial test-time improvements. The gains are especially striking on the Whats Up datasets, where performance improves by up to 85.7 %. turning these previously challenging tasks into tractable ones.
Following Li al. (2025), we further convert the WhatsUp datasets into directional variants with 2 x 2 group structures. As shown in Table 8 (in Appendix C.2), Algorithm again yields significant improvements up 135.1 % relative gains and 95.5 relative error reduction-on on top of
'While the absolute boosts may appear modest compared to GroupMatch-inducee gains, they are highly significant: for comparison, scaffolding GPT-4V yields only 1.25-point gain on the Winoground dataset, improving performance from 50.75 (Zhang al., 2024a) to 52 (Vaishnav & Tammet, 2025).
8

---

Published as a conference paper at ICLR 2026
SugarCrepe (1x2
WhatsUp (1x4 groups)
TTM (Ours)
Raw
89.0
60
(+85,7%)
83.7
Accuracy
80
50
70.5
71.5
66.1
30.6
30.9
60
30
Replace Rel
Attr
Swap
Attr
WhatsUp
WhatsUp
SigLIP-B16
Figure 3: TTM results on benchmarks without metric- boosts: for 1 x k groups, GroupMatch (and thus SimpleMatch) coincide with GroupScore. Left: results on four SugarCrepe subsets consist- ing of x 2 groups. Middle: results on both WhatsUp subsets consisting of x groups.
Table 2: Performance on non-grouped variants of Winoground, MMVP-VLM, and ColorSwap. Raw model performance reported under GroupScore, SimpleMatch corresponds to the performance of global assignment defined in Eq. (2), and TTM corresponds to the performance of the global variant of Algorithm 1. We report absolute gains (A), relative gains, and relative error reduction of TTM over SimpleMatch.
Datasets
SigLIP-B16
SimpleMatch
+ TTM
A
Error Red.
Winoground 10.25
44.38
46.78 1.05
+2.4
(5.4% f)
4.3 %
MMVP-VLM
22.96
39.63
44.54 2.02
+4.9
(12.4% % f)
8.1 %
ColorSwap 30.33
88.00
92.00 1: 1.24
+4.0
(4.5% % f)
33.3% |
SimpleMatch. Together, these results demonstrate that TTM is broadly effective across both l x k and groups, even when metric-induced effects are absent, as in the case of x k groups.
4.4 TTM IMPROVES MODELS WITHOUT GROUP STRUCTURES
To further assess the generality of Algorithm 1, we evaluate its global variant introduced in Section 3.2.1 on datasets without any predefined group structures. Specifically, we flatten Winoground, MMVP-VLM, and ColorSwap by removing local k x k groups, resulting in general dataset with an image set Sy and a caption set
We report three metrics: (i) raw GroupScore (with the extra knowledge of the group structure), (ii) global assignment accuracy obtained via SimpleMatch under Eq. (2), and (iii) TTM-boosted performance achieved using the global variant introduced in Section 3.2.1. Results show that even global assignment without group structures substantially outperforms the vanilla GroupScore, demonstrating the effectiveness of using matching-based supervision generate high-quality pseudo- labels. More importantly, applying the iterative global TTM algorithm yields further gains over global assignment alone, with especially large relative error reductions on ColorSwap. i.e., 33.3% relative error reduction on ColorSwap (see Table 2). This demonstrates that the test-time matching principle generalizes effectively beyond group-structured datasets.
4.5 ANALYSES AND ABLATIONS
Group matching provides strong supervision signals. The key advantage of GroupMatch over GroupScore lies matching and group structure, we examine the raw performance of CLIP-B16 and SigLIP-B16 under different evaluation metrics. addition to GroupScore and GroupMatch, we consider (i) global matching under Eq. (2), which performs matching but ignores group structure, and (ii) individual matching within groups, which preserves group structure but doesn't 't perform matching: assign captions to images independently within the group. As shown in the left plot of Fig. 4, GroupMatch provides the strongest supervision signal among all metrics, making it most effective for guiding pseudo-labeling.
ability leverage matching within local groups. To assess the benefits of

---

Published as a conference paper at ICLR 2026
Row Mode: Ferfermance under Horest Werris
CLINS15
87
TTM with Drado Hareing
71
TIN wen Deffrent
SigLIN915
800
27.0
TTM
Jccoracy 2
TTM Oradi
35.0
human Performance
2.5 -18% coverage
2.0, -21% coverage
GroapScore Guza Matco
Ind Ruach
GroAp
Iaration
Figure 4: Left: Raw performance CLIP-BI6 and SigLIP-B16 on Winoground under different evaluation metrics. Middle: Skyline performance of TTM with oracle matching on Winoground with SigLIP-BI6, illustrating the upper bound achievable by TTM. Right: Effect of the initial threshold on TTM performance, evaluated on Winoground with SigLIP-B16.
Skyline performance with oracle matching. To study the full potential of TTM, we evaluate an oracle variant that incorporates pseudo-labels into S, if and only if they are correct (i.c., with oracle access). As shown in the middle plot of Fig. 4, this oracle variant enables TTM bootstrap more aggressively, approaching human-level performance on Winoground. This suggests that improving pscudo-label polenialp through the incorporation of external supervision- could further enhance the effectiveness of TTM.
Threshold selection for TTM. As discussed in Section 3.2, we adopt decaying threshold schedule that begins with high-quality pseudo-labels and gradually expands coverage. In our experiments. the final threshold Ty is set to either 0 (full coverage) or 0.1 (typically covering more than 90% of the data). The optimal initial threshold Ti is more dataset- and model-dependent. If training or hold-out split is available, can be selected based on matching results on that data (c.g.. see the left and middle plots of Fig. 2). Otherwise, we find effective to set FΙ such that roughly 15%-30% of the groups are matched initially. The right plot of Fig. 4 shows TTM results on Winoground with SigL.P--B16 for 71 E {2.5. 2, 1.5], corresponding to roughly (18% 21%, 30%} initial coverage. While performance varies slightly across these choices, all yield consistent gains, highlighting that TTM robustly improves model performance at test time. For the global matching variant, we find effective set тi such that about 50% of the data pseudo-labeled initially. See Appendix C. for further discussion and complete hyperparameter settings used in our experiments.
5 DISCUSSION
This work revisits the long-standing puzzle of compositional reasoning, where modern multimodal models often appear to perform no better than random guessing (Thrush et al., 2022; Diwan et al., 2022; Tong et al., 2024; Burapacheep al., 2024; Li al., 2025). We show that this apparent limitation partly arises from the evaluation metrics themselves, which systematically underestimate model capability. We introduce GroupMatch as a metric correction that yields a more faithful evaluation and can be translated back to the standard metric via simple overfitting step; under this correction, GPT-4.1 surpasses estimated human performance on Winoground. Building on this insight, we propose Test-Time Matching (TTM), an iterative, self-improving algorithm that further bootstraps model performance without external supervision. TTM enables SigLIP-BII to outperform GPT-4.1 AMVP-VLM establishing new state of the art. Experiments across 16 dataset variants demonstrate that TTM consistently improves performance across diverse settings, including those without inetric-induced effects or predefined group structures.
Moving forward, we highlight two promising directions:
Recalibrating model evaluation. The same model on the same dataset can yield vastly different results under different metrics. This underscores the need for more robust, transparent, and reliable evaluation protocols for compositional reasoning and beyond (Schaeffer al., 2023).
Extending TTM beyond compositional reasoning. While developed in the context of composi- tional reasoning, the core principle of TTM- iterative, matching- based self-training at test time is general. A natural next step is explore this idea in broader multimodal language settings.
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
YZ conceived the project, developed the algorithms, performed the majority of the implementation and experiments, and wrole the manuscript. JZ and FT assisted with the implementation; JZ additionally conducted experiments on the WhatsUp datasets.
ACKNOWLEDGMENTS
YZ acknowledges support from NSF IIS IIS-2425006.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08/1 087744 2023.
Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning. In Forty-second International Conference on Machine Learning, 2025.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. Artificial intelligence review, 11(1::11-73, 1997.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What required and can it be learned? In International Conference on Learning Representations, 2019.
Maria-Flood MariiaFFloriaa Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference on Computational Learning Theory, pp. 35-50. Springer. 2007.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural computation, 466):888-90,, 1992.
Jirayu Burapacheep. Ishan Gaur, Agam Bhatia, and Tristan Thrush. Colorswap: A color and word order dataset for multimodal evaluation. arXiv preprint arXiv:2402.04480 2024.
Rui M Castro and Robert D Nowak. Minimax bounds for active learning. International Conference on Computational Learning Theory, pp. 5-19. Springer, 2007.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et
al., eds.; (b0o) reviews). IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 295-305, 2022.
François Chollet. On the measure of intelligence. arXiv preprint arXiv: 1911:11111.0044: 2019.
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: Technical report. arXiv preprint 24Xiv:2442..00 1240404 2024.
William S Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American statistical association, 74(368):829-836 1979.
William S Cleveland and Susan Devlin. Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American statistical association, 83(403):596-610 1988.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Sanjoy Dasgupta, Adam Tauman Kalai, and Adam Tauman. Analysis of perceptron-based active learning. Journal of Machine Learning Research, 10(2), 2009.
11

---

Published as a conference paper at ICLR 2026
Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. arXiv preprint wtXi0:221..0076. 2022.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoen- coders. Advances in Neural Information Processing Systems, 05:29374-29388. 2022.
Steve Hanneke. Theory of active learning. Foundations and Trends in Machine Learning, 7(2-3),
2014.
Moritz Hardt and Yu Sun. Test-time training nearest neighbors for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Cheng-Yu Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36:31096-311 31116, 2023.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022.
Jonas Hübotter, Sascha Bongni, Ido Hakimi, and Andreas Krause. Efficiently learning at test- time: Active tine-tuning of LLMs. In The Thirteenth International Conference on Learning Representations, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with vision-language models? investigating their struggle with spatial reasoning. arXiv preprint arXiv:2310. 19785, 2023.
Akshay Krishnamurthy, Alckh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. Journal of Machine Learning Research, 20(65)11-50, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self self-training for gradual domain adaptation. In International conference on machine learning, pp. 5468-548644 -448.. PMLR, 2020.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Siting Li, Pang Wei Koh, and Simon Shaolei Du Exploring how generative milms perceive more than clip with the same vision encoder. Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics, 2025.
Zhiqiu Lin, Deepak Pathak. Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366-384. Springer, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. In Conference learning theory, pp. 3806-3832. PMLR, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8763-8763. PmLR, 2021.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? Advances in neural information processing systems, 36:55565-55588 2023.
12

---

Published as a conference paper at ICLR 2026
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi cemi-supervised learning with consistency and confidence. Advances in neural information processing systems, 13:596-608, 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt, Test-time training with superviiirv for generalization under distribution shifts. In International conference on machine learning, pp. 2229.9248. PMLR, 2020.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Can- dace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 9568-9578, 2024.
Mohit Vaishnav and Tanel Tammet. A cognitive paradigm approach probe the perception-reasoning interface in vlms. arXiv preprint arXiv: 13620, 2025.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
Yifan Wu, Pengchuan Zhang, Wenhan Xiong, Barlas Oguz, James C Gee, and Yixin Nie. The role of chain-of-thouu of-thhught in complex vision-language reasoning task. arXiv preprint 1XXX::231100999: 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in neural information processing systems, 34: 18488-148811,, 2021.
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain tf-touhh prompting for large multimodal models with multiple image inputs. arXiv preprint ar arXiv:2401.02556 2024a.
Jifan Zhang, Yifang Chen, Gregory Canal, Arnav Mohanty Das, Gantavya Bhatt, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Shaolei Du, Kevin Jamieson, and Robert Nowak. Labelbench: A comprehensive framework for benchmarking adaptive label-efficient learning. Journal of Data- catarcentric Machine Learning Research, 2024b.
Zhehao Zhang, Ryan Rossi, Tong Yu, Franck Dernoncourt, Ruiyi Zhang, Jiuxiang Gu, Sungchul Kim, Xiang Chen, Zichao Wang, and Nedim Lipka. Vipact: Visual-perception enhancement via specialized vlm agent collaboration and tool-use. arXiv preprint arXiv:2410. 16400, 2024c.
Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
Yinglun Zhu and Robert Nowak. Active learning with neural networks: Insights from nonparametric statistics. Advances in Neural Information Processing Systems, 35:142-155, 2022a.
Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. Advances in Neural Information Processing Systems, 35:35379-35391 2022h.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Compositional reasoning and evaluation metrics. Contrastive vision-language models (VLMs) such CLIP (Radford al., 2021) and SigI.IP (Zhai et al., 2023), and multimodal large language models (MLLMs) such as the GPT (Achiam et al., 2023; Hurst et al., 2024) and Gemini (Team et al., 2023; Comanici al., 2025) series, have achieved remarkable progress across wide range of multimodal tasks. Yet both VLMs and MLLMs struggle on benchmarks specifically designed to test compositional reasoning the ability to systematically combine objects, attributes, and relations to interpret or reason about novel configurations (Lake et al., 2017: Bahdanau et al., 2019; Thrush et al., 2022: Hsieh et al.. 2023; Kamath et al., 2023; Tong et al., 2024; Burapacheep et al., 2024). These benchmarks are typically organized into small groups of images and captions that differ in subtle but systematic ways (e.g., captions with identical words but different orderings). The prevailing evaluation metric, the GroupScore, requires models to correctly assign each image to its corresponding caption and cach caption to its corresponding image via isolated pairwise comparisons. While rigorous, this metric is also unforgiving: raw model performance often falls at below random guessing (Thrush et al., 2022; Diwan et al., 2022; Tong et al., 2024; Burapacheep et al., 2024; Li al., 2025).
Despite recent attempts to improve compositional reasoning in frontier multimodal models (Wu al., 2023; Zhang al., 2024c; Vaishnav & Tammet, 2025), progress remains modest. For instance, the previous state of the art on innoground-ccien
by scaffolding and prompt tuning GPT-4V (Wu et al., 2023: Vaishnav & Tammet. 2025) was only 58.75, still well below the estimated human performance of 85.5 (Thrush et al., 2022).
Our work takes a complementary perspective to prior efforts by revisiting the evaluation metrics used in compositional reasoning. We introduce group matching score (GroupMatch) that evaluates the best overall matching rather than isolated pairwise comparisons, revealing substantial hidden capability in both VLMs and MLLMs. Crucially, by simply overfitting to the induced matchings at test time, this hidden capability transfers into higher scores under the original GroupScore, closing much of the reported gap. With this adjustment, GPT-4.1 improves from 69.75 to 91.38 on Winoground- yielding the first result to surpass the estimated human performance of 85.5. This finding echoes broader observations that measured capability be highly sensitive the choice of evaluation metric (Schaeffer et al., 2023), underscoring the need for continued research on evaluation protocols for frontier models.
Test-time training, pseudo-labeling, and adaptive schedules. Test-time training adapts models during inference to improve performance, with roots in early work on local learning and instance- specific adaptation (Cleveland, 1979; Cleveland & Devlin, 1988; Bottou & Vapnik, 1992: Atkeson et al., 1997). The idea has regained attention in the era of large pretrained models, where test-time self-supervision can enhance performance without additional labeled data (Sun et al., 2020; Wang et al., 2020; Gandelsman et al., 2022; Chen et al., 2022). Recent studies show that finetuning on retrieved data based on test prompts can significantly improve large language models (Hardt & Sun, 2024; Hübotter al., 2025), and test-time training has become key component in tackling reasoning-heavy benchmarks such as ARC (Chollet, 2019; Chollet et al., 2024; Akyürek et al., 2025).
Our test-time matching algorithm (TTM) shares this motivation but differs in key aspects. Most prior methods adapt to each test instance independently, producing per-instance finetuned models and often relying on instance-specific in-context examples (Akyürek et al., 2025). In contrast, TTM leverages GooupMatch-nnduced pseudo-labels labels across the entire test set, iteratively updating a single model through an adaptive thresholding schedule. This connects naturally to the literature on self-training (Kumar et al., 2020) and semi-supervised learning (Zhu, 2005; Chapelle et al., 2009; Sohn et al., 2020; Zhang et al., 2021, 2024b), where pseudo-labels drive improvements. A central contribution of our approach is to exploit matching and group structure- both locally (Section 3.2) and globally (Section 3.2. 1) to generate high-quality labeds-labels.
Finally, our adaptive thresholding schedule resonates with classical ideas in active learning (Castro & Nowak, 2007: Balcan et al., 2007: Dasgupta et al., 2009; Hanneke, 2014; Krishnamurthy et al., 2019; Puchkin & Zhivotovskiy, 2021; Zhu & Nowak, 2022a.b), though with reversed logic: whereas active learning typically queries the most uncertain examples for human annotation, our approach begins with the most confident pseudo-labels and gradually relaxes thresholds to expand coverage. This
14

---

Published as a conference paper at ICLR 2026
confidence- first perspective is central to the effectiveness of TTM, enabling consistent performance gains without any external supervision.
B PROOFS AND SUPPORTING RESULTS FROM SECTION
B. PROOFS OF PROPOSITION AND PROPOSITION 2
Proposition 1. For random similarity scores s E R P(GroupScore(s) =
(k-1))
(2R-1)1*
Proof. Because the entries of s are i..d. sampled from continuous distribution (here unif 1])), ties occur with probability 0, so we may use strict inequalities throughout.
Denote d; := sü and, for i * j. set myj == min d, )- By the definition of the GroupScore, the event (GroupScore(s) = }} is equivalent to requiring 8ij < mij and Sjz < my for every 3 j. Conditioning on the diagonal =
and using independence of the off-diagonal entries,
IP( ((rroupScore(s)
d) II P(sy <m;)) (8j+ < may) icj
II m
< be the order statisties of (d1,...,dk)) We then have mij T Tmin (r()) where r(-) is the rank, hence
Let 0 < σ1 <
II m i<j
II
2(k
2(k-a)
Since (.1,..., .T.) are the order statistics of i.i.d. unif( 1)) samples, their joint density is k! on the ordered region {0 21 < ... < I < 1} (and O elsewhere). Therefore.
P(GroupScore( =1 = k! /
Il n
2(k-a) dxi dak-
Jocer a=1
For < k and ye E (0, 1]. define
le(z) = J Jocnemecnes
8
II
2(k-a) a) dx ... dæg.
x
We claim that, for l = I....,k.
y ((2k-()
Iz(1)
IT r(2k - r)
This is proved by induction on For 1=
1" x 2k- dx
y22k
2k:
Assume it holds for e- - 1. Then
Ie(v) 10¹ xt.
1 2(k-() 1) I (xz) dxe
11 r(2k r) / Ze.
(1--)))
dre
y ((26-1)
π (-1 r(2k - r) 1(2k E)*
since 2(k: 1) + (e 1)(k (e 1)) l(2k: 9) 1. Thus the claim holds. Taking l k and
= 1 gives
S
2(k-a) dx dxg [+()) x
1
I r(2k r)
a=l
15

---

Published as a conference paper at ICLR 2026
Therefore,
P(GroupScore(s) =1) k! 1 r(2k r)
(k 1)! (2k 1)!
Proposition 2. For random similarity scores R P(GroupMatch(() -) E!
Proof. There are k! distinct injective matchings. Since the random variables {8ig) are continuous, ties occur with probability 0. By symmetry, each injective matching is equally likely to achieve the maximum total similarity. Hence, the probability that the ground-trute matching x² attains the maximum is X
B.2 SUPPORTING RESULTS FOR GENERAL RECTANGULAR GROUPS
Without loss of generality, we consider a group of m images (7,) and captions {C. with k. We assume the ground-truth pairings is {(4, C,) (hidden from the learner). As in the main text, we study a random guessing model that assigns i.i.d. similarity scores 8i := sim 4, C2) ~ ([0 1]) for each pair (45r Cý). and collect them into a similarity matrix & Rmxk
GroupScore for m x groups. Analogous the k k and le cases, the GroupScore for groups can be defined as
Vi [m] 86 maxjØi 8ij.
GroupScore(s))
otherwise.
Under random guessing model, the probability of achieving a GroupScore for rectangular group is given below.
Proposition random similarity score s R.nk PP(GroupScoress) = g
Proof. Since the random variables {st are continuous, ties occur with probability For each row i, by symmetry, the probability that 84 is the largest among the A: i.i.d. entries {8ig [ is 1/k. Since rows are independent, we have
P(Vi E [m] 85t
max s.y) j+t
km
GroupMatch for m k groups. We extend GroupMatch to the general rectangular case by consid- ering injective matchings π [m] + [k] (i.e., <1)) + (jj for i j). With the ground-truth injective matching π" i i, we define GroupMatch as
1 if Em (1, Em St,(()) VT#T",
GroupMatch(s) ==
0 otherwise.
Under the random guessing model, the probability of achieving a GroupMatch of for rectangular group is given below.
Proposition 4. For random similarity scores s Rmnkk, IP(GroupMatch(3)
(k-m)!
Proof. There are TT-myt distinct injective matchings. Since the random variables {siÿ) are continu- ous, ties oceur with probability 0. By symmetry, each injective matching is equally likely achieve the maximum total similarity. Hence, the probability that the ground-truth matching x attains the maximum is
k!
&!
(k-m)
(p-m))
16

---

Published as a conference paper at ICLR 2026
GroupMatch helps for rectangular groups.
For random similarity scores R
P(GroupMatch(s) = =1) (k -mm!
1 k! k(K 1).--) + km
1
P(GroupScore(() 1),
with strict inequality for any 2 2 and equality for (GroupMatch and GroupScore coincide when m = 1). Moreover, if the ground-truth injective matching x* identified, overfitting to the matching x* test time guarantees GroupScore of 1. Thus, as in the square case, can improve model performance under GroupScore via SimpleMatch: (i) selecting the most likely matching under GroupMatch and (ii) overfitting the matching test time transfer gains.
C OTHER DETAILS FOR EXPERIMENTS
C.I ADDITIONAL DETAILS AND HYPERFARAMETIE
We provide additional experimental details and hyperparameter settings below. For TTM. we set the number of iterations to T = 10 and train for 20 epochs per iteration by default, except on Winoground where we train for 30 epochs per iteration. Across all experiments, we use Adam (Loshchilov Hutter, 2017) with weight decay 0.05 and ($1, 52) (0.9, 0.999). The learning rate follows a cosine decay schedule and is restarted at each iteration with a multiplicative factor of 0.95. Optimizer states are reset at each restart, with the exception of SigLIP-BB - 16 on Winoground. We use a batch size of 50 for 2 x datasets and 100 for groups of size 2 x per batch).
datasets; the batch size defined at the group level (e.g., 50
By default, we do not apply data augmentation during training, as many datasets are designed to be sensitive to location color. However, find it beneficial to apply simple resizing (factor 1.1) followed by random cropping for the following dataset-mode pairs: Winoground with SigLIP-L16, MMVP-VLM with SigLIP-BII Colp with SigLIP-B16, MMVP-VLM with SigLIP-B16 under global matching, and CLIP-B32 with WhatsUp A-Left-Right.
In Tables 3 to 5, report, for each dataset-model pair, the initial threshold mı, the final threshold Tr, the threshold decay schedule (linear or cosine), and the learning rate (Ir). For group matching (Tables and 4), we use absolute thresholds. For global matching, we adopt the percentile-based thresholding mentioned in Section 3.2.1: at iteration é, the top - ti fraction of pseudo-labels (ranked by similarity) is selected.
In our experiments, the final threshold Tr is set to either 0 (full coverage) or 0.1 (typically covering more than 90% of the data). The initial threshold 7₃ more dataset- and medeeeddeenndnnt For group matching, we find it effective to set such that roughly 15%-30% of the groups are matched initially, though some cases we use thresholds outside this range when they yield better performance (e.g.. higher selection fractions for ColorSwan wap with SigLIP models and lower fractions for WhatsUp 2 x 2 variants with CLIP-B32), For global matching, performance tends to improve with a larger initial selection fraction-typically around 50%.
Table 3: Hyperparameters used for experiments in Section 4.2.
Dataset
Model
T1
Tr
Schedule
Ir
CLIP-B16
0.9
linear
2.0 x 10-5
Winoground
SigLIP-B16
2.0
linear
1.0 x 10
SigLIP-L16
2.0
0.1
cosine
4.0 x 10-5
CLIP-B16
2.0
linear
1.0 x 10-5
MMVP-VLM
SigLIP-B16
2.0
0.1
cosine
2.0 x 10-5
CLIP-B16
2.3
cosine
4.0 x 10
ColorSwap
Sig
cosine
4.0 10-5
SigLIP-L11
2.5
cosine
4.0 x 10-5
WW slightly increase the batch size when the total number of groups is just above multiple of the default size. For instance, if the dataset contains 102 groups, we the batch size to 51.
17

---

Published as a conference paper at ICLR 2026
Table 4: Hyperparameters used for experiments in Section 4.3.
Variant
Model
71
7T
Schedule
Ir
Replace Relation
SigLIP-B16
2.1
0
cosine
1.0 x 10-5
Swap Attribute
SigLIP-B16
1.8
0
cosine
1.0 x 10-5
Swap Object
SigLIP-B16
2.0
0
cosine
1.0 x 10-5
Add Attribute
SigLIP-B16
2.5
0
cosine
1.0 x 10-5
WhatsUp (1x4)
CLIP-B32
0.55
0
linear
x 10-5
WhatsUp B (1x4)
CLIP-B32
0.80
0
linear
1.0 x 10-5
A- A-Left-Right (2x2)
CLIP-B32
0.25
0
linear
1.0 x 10-5
A- A-On-Under (2x2)
CLIP-B32
0.85
0
linear
1.0 10-5
B-I B-Left-Right (2x2)
CLIP--33
0.50
0
cosine
2.0 x 10-5
B- B-Front-Behing (2x2)
CLIP-B32
1.30
0
cosine 2.0 x 10-5
Table Hyperparameters used for experiments in Section 4.4. We adopt percentile- threshold-
ing: at iteration t, the top 1- - fraction of pseudo-labels (ranked by similarity) is selected.
Dataset
Model
TT
Schedule
Winoground MMVP-VLM
SigLIP-B16
0.50
0
linear
1.0 x 10 10-5
SigLIP-B16
0.55
linear
2.0 x 10 10-5
ColorSwap
SigLIP-B16
0.50
linear
4.0 10-5
C.2 COMPLETE RESULTS FROM SECTION 4.3
We present complete empirical results for Fig. 3 below in Tables and 7. Following Li al. (2025), we further convert the WhatsUp datasets into four directional variants with 2 x group structures and present results in Table 8: Algorithm again yields significant improvements up to 135.1 % relative gains and 95.5% relative error reduction- top of SimpleMatch. Together, these results demonstrate that TTM is broadly effective across both k k and settings, even in cases where evaluation metrics themselves cannot induce gains.
Table 6: Performance on SugarCrepe datasets (1 groups) without metric-induced boosts: for 1 x k groups, GroupScore and GroupMatch coincide. Raw SigLIP-BIL performance is reported under GroupScore, and TTM corresponds the performance of Algorithm We report absolute gains (A), relative gains, and relative error reductions of TTM over the raw model performance.
Datasets
SigLIP-BIc
TTM
A
Error Reduction
Replace Relation
70.48
76.23 0.51
+5,8
(8.2%
19.5%
Swap Attribute
71.47
77.36 0.71
+5.9
(8.2% f)
20.6 |
Swap Object
60.41
66.12 1:2.6
+5.7
(9.5%
14.4%
Add Attribute
83.67
88.95 0.83
+5.3
(6.3% f)
32.3% |
C.3 RUNTIME ANALYSIS AND EFFECTIVENESS OF TTM UNDER A SMALL NUMBER OF ITERATIONS
Runtime analysis. The runtime of TTM scales O(T Cit,, where T is the number of iterations and Cj denotes the model finetuning cost. While TTM also requires selecting the maximum maximum-similarii similarity matching within each group, this cost is dominated by the finetuning cost. Specifically:
The per-group matching cost P(k) = O(k2) for algorithm) or P(k) O(k) for 1 groups. Since compositional benchmarks very small group sizes (k for groups or & 4 for 1 x & groups), we can safely treat P(k) = 0(1).
groups (via the Hungarian
18

---

Published as a conference paper at ICLR 2026
Table Performance on WhatsUp A/B datasets (1 4 groups) without metric-induced boosts: for 1 x l groups, GroupScore and GroupMatch coincide. Raw CLIP-B32 performance is reported under GroupScore, and TTM corresponds the performance of Algorithm We report absolute gains (A), relative gains, and relative error reductions of TTM over the raw model performance.
Datasets
CLIP-B32
TTM
A
Error Reduction
WhatsUp A WhatsUp B
30.58
56.8
184
+26.2
(85.7% 1)
37.7% % +
30.88
49.94 2.58
19.1
(61.7% f)
27.6 % +
Table 8: Performance on WhatsUp directional variants: LR: left-right, OU: on-under; FB: front-behind. Raw CLIP-B32 performance is reported under GroupScore. SimpleMatch corresponds the performance under GroupMatch (Section 3.1), and TTM corresponds the performance of Algorithm 1, We report absolute gains (Á), relative gains, and relative eroo reductions of TTM over SimpleMatch.
Datasets
CLIP-B32
SimpleMatch
TTM
A
Error Reduction
A-LR
40.78
95.87 = 4.42
+55.1
(135.1% % 1)
93.0% %
A-OU
3.88
78.64
99.03
+ +0.4
(25.9% f)
95.5 %
B-LR
0
55.88
82.84 1: 0.49
+7700
(48.2% f)
61.1 %
B-FB
0
47.06
66.67 =
19.6
(41.7% % 1)
37.0% %
Matching over all groups therefore costs (n)), which is dominated by the model finetuning cost Ct-
Thus, the overall runtime is dominated by finetuning and scales O(T Cit). In practice, the finetuning cost Cn can be further reduced via efficient finetuning techniques (Hu al., 2022).
Effectiveness of TTM under A small number of iterations. While we use T = 10 in our main
experiments with contrastive vision-language models, we also evaluate TTM with = 3 and T
As shown Table 9, TTM continues to yield substantial gains even with only T 3 T iterations.
Table 9: Performance on Winoground, MMVP-VLM, and ColorSwap under varying number of iterations (T). Raw model performance reported under GroupScore. SimpleMatch corresponds to the performance under GroupMatch (Section 3. and TTM corresponds to the performance of Algorithm We report absolute gains (A), relative gains, and relative error reductions of TTM over SimpleMatch.
Dataset (Iterations)
Raw
SimpleMatch
TTM
A
Error Red.
Winoground
SigLIP-B16 (T = =3
10.25
67.00
71.50 1.19
+ +4,5 (6.7% 1)
13.6 % 1
SigLIP-B16 (T 5)
10.25
67.00
71.88
1.48
(7.9% 1)
14.8% % +
SigLIP-B16 (T' 10)
10.25
67.00
72.50 0.64
(8.5%
16.7% % +
MMVP-VLM
SigLIP-B16 (T= =3)
22.96
81.48
85.19 0.91
(4.7 1)
20.0% 4
SigLIP-B16 (T ==:5 =5)
22.96
81.48
87.04 +: 1.11
+ (6.6 1)
30.0% %
Sig (T =10)
22.96
81.48
89.44 0.96
8.0 (..0%
43.0% 4
ColorSwap
SigLIP-B16 (T =
30.33
88.00
93.58
(..6 1)
46.5% +
SigLIP-B16 (T'5))
30.33
88.00
93.58 + 0.86
+ (6.6 1)
46.5 %
SigLIP-B16 (T' 30.33
88.00
94.25 0.43
+ 6..3
52.1 %
19

---

Published as a conference paper at ICLR 2026
D THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMss were used to polish the writing of this paper.
20


Reviews:
[{"review_id": "9Uj7L4FBf7", "reviewer": "Reviewer_QzQa", "summary": "This paper argues that multimodal models' poor performance on compositional reasoning benchmarks is partly due to flawed evaluation metrics that underestimate their true capabilities. The authors first introduce a new evaluation metric, the \"group matching score\" (GroupMatch), which evaluates the best overall matching within a group and uncovers substantial \"hidden competence\" in existing models. Building on this, they propose Test-Time Matching (TTM), an iterative, supervision-free self-training algorithm that bootstraps model performance by finetuning on high-confidence pseudo-labels generated at test time. This approach achieves new state-of-the-art results, notably enabling GPT-4.1 to surpass estimated human performance on Winoground and SigLIP-B16 with TTM to outperform GPT-4.1 on MMVP-VLM.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. One strength of the paper is its insightful critique of existing evaluation metrics, such as GroupScore, which it convincingly argues systematically underestimate model capabilities in compositional reasoning. The introduction of the \"GroupMatch\" score provides a simple, intuitive, and more effective alternative that reveals significant \"hidden competence\" in existing models.\n\n2. The proposed Test-Time Matching (TTM) algorithm is an effective supervision-free method that iteratively bootstraps model performance. This self-training approach, which uses high-confidence pseudo-labels at test time, provides a practical way to unlock the latent abilities identified by the new metric.\n\n3. The paper presents extensive and strong empirical results across 16 dataset variants, achieving new state-of-the-art (SOTA) performance. Notably, the SimpleMatch method allows GPT-4.1 to surpass estimated human performance on Winoground for the first time, and TTM enables a smaller model (SigLIP-B16) to outperform GPT-4.1 on MMVP-VLM.", "weaknesses": "1. The TTM algorithm relies on iterative finetuning at test time, which is computationally expensive and time-consuming. Discussion about the trade-off between compute and performance is needed. For example, compare a small model with more iterations, and a large model with fewer iterations.\n\n2. The performance of TTM appears sensitive to the choice of new hyperparameters, particularly the threshold scheduling used for pseudo-label selection. The paper notes that the initial threshold is \"dataset- and model-dependent\" and requires careful setting, which may make the method difficult to apply robustly to new datasets or models without costly tuning.\n\n3. GPT-4.1 is closed-sourced. It is necessary to conduct experiments on an open-sourced MLLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s10CYw6RW8", "reviewer": "Reviewer_smg6", "summary": "This paper revisits the problem of compositional reasoning in multimodal models, which are often reported to perform near random chance on benchmarks such as Winoground. The authors first identify that existing evaluation metrics (notably GroupScore) systematically underestimate model capability. They propose a new metric, the Group Matching Score (GroupMatch), which evaluates the best overall image–caption matching within each group. This new metric reveals hidden competence in both contrastive VLMs (e.g., CLIP, SigLIP) and MLLMs (e.g., GPT-4.1), enabling GPT-4.1 to surpass estimated human performance on Winoground. Building on this, they introduce Test-Time Matching (TTM), an iterative, supervision-free self-training algorithm that converts confident model-induced matchings into pseudo-labels, progressively bootstrapping model performance during inference. TTM yields further non-trivial improvements—e.g., SigLIP-B16 surpasses GPT-4.1 on MMVP-VLM—achieving new state-of-the-art results across 16 benchmark variants, including datasets with and without group structures. In summary, the paper’s key contributions are: (1) redefining evaluation through GroupMatch to uncover hidden compositional ability; (2) proposing TTM for test-time self-bootstrapping without external supervision; and (3) demonstrating broad applicability and new SOTAs across both grouped and non-grouped multimodal reasoning benchmarks", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper reframe multimodal compositional reasoning evaluation. By identifying that existing metrics systematically underestimate model ability, it introduces the GroupMatch score as a new evaluation paradigm that better reflects true multimodal alignment. Moreover, the proposed Test-Time Matching (TTM) algorithm creatively extends test-time training and self-training principles to multimodal evaluation without external supervision. This combination of metric reformulation and unsupervised adaptation constitutes a distinctly original contribution in the field. The derivation of probabilistic properties of existing and proposed metrics offers analytical clarity on why previous benchmarks yield misleadingly low scores. The empirical evaluations are comprehensive—covering 16 datasets and multiple model families (CLIP, SigLIP, GPT-4.1)—and are backed by detailed ablations, sensitivity analyses, and comparisons. The results are consistent and reproducible, indicating high experimental quality.", "weaknesses": "1. The SimpleMatch and TTM procedures explicitly overfit to test data using pseudo-labels derived from the model’s own predictions. Although the paper positions this as “test-time adaptation,” it effectively tunes on test distributions, which may blur the boundary between evaluation and training. The authors should clarify the implications of this for benchmark comparability and fairness, and discuss possible countermeasures such as validation-based stopping criteria or out-of-distribution robustness checks.\n\n2. Although the paper covers 16 dataset variants, most are derived from similar group-based compositional reasoning benchmarks (Winoground, MMVP-VLM, Colorswap, SugarCrepe, Whatsup). It remains unclear whether TTM generalizes to broader multimodal tasks such as retrieval, captioning, or visual reasoning benchmarks like ARO or GQA. Testing beyond compositional group setups would strengthen claims about general applicability.\n\n3. While the paper presents sensitivity studies on threshold decay, it does not deeply analyze the types of errors propagated during pseudo-labeling or how inaccurate matchings affect subsequent iterations. An empirical study on pseudo-label noise, or visualization of incorrect matchings and their impact, could make the bootstrapping dynamics more interpretable and trustworthy.\n\n4. The iterative self-training procedure (ten iterations, multiple epochs each) likely increases inference and adaptation time significantly. The paper would benefit from reporting computational overheads, memory costs, and convergence behavior, as well as exploring lightweight or one-shot alternatives to make TTM practical for real-world applications.\n\n5. The paper would benefit from a more formal characterization of when and why TTM converges, or how pseudo-label noise affects performance over iterations. Connections to existing theories in semi-supervised or test-time training (e.g., Sun et al., 2020; Gandelsman et al., 2022; Akyürek et al., 2024) could strengthen the methodological depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BOFo8eQJgD", "reviewer": "Reviewer_4Q99", "summary": "This paper proposes a more relaxed and losser evaluation metric, GroupMatch, for multimodal compositional reasoning and an iterative test-time finetuning method (TTM). Instead of modifying model architectures, the authors show that changing the metric from GroupScore to GroupMatch can boost the performance of existing models. Also, the author claim that TTM further boosts performance through the pseudo-label–based training of model parameters at test time.\n\n**For me, I highly believe this work is lower than the bar of ICLR.** The reasons are one-by-one listed in weaknesses.", "questions": "1. Does the method generalize beyond image-text tasks to other modalities (e.g., audio-text, video-text)?\n\n2. Does GroupMatch truly measure reasoning ability, or just alignment under a softer rule?\n\n3. Can models cheat the metric GroupMatch without improving their underlying reasoning as it is more looser and relaxed?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors propose a simple but effective TTM method with impressive empirical gains.\n2. The authors claim that the results achieved by the method surpass human performance on Winoground.", "weaknesses": "1. **Lack of clear rationale and rigorousness.** The motivation behind using a looser metric is not fully clear. The authors should give more explaniation for the connection between the relaxed metric GroupMatch and the real reasoning ability of models. Does GroupMatch truly measure reasoning ability, or just use a looser rule to make results look good? I still believe that the standrad strict metric is more rigorous than the GroupMatch desinged in this paer.\n\n2. **Risk of inflation.** Because GroupMatch is easier to satisfy, models may appear much stronger without any actual improvement in reasoning. This could distort comparisons across methods or benchmarks. I indeed do not believe using a looser evaluation metric is absolutely better than the use of a strict metric. \n\n3. **Lack of theoretical grounding.** The paper claims that GroupMatch “reveals hidden capabilities,” but provides limited formal justification or analysis on why this metric better reflects reasoning.\n\n4. **Lack of theoretical grounding.** The paper could provide more explicit analysis of the runtime/efficiency implications of TTM, especially for large-scale deployments.\n\n5. **Limited robustness analysis.** There is no clear evaluation of how sensitive GroupMatch is to noise, adversarial perturbations, or different group sizes. If the metric is fragile, its reliability as an evaluation tool is questionable.\n\n6. **Unconvincing results.** Since TTM explicitly uses the test data for adaptation, and GroupMatch relaxes the matching requirement. So, the authors should report the performance improvment brought by TTM using the original stric GroupScore, which makes the restults more solid.\n\n7. **Poor practical applicability.** TTM need to finetue the model' parameters on test data, which has worse practical applicability in real-world scenarios, e.g., test data may be scarce and models may be very large (such as GPT-4,5, and Gemini).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvJwdAH5tK", "reviewer": "Reviewer_NrUV", "summary": "This paper proposes Test-Time Matching (TTM), a method that enhances compositional reasoning in multimodal models by redefining evaluation metrics (GroupMatch) and applying unsupervised test-time adaptation. While the idea of leveraging matching structures is interesting, the improvements mainly come from metric relaxation rather than genuine reasoning gains. The proposed TTM is conceptually similar to prior test-time adaptation methods, incurs high computational cost, and shows limited practicality beyond small benchmarks.", "questions": "1. Since much of the reported improvement comes from the new GroupMatch metric, does the proposed method still yield significant gains under conventional metrics? The dependence on the new metric raises concerns about fair comparison.\n\n2. TTM requires iterative matching and fine-tuning during inference. How does this impact efficiency compared to standard zero-shot evaluation? A runtime analysis would be helpful.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important problem in multimodal compositional reasoning and highlights biases in existing evaluation metrics.\n\n2. The proposed GroupMatch metric and Test-Time Matching (TTM) framework offer an interesting perspective on improving model performance without additional supervision.", "weaknesses": "1. The paper mainly improves evaluation metrics (GroupMatch) rather than actual model reasoning ability. The performance gains may stem from relaxed scoring rather than genuine compositional understanding.\n\n2. The Test-Time Matching (TTM) algorithm is conceptually similar to existing test-time adaptation methods (e.g., TENT, AdaContrast). The contribution appears incremental rather than fundamentally new.\n\n3. TTM requires iterative matching and fine-tuning during inference, which is computationally expensive and unsuitable for real-world deployment or online systems.\n\n4. The reported improvements are mostly on small datasets (e.g., Winoground). It is unclear whether the method generalizes to larger or more realistic multimodal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)