The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs
Authors: Hong Li
Nanxi Li
Yuanjie Chen
Jianbin Zhu
Qinlu Guo
Cewu Lu
Yong-Lu Li
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-modal LLM
Visual Reasoning
Association
Primary area: datasets and benchmarks
TL;DR: In this paper, we benchmark MLLM's ability on association tasks at various semantic concepts based on an annotation-free association reconstructed method.

Abstract: Multi-modal Large Language Models (MLLMs) have exhibited impressive capability. However, recently many deficiencies of MLLMs have been found compared to human intelligence, $\textit{e.g.}$, hallucination. To drive the MLLMs study, the community dedicated efforts to building larger benchmarks with complex tasks. In this paper, we propose benchmarking an essential but usually overlooked intelligence: $\textbf{association}$, a human's basic capability to link observation and prior practice memory. To comprehensively investigate MLLM's performance on the association, we formulate the association task and devise a standard benchmark based on adjective and verb semantic concepts. Instead of costly data annotation and curation, we propose a convenient $\textbf{annotation-free}$ construction method transforming the general dataset for our association tasks. Simultaneously, we devise a rigorous data refinement process to eliminate confusion in the raw dataset. Building on this database, we establish three levels of association tasks: single-step, synchronous, and asynchronous associations. Moreover, we conduct a comprehensive investigation into the MLLMs' zero-shot association capabilities, addressing multiple dimensions, including three distinct memory strategies, both open-source and closed-source MLLMs, cutting-edge Mixture-of-Experts (MoE) models, and the involvement of human experts. Our systematic investigation shows that current open-source MLLMs consistently exhibit poor capability in our association tasks, even the currently state-of-the-art GPT-4V(vision) also has a significant gap compared to humans. We believe our benchmark would pave the way for future MLLM studies.  $\textit{Our data and code are available at:} https://mvig-rhos.com/llm_inception.

Published as a conference paper at ICLR 2025
THE LABYRINTH OF LINKS: NAVIGATING THE Asso- CIATIVE MAZE OF MULTI-MODAL LLMS
Hong Li, Nanxi Li, Yuanjie Chen, Jianbin Zhu, Qinlu Guo, Cewu Lu, Yong-Lu Li' Shanghai Jiao Tong University, Shanghai Innovation Institute {hong 1f, andyc 03, ey 2200033, bfnn pi g, guoq! n y 11)8sttu.ee jteeeuu...ducco
ABSTRACT
Multi-modal Large Language Models (MLLMs) have exhibited impressive capa- bility. However, recently many deficiencies of MILMSs have been found com- pared to human intelligence, e.g., hallucination. To drive the MLLMs study, the community dedicated efforts building larger benchmarks with complex tasks. In this paper, propose benchmarking essential but usually overlooked in- telligence: association, human's basic capability to link observation and prior practice memory. To comprehensively investigate MLLM's association perfor- mance, we formulate the association task and devise a standard benchmark based adjective and verb semantic concepts. Instead of costly data annotation and curation, we propose a convenient annotation-free construction method trans- forming the general dataset for our association tasks. Simultaneously, vise a rigorous data refinement process to eliminate confusion in the raw dataset. Building on this database, we establish three levels of association tasks: single- step. synchronous, and asynchronous associations. Moreover, we conduct com- prehensive investigation into the MILMs' zero-shot association capabilities, ad- dressing multiple dimensions, including three distinct memory strategies, both open-source and closed-source MILMMs, cutting-edge Mixture-of-Experts (MoF) models, and the involvement of human experts. Our systematic investigation shows that current open-source MLLMs consistently exhibit poor capability in our association tasks, even the currently state-of-the-art GPT-4V(vision) also has a significant gap compared to humans. We believe our benchmark would pave the way for future MLLM studies. Our data and code available at: https: /ttp:///viiiiii www.iggrhos... com/ 11m wwt.///mmmnnettioooooooooo
de-
INTRODUCTION
Multi-modal Large Language Models (MLLMs) have recently made significant breakthroughs in perceiving diverse modality input and solving a broad range tasks Zhang et al. (2024a); Carolan ct al. (2024). As (PT-4V(ision) Achiam et al. 2023) and Gemini [eam et al. (2023): Reid et al.
(2024) address challenges that researchers have been exploring for a considerable period. Subse- quently, numerous researchers have developed diverse open-source MLLMs Al et al. (0024) Bai cet al. (023b)); Wang et al. (2024b); Dong et al. (0024) Liu et al. (2023a): Li et al. (2024a) Ye el al. (2023 2024). These models usually use the Large Language Model (LLM) the core component and expand to multi-modal with specific module Yin et al. (2023) that transfers multi-modal tokens into language tokens, achieving alignment between different modality encoders.
MLLMs demonstrated ability in visual reasoning, which requires understanding the input query and then making judgments based on the visual content. Much prior work has been dedicated to evaluating the levels of their visual reasoning capabilities. However, to the best of our knowledge, how to evaluate the association ability of MLLMs overlooked. Association one of the most fundamental capabilities of human intelligence. It provides the foundation for creative thinking and helps humans to summarise scattered information into structured knowledge to enhance memory and understanding Mednick (1662) Ausubell (1963), perception, rule discovery, embodied AI, etc.
"Correspondence Yong-Lu Li <yonglu li@sju.edu.en..

---

Published as a conference paper at ICLR 2025
Jenticn canditite may howe Brved with the quen??
Association
Human .M
Instructin:: (nsorens)
(frisc olin Qursstin lost sction))
182.3 140.2
Imaged
plane
Image2. (V 56S0_
Close-source
step
stivtrch ssoodation? srribse?))
Gemini 1.5 Flash
OPT-NV
Introction)
88.3 27.8
59.0 15.4
Imagge
Open-source
Update memorr
Mot 5.99
oe toley?
aa pilce suriing
Sequential association
Association
(fuuc slen dumiton instructio))
Obonvitoe
Quary Enage
6688 3.94
1.:00
V O1 ULAVA
Imago2. (X printassoosstep) Stopp)
Afternaon Chat
MLLM Power Ranking (Max Mean Step)
a) Association question
bj Practical association task
Figure 1: Our insight and proposed association task. a) Association is always in our lives. b) Our proposed practical association task. c) The performance of current MLLMS and human experts. The results demonstrate a significant gap between current MLLMs and humans in association tasks.
In this paper, we aim to devise standard benchmark to evaluate MLLM's capability in association tasks. As shown in [a. we always connect observations with previous practice memory. This association is usually the semantic concept shared between different objects, representing common features in general dataset samples. For instance, the "painted airplane" and "painted bus" share the same attribute of "painted", which can be seen association chain in the association task. Next, is natural to ask can we design a general method to build an association benchmark based on existing datasets without much extra effort? To this end, we propose annotation-free associ- ation construction method, which easily transfers the general dataset for the association task. As in Figure Db. we devise standard association benchmark using the object's concepts as an association chain. Given the benchmark, we comprehensively investigate MLLMs' capabilities in association tasks across multiple memory strategies, multiple open-soures and closed-source MLLMs. well as sutting-edge -fxturr-of-Expertss (MoE) and human expert testers.
Extensive experiments demonstrate that current open-source MILMS consistently have weak abil- ity in association tasks, and even the closed-souree source GPT4-V Achiam et al. (2023) and Gemini-1.55 Flash Reid et al. (004)) also have a huge gap from human performance. As Figure shows, the best closed-source Gemini-1.5 1.5-Flash attains an average mean-step of 27.8, while humans attain 140.2 in the attribute of adjective concepts. As a complementary of existing benchmarks, we believe our benchmark and baselines will pave the way for human- like intelligent agents.
In conclusion, our main contributions are: 1) To evaluate the association ability of MLLMs, we pro- pose multi multi-modal task and corresponding convenient annotation-free construction method that can transform the general dataset for association tasks within various semantic concepts. 2) We devise standard association benchmark based adjective semantic concepts within objects and verb semantic concepts within actions and further evaluate MLLMs' performance through exten- sive experiments. 3) We systematically investigate MLLMS' capability on association tasks through tuning-free methods and propose potential future directions for association tasks.
2 RELATED WORKS
Multi-Modal Visual Reasoning. Various works are dedicated to understanding and reasoning about the semantic concept between multi-images. Several works, such as Visual Genome Krishna et al. (017)) and HOI et al. (2022), target human-object interaction (HOI) tasks to investi- gate the visual relationship between different objects. Recent work Zhang et al. (2024b) investigates the MLLM's ability in low-level perception question-answering and description tasks with paired input images. Despite their success in perceiving and understanding multi- multi-images, these methods confined single-step evaluation, lacking the investigation of multi-step association ability.
Multi-Modal Large Language Model. There has been surge of MLLMs in the deep learning community, which off the-the-shell LLMSs support multi-modal inputs and demonstrate promise
2

---

Published as a conference paper at ICLR 2025
for zero-shot generalization. For instance, LLa Liu et al. (2023a) InstructBLIPP Dai et al. (2033 Dong et al. (2024). Owen-VL. Bai et al. (2023a). MiniGPT-4 Zhu et al.
2023), and mPLUG-Owl2 Ye al. (2023) make leading in last year according to their powerful visual perception capabilities. More recently, these models Wang et al. (2024b); Li et al. (202a);;
Ye et al. (2024) are updated to improve their ability in multi-images or video input. In this work, we mainly evaluated the leading open-source models Qwen2-VL Wang et al. (2024b), mPLUG-
Ow13 Ye et al. (2024), and AaaaVA-OneVissa Li al. (2024a).
Tuning-Free Engincering. The development of MLLMs has led to a significant improvement in in- context learning Doveh et al. (2024): WWes al. (2024). Additionally, the use of visual and language prompts has been shown to enhance the performance of MLLMs. Research has indicated that visual cues such as listed numbers Yan al. (2024) and shapes (Shtedritski Shaees/Shterritkii et al. (2023): Mani et al. (2020) can be effectively understood by MLLMs. Similarly, carefully designed language prompts Liu et al. (2023b), Chain-of-Tie -hain-of-Thhught (CoT))We et. (2024). tree-of-mixed-tie
(ToMT)Hu et al. (2023),
have shown promising results. Common knowledge SSaa al. (2023); Liu al. (2024a) can be seen as an instruction of Prior knowledge about how to make decisions on a task. As a result, we chose
to employ the general prompt-engineer one-shot, CoT, and Common knowledge.
3 ANNOTATION-FREE ASSOCIATION CONSTRUCTION
For a general dataset with N samples, it can be formed as { Wi) Ix E X, (cI,...,,,, CnJ. 1,2,....NV, ... Let with individual modality, the sample is described as
= (x:"h) ...., x2 (1) ) be sample with & modalities, more specifically, for task
(z,"", .). is the annotation for sample I,, which may existed in various granularities. These annotations indicate a semantic concept, with object categories as nouns, actions as verbs, as well as attributes, and affordance as adjectives.
Given this definition, labels represent subset of link concepts present in the given sample. The core of human association, on the other hand, involves identifying the overlapping concepts between newly acquired observations and prior practice memory. Hence, intuitive to use annotations from the raw dataset to construct the association chain, which reflects the common concepts shared by two samples, such as the presence of the "shoot ball" action both samples. In the following, we first generate possible chains for the association task, then create the ground-truth labyrinth to reason the chain behind the association.
Generating Semantic Concept Association Chain. Association connects objects by any poten- tial links. We defined associations existing when any pre-defined semantic concept appears between objects. To this end, for general dataset {(x;, ys) x; X,y; (cI...., Co. ), = 2, ..., N). we randomly select two samples to form a new sample pair for association. If the selected samples have identical labels or share at least one common label, we assign a corresponding label of 1; otherwise, we assign a label of O. Hence, we get the new association dataset as:
(1,2 2, E X, <i<j S N}, where žij {
if ya Ny + otherwise
(1)
In this way, for each sample x; in the original dataset, we construct a positive association set with K positive samples, at .......p'. J. where at least one potential association concept exists. Additionally, devise another negative association set with L samples, σ; .q....../f. g.1,. in which no predefined association chain present.
lp!....,
Deducting the Evidence of Association Chain. Associations implicit links connect ob- jects, which make correct decisions with unspecified links. To uncover the labyrinth, we devise deduction steps that reason the association links after association. This evidence will fused into memory and influence subsequent steps. To realize this, we collected the full set of shared con- cepts within all possible positive set .र. as Ô = U {zi Zij = y: O yj. 1 < i < S N). In this setting, the dataset deducting the evidence of the association chain is depicted as:
{ pi s:), x., P. X,p E at 8
/ c CC.
where p represents the correctly predicted positive sample at the stage of chain association, and 8; (s!...., R denote the R common concepts.
3

---

Published as a conference paper at ICLR 2025
Evaluation of the Association Task. Based on the above definition, we can easily construct the association task. First, we randomly select one sample, quuary as the initial query image. Then, for each step We randomly choose candidate samples P and of from the positive Equery and nega- tive queryy sets, respectively, Next, we deliver new observations and previous practice memory into MLLMS determining which candidate shares common concepts with the query, the output of t-th step described O₁. If the model makes the correct decision, Or = pi, the query and correctly predicted images will be used to deduct the chain of association that updates memory. Then convert to the next association step, the correctly predicted samples taken as the query image and repeat- edly select candidate samples to conduct association and deduction progress. If make an error, we calculate the maximum association steps and exit. The above description can be depicted as:
[1 + step(F(xquen) (uurryy> (query) ))) and Zquery = pt if 0g = =p; 44
t= =tep
otherwise (output / and exit), (3)
where Equery E { ØN} and F represents the forward computation of MLLMs. step is the computation of combined steps of association and deduction.
4 ASSOCIATION BENCHMARK
In this section, we introduce a benchmark based on the semantic concepts of adjectives and verbs, iee, object attributes and affordances, and human actions. Specifically, we utilize the annotation- free construction method proposed Section 3 on the Object Concept Learning (OCL) Li et al. (023) generate attribute and affordance association datasets, and on the Pangea Li et al. (024c) to generate action association dataset In the following, we comprehensively investigate the association ability from the single-step association, synchronous association, and asynchronous association settings as shown in Figure
4.1
CONSTRUCTING ASSOCIATION TASK
4.1.1 SINGLE-STEP ASSOCIATION
The association refers to the link between the current observation and prior practice memory (Fig- ure (p). A single-step association represents one phase within a broader association task, where memory is indispensable decision-making late prior practice, guiding the decision-making process. In the experiment, we compute the associ- ation and deduction success ratio as the main single-step association metric (Section 5.1.2)
this case, we include the correct memory to simu-
4.1.2 SYNCHRONOUS ASSOCIATION
Synchronous association, where each step adheres to the same principle throughout the entire pro- cess, is a core capability of human intelligence (Figure With this ability, humans can gradually unveil the underlying rules of the task and reduce the likelihood of errors. Utilizing our constructed association dataset, we take different input-pairos samples with the same association concepts & to evaluate the synchronous association. In this setting, the association dataset can be depicted as:
{1 if è C {yx n
{ (1,, Ziz) D,, E X,15 < < where
(4)
0
if yf Nツ, 9
It is worth noting that current MLLMs lack memory during the inference stage, relying solely on the input samples. address this problem, we introduce memory base to imitate the human's memory in the synchronous association. Specifically, we transfer the inference process into the memory base after each step. The updated memory and the input sample are then used for the next step. In the experiment, compute the Max Mean steps metric to evaluate the model (Section 5.1.2
4.1.3 ASYNCHRONOUS ASSOCIATION
When there are multiple principles in games, the underlying principle will gradually change as the game progresses. For example, the first steps with "metal" the chain, then "furry" and again
'To further demonstrate the capability on verb concept, further implemented on action HMDB Kuchne 201 datasets. The results included in the section of the supplementary material.
4

---

Published as a conference paper at ICLR 2025
b) Synchronous Association
a) Single-step Association
fresh
fresh
fressh
fresh
fresh
Asynchronous Association
Candicate Image1 Candidate Image2
Choose the related one
metal
Figure 2: Semantic figure of association question. (Left) Single-step association with a fixed correct memory. (Right) Synchronous and asynchronous association with dynamic memory. Synchronous involves one category in the chain, while asynchronous improves complexity with two categories.
"metal", constituting loop strategy (Figure 23). In this case, we need call back the memory from the previous step instead of the last step. Similar to the synchronous association, we utilize the association dataset for the asynchronous association setting by selecting multiple shared concepts between different input samples. We formulate the dataset as:
(1,,, T, E X,i < =, N), where 2ig {
if CT Eyr (25) if y ny =0
(5)
where (m indicates m shared concepts selected from C. In addition, we also utilize the memory base that the strategy the same as the synchronous association for the asynchronous association.
4.2 DATA REFINEMENT FOR ASSOCIATION TASK
Depending on annotation- association construction methods, we transfer the general dataset for the association task. The dataset has paired input samples and is labeled with whether they have common concepts for the association step. Furthermore, paired input samples with shared concepts were created for the deduction step. While these support the challenges in association tasks, there is still a possibility of confusing samples. To address this, we introduce a data refinement method.
We implemented three-step strategy ensure data quality, including an Image resolution filter, MLLM verification, and Human expert evaluation. Specifically, the Image resolution filter screened all images with less than 50, 000 pixels to ensure superior visual quality. The MLLM verifica- tion takes a question-answer answer strategy with OpenAl's GPT4-V V Achiam et al. (2023) and Google's Gemini-1.5-Fiie Reid et al. (2024) to ensure each annotation of raw data exists the image. Then, the human expert evaluation is conducted through our custom-designed interface, enabling testers to complete the association task and eliminate low-quality samples or those with ethical concerns
Our benchmarks are implemented in adjective concepts and verb concepts, which include attribute and affordance in OCI, et al.] (2023) and action Pangea [i al.] (2024c). In the OCL dataset, selected eight attributes with good perception performance, such as "metal, ripe, fresh, natural, cooked, painted, rusty, furry", and eight affordances "sit, imprint, push, carry, cut, clean, open, break".. In addition, selected eight actions of "run, hit, drive, dress, cooking, build, shake, cut".
4.3 BASELINE FOR ASSOCIATION
There are various methods improve the concept perception. Our focus lies in exploring tuning- free methods, which harness the inherent capabilities of the model. To this end, we employ popular prompt engineers to improve the understanding of MLLMs, including common knowledge (Com- Know), one-shot, and chain-of-thoure (CoT).
We compare the performance with and without the MILM verification in the section G of supplementary. 'The detailed visualization and description refer Subsection A.1 in the supplementary.
5

---

Published as a conference paper at ICLR 2025
For the memory base in association, introduce three different memory strategies that transfer each inference process into memory after each association step. including structure memory (StructM), natural language memory (NLM), and chain memory (ChainM), StructM and NLM both simulate human memory, differing only in their descriptive approaches. They incorporate memory attention mechanisms to determine whether to reinforce forget memories. Specifically, The underlying strategy of memory base is that if one type of memory knowledge my appears in evidence at one step, we add repetition weight w, the attention weight W; otherwise, the attention weight decays, i.e,, subtracts forgetting decrement dy. This strategy is described as:
W ma + Wy, mf E evidence (m,, df:
W ma
otherwise
StructM uses raw structure memory as input, while NLM transforms it into descriptions that are more aligned with human language. Additionally, ChainM is more closely aligned with the task, as it stores the previous inference process as a chain. In the ChainM strategy, each inference process is treated as a subchain, represented as objl > concept - > obj2, and is concatenated into the prior memory. The detailed description and practice examples refer to section A.2 in the supplementary.
EXPERIMENTS
5.1 SETTINGS
5.1.1
IMPLEMENTATION DETAILS
We systematically conduct experiments that include concept perception, single-step, synchronous, and asynchronous association. Specifically, we implement the task as a multi-choice setting in all experiments, taking one query image and two candidate images with one correct as input and output correct image index. Based on this, we first devise preliminary concept perceptions that involve popular prompt engineering skills on open-source MLLMs to investigate perception capabilities in attribute concept. Then, we convert to evaluate MILM's 's association capabilities that make de- cisions based on current observation and prior practice memory, i.e., input with additional content of previous practice. We develop single, synchronous, and asynchronous associations according to fixed or dynamic memory. The single-step association means the model decides with observation and correct prior practice. Meanwhile, the synchronous and asynchronous association model a dynamic task that iteratively arrives at a decision and then deducts the underlying evidence, which means the memory may have wrong information for the next judge.
For single-step, synchronous, and asynchronous association, we design three types of memory bases, i.e,, Structure Memory (StructM), Natural Language Memory (NLM), and Chain Memory (ChainM). In addition, we involve the baseline of No Memory (NoM) which means determining at a dynamic setting without the memory base. For the detailed description of the type of memory base and the usage in the prompt, please refer to the section A.2 in the supplementary.
We utilize three open-source MLLMs in preliminary concept perception: QWen-VL Bai et al. (2023)). LLaVA-NeXT-7B Liu et al. (2024a), and LLaVA-NeXT-12 For formal association, we utilize three new-versions MLLMs that break through the MLLM's capabilities in multi-imager LLaVA-OneVision Li et al. (2024a). QWen2-VL, [ang et al. (2024b), and mPL.UG-OW13 Ye et al. (024). Besides, we evaluate the performance of Mixture-of Mfxture-of-Fipertt (MoE) that combined three open-source MI..Ms. In experiments, open-source MILM is run on a single NVIDIA A100 80G GPU. Apart from open-source MLLMs, we include the evaluation of the closed-source MLLMs of GPT4-V Achiam GPT4-V/cchiam the results of three human experts to demonstrate the gap between MLLM with human intelligence.
al. (2023) and Gemini-1.5-Fian Reid et al. (2024). Simultaneously, we involve
5.1.2 METRICS FOR ASSOCIATION TASK
Max Mean Step. In an association task, max-step indicates the maximum number of steps in one round of association, i.e., the maximum length of correctly predicted association chain. While mean-step refers to the average maximum association step across multi-rounds of association tests.
FFor the detailed description of the setting of concept perception, please refer section D

---

Published as a conference paper at ICLR 2025
Models (Success Ratin)
Type
Concept
Memory
L.Naa-CheViiio
QWen2-V1.
mPI.DG--Ow1
Gemini-1.5 1--Pash
GPT-4o
Avg.
NoM
75.52
76.36
54,05
75.40
84.49
73.16
Astribute
StructM
78.41
T7.74
72.67
87.30
88.83
80.39
(Adjective)
NI
80.40
56.63
73.33
$6.30
89.87
83.72
ChainM
77.73
Association Success Ratio
Affondance
StracMM
NoM
70.61
73.30
75.10
75.39
73-02
53.50
17.78
84.45
84.93
85.36
61.25
72. 44
(Adjective)
NLM
76.13
79.37
67.79
85.40
86.76
79.10
ChainM
74.38
62.07
68.46
80.02
$1.54
77.40
75.74
57.94
Action
StructM
75.41
52.10
73.92
86.
58.72
81 /B6
(Verb)
NUM
78.06
89.58
86.13
82.48
ChainM
76.92
65.50
69.50
87.58
85.90
81.10
Success Rutio
Deduction
Affordance (Adjective)
Autribute (Adjective)
Action (Verh)
21.07
46.61
49.38
58.11
15.98
49.90
42.80
33.30
61.21
33.08
65.82
55.69
78.14
57.28
27.30
62.53
50.00
26.15
Table 1: Mean success ratio of each concept on single-step association with four memory strategies across open-source and close-source MLLM. The best and second results of each concept are shown in bold and underline, respectively. For detailed results on each category refer to Table 5
Success Ratio. For concept perception and single-step association, suppose the total number of samples of each concept 6 is T; and the correctly judged based on the shared concept is 7 samples.
For any association and deduction step, the success ratio concept é is defined r.] T
T,
5.2 RELATIONSIIIP BETWEEN PERCEPTION AND ASSOCIATION
In this subsection, investigate the MLLM's ability in attribute concept perception that is im- plemented on the OCL Li et al. (2023). We devise multiple task complexity, including individual perception and combination perception. The detailed description and results shown in section D in the supplementary. These results indicate that while popular prompt engineering skills improve performance in specific cases, i.e., prompt engineering skills compared to Normal, they still exhibit limited capabilities in the widely-used object understanding benchmark. For instance, in the indi- vidual perception, the highest perception success ratio of 0.690 reflects an improvement of 0.190 over the random baseline of 0.5. Furthermore, the combination perception attains the success ratio of 0.542, improving 0.292 over the random baseline of 0.25.
The association is capability built upon foundational perception abilities. Hence, this prompts to reduce the complexity perception and disentangle the evaluation perception from association. In practice, selected categories with good perception performance for association tasks. This enables a more rigorous assessment the MLLM's associative capabilities.
5.3 RESULT OF ASSOCIATION
Based on the finding of concept perception, we convert to evaluate MLLM's association capabil- ity. In the following subsection, we first access MLLM's ability in the single-step association that makes decisions with observation and correctly fixed memory. Then, evaluate its associative capabilities in a dynamic setting, with evidence derived from the MILM's deduction. We further investigate the efficacy of MI.MMs in straightforward synchronous association encompassing only one semantic concept within the association chain. Furthermore, we increase the task complexity to attain asynchronous association by involving two semantic concepts in the association chain.
5.3. SINGLE- STEP ASSOCIATION
Table shows the mean result in adjective and verb concepts, i.e., attribute and affordance on the OCL et al.] (2023) and verb on the Pangea Li al. (2024c). We focus on the metric of "success ratio" in this part, and the detailed result each item can be found in supplementary Table 5 6
The result shows that GPT4-o achieves the highest performance adjectives and Gemini-1.55Flaas makes advances in verbs, but those all remain certain gap from humans, i.e., success ratio com- pared to 1. It deserves noted that humans are unlikely errors as association links are given in context. Moreover, comprehensive data refinement ensures the data quality that prevents any errors in- duced by annotations confusion. Besides, the association has certain improvements across various concepts, iee,, the proposed memory strategy is higher than NOM. Simultaneously, the NLM at-
7

---

Published as a conference paper at ICLR 2025
Atribute Synchronous Association
Affondence Synchrenous Association
Action Synchronous Association
150-
400
100
140
140
Dverage the
120-
120
60-
SICR
FLUG-ON
Figure 3: The average Max Mcan step on the individual concept synchronous association across three open-source MLLM and humans. Different columns within each MLLM indicate different memory strategies. For detailed results on each category refer to Table g in the supplementary.
Asynchronous association with paired categories across different concepts
Attribute
Affordance (Moan
(Mean Stop!
30
Pwerage Moon Shop
$6"
25-
140.4
Aucrage Mean
Average
46'0
StructM
MLN
Chaint
NaM
StricM
MUN
Chart
StructM
MUN
Charn
Actr iuttllttte (Max-Stupl
Alfordance Stup)
Action (Mas-SSupp
200
200
70-
Aieeraee Mov. Stop
Avcrape: Stop
162.7
60.3
40
10
40.1
Stn.ctM
Chaine
StructM
MiN
Charri
Structt
Chaine
Figure 4: The average Max Mean step of asynchronous association with paired categories across different concepts. The upper subfigure mean-step, while the lower subfigure is max-step. The detailed results each different category group refer Table
12 T3 in the supplementary.
tained the strongest performance, we speculate that due to the current MILM being trained on large amount of internet visual language data, lacking structured data training.
5.3.2 SYNCHRONOUS ASSOCIATION
The primary focus of our design revolves around the stability of MLLMs in synchronous association within multi-step settings. The Max Mean step metrics serve as direct measures of the length of continuous association chain, offering more precise insights into the capacity to comprehend rules and uphold stability. Intuitively, when humans master the rules through some examples or previous association processes, they can keep going without making mistakes.
Figure 3 shows the attribute, affordance, and action results in the synchronous association with four memory strategies. The results clearly show significant gap between open-source MILLMs and hu- man experts, i.e., the green column compared to others. In attribute synchronous association, human experts achieve the mean-step of 350, while the average step of the open-source models below 20. Additionally, different memory strategies exhibit consistent trend with single-step associations across various approaches, with NLM showing slightly better performance in both max and mean step evaluations compared to other memory strategies. Furthermore, the open- outperforms ALaVA-One Vision and APLUG-OWL1 which we attribute to its pre-training methods that achieve stronger cross-modal alignment, resulting in greater performance gains. This guides
QWen2- VL
8

---

Published as a conference paper at ICLR 2025
Deduction step
They have "natural" attribute.
New memory:
Memmry
Fresh.
and birds have fresh attributes.
imagel
Imagez
Association step
stop
Memory
Imaget
Image?
Quary mmge
Imagel
Imagee
Query Fmage
Image1
Image?
Fallure Error Deduction
Fallure Association
Figure 5: Failure cases of GPT4-V in asynchronous association with paired attribute categories. (Left) The error arises from the deduction. (Right) The error originates from the association.
future exploration toward improving the associative capabilities MLLMs. The detailed analysis of comprehensive results refers supplementary.
5.3.3 ASYNCHRONOUS ASSOCIATION
In the following, we proceed to investigate the MLLM's ability in asynchronous association. That improves the complexity by introducing two different semantic categories compared to synchronous association. Figure summarises the results of asynchronous association in the OCL and Pangea datasets. We involve four different semantic concept pairs for each dataset. In all datasets, we take the same memory strategies the synchronous association.
From the result, we can easily found asynchronous association can a challenge for humans in some cases. For instance, in the Pangea action, the human expert achieves average mean-step of 33.5 and an average max-step of 58.0, which is lower than synchronous association. In addition, meets our expectation that MoE outperforms the individual open-source MLLM in all cases, i.e., the max-step and mean-step of MoE are higher than open-source MLLMs. It catches our attention that closed-source MLLM has close performance open-source MLLM, i.e., GPT4-V, and Gemini- 1.5-Flash compared to open-source MILLMs. This indicates that all current models exhibit poor performance on association tasks when the gap in perceptual capabilities is minimized. Furthermore, Gemini- 1.5-Fashh outperforms GPT-4V in certain cases, which speculate is due to the MLLM verification step in data refinement. This step employs a question-answer strategy to verify existence of concepts, initially relying Gemini-1.5-PP only when Gemini- 1.5-Falh is unable to provide a decision (Subsection
for judgment and deferring GPT-4V
5.4 ANALYSIS OF RESULTS
In the following, we first discuss some failure cases and then deeply analyze the potential reasons why MLLMs are inferior to humans via attention and underlying capabilities! 5
Analysis of Failure Case. We divided the emors into two types according to the stage our benchmark, i.e., Error deduction, and Error Association. For errrr deduction, the MLLM predicts the error links for associated objects. This then causes the error the association step (Left of Figure 5). Conversely, for error association, MLLM has the correct memory and makes emros due to the limited perception capability (Right of Figure 5). More interestingly, these failures are consistent with humans, may derive error information and further induce incorrect judgments.
Analysis of Attention. We demonstrate that there is a significant gap between MLLMs and hu- mans. We speculate their two main factors: lack of multi-image instruction tuning and limitations in contextual understanding. The first statement is also observed from existing work SSong et al. (2024); ( et al.) (2024)). that the current MILM has a weak ability in multi-image understanding since the lack of its instruction data. In addition, recent research Wang et al. (2024a) has demon- strated MI.LMss poor performance handling long contexts, i.e., memory. We also visualize the attention map of open-source MLLM Qwen-VL OCL attribute concept to support this observa- tion, as Figure we can easily find that response has predominate attention at the position close to response instead of the part for decision-making in both StructM and NLM. 'Due limited space, the detailed analysis refers Section C in the supplementary.

---

Published as a conference paper at ICLR 2025
Attention Map
StructM Attention Map
Rei memory predecie darkoetee)
Related memory precesion (mporran derkeled))
0.4
Foput amrpins conterwe))
samples
(imp-tantt be derkened)
0.3
78
28
Avarage
Awargt
Marroy
Mentory
Oegina
Vervyy
Degral
Carditen
CLtbe
Inerodiinn
Irage)
traccnn
Indiadton
/Prompp (Context
Prompt (Conloxt Order)
Cefertiss tnd rolttrre1
Given the
1ipho Fordaros Car
(n orgnal mas
Otgna
Garchdat
Smorth mentre
Part
Iobly afferderce
Uecsere: ottt bute organel maag.
Iraget
Image::
Oprone Brrggei, Imuge2]
Iaam
Figure 6: Attention weight of QWen VL with StructM and NLM strategies in asynchronous associ- ation. A deeper color means greater attention. The frame marks the critical area in the input context.
WL
5.5 ABLATION STUDY
We conduct ablation studies that take different example sizes in the associations to analyze the selection of an initialized memory base. Table 2 summarises the results for example sizes of 1, 3, and the ILaVA-One Visionn [Li et al. (2024a) and QWen2-V VI. Wang et al. (2024b). Regardless of whether considering the max mean step, the influence of different sample sizes is minimal, with an average gap 1.51 for the maximum step and 0.05 for the mean step. As the maximum step cases, while the mean step reflects we have chosen a sample size of 3 based on the mean step results for all our experiments.
indicates peak performance in
stable performance,
Example size
LLaVa-OneVision
QWen2-VL VL
Avg.
StructM
NLM
ChainM
StructM
NLM
ChainM
30.3 3.22
33.3
3.92
28.3
3.34
21.5
2.05
36.8
4.36
27.8
3.41
29.67
3.38
26.3 3.14
36.8
3.94
31.8
3.35
21.8
2.26
43.8
4.43
22.8
3.45
30.55
3.43
32.5
3.17
36.5
3.90
32.8
3.35
21.5
2.09
38.8
4.53
25.0
3.39
31.18
3.40
Table 2: The ablation of example sizes the association. (Detailed results in Table 14]
LIMITATIONS AND DISCUSSION
In this paper, our investigation is limited to MLLMs' zero-shot ability in association tasks across adjectives and verb semantic concepts. Experiments demonstrate that although MLLMs make ad- vances in other scenarios, they exhibit weak ability in association. may be due to the lack of learning of unpaired data. To the best of our knowledge, current MLLMs are trained on image-text pairs and interleaved image-text pair data, providing them with a powerful ability comprehend input information. However, the association task requires MILM have the capability of inference on unpaired sequence data, as well as the ability to gradually uncover the un- derlying principles through the process of all prior decision-making In the future, an urgent study is still needed to develop a paradigm that links new learning with prior learned knowledge, which may enhance MLLM's association capabilities. We believe that the next stage of MLLMs should expand the learning of unpaired data, which may require the creation of new learning framework. This advancement will help narrow the gap between MLLMs and human intelligence.
speculate that this deficiency
CONCLUSION
In this paper, we propose benchmark to evaluate the association ability of MLLMs via an annotation- association construction method that easily transfers general datasets for association tasks. Using this method, devise standard benchmark based adjective and verb semantic concepts as the association chain. Expanding experiments demonstrate that current open-source MLLMs and even GPT-4V have significant gap compared humans.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grant No.62306175. 15, -CF-Tencent Rhino-Bird Open Research Fund.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087744 2023.
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu. Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 0l.ai, 2024.
David P Ausubel. The psychology of meaningful verbal learning. 1963.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023b.
Kilian Carolan, Laura Fennelly, and Alan F Smeaton. A review of multi-modal large language and vision models. arXiv preprint arXiv:2404.01322, 2024.
An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.00 015444 2024.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Wenliang Dai, Jumnan Li, DONGXU LI, Anthony Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- pprppse vision-language models with instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp- 49260, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. didemlm-xcooppseee Mastering free- form text text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420. 2024.
Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky. Towards multimodal in-context learning for vision and language models, 2024.
Pengbo Hu, Ji Qi. Xingyu Li, Hong Li, Xinqi Wang, Bing Quan, Ruiyu Wang, and Yi Zhou. Tree-of- mixed- thought Combining fast and slow thinking for multi-hop visual reasoning. arXiv preprint arXiv:2308. 09658, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Huaizu Jiang, Xiaojian Ma. Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard- hoi: Benchmarking sho-shot visual reasoning for human--obect interactions. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 19056-19065. 2022.
11

---

Published as a conference paper at ICLR 2025
Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip Ilievski, and Jay Pujara. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. arXiv preprint arXiv:2404.1359/ 2024.
DeBrae Kennedy-Mant and Jake Gord. "model cards for model reporting" in 2024: Re- classifying category of ethical considerations terms of trustworthiness and risk manage- org/
www.nticscchhooooooooooooo ment. ArXiv, 22222403311994, 2024. ctt.
GorpusID:26869112.
Jihyung Kil, Zheda Mai, Justin Lee. Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, and Wei-Lun Chao. Compbench: comparative reasoning benchmark for multi- modal llms. arXiv preprint arXiv:2407. 16837, 2024.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV), 2011.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey benchmarks of multimodal large language models. arXiv preprint arXiv:2408.0866 2024b.
Yong-Lu Li, Yue Xu, Xinyu Xu, Xiaohan Mao, Yuan Yao, Siqi Liu, and Cewu Lu. Beyond object recognition: A new benchmark towards object concept learning. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 20029 -20400, 2023.
Yong-Lu Li, Xiaoqian Wu, Xinpeng Liu, Zehao Wang. Yiming Dou, Yikun Ji, Junyi Zhang, Yixing Li, Xudong Lu, Jingru Tan, et al. From isolated islands to pangea: Unifying semantic space for human action understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 16582-16592, 2024c.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint 2xXiv:2310.0:7:. 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https://
llava vl. vl.github. ho/blcg//024 01 30- llava-nexe
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Arjun Mani, William Hinthorn, Nobline Yoo, and Olga Russakovsky. Point and ask: Incorporating pointing into visual question answering. 2020.
Sarnoff Mednick. The associative basis of the creative process. Psychological review, 69(3):220, 1962.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge- based visual question answering. In 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 11973,14983, 2023. doi:
10.1009CCVVS772222233114
Alcksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about red circle? visual prompt engineering for vlms. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 11987-11997, October 2023.
Dingjic Song, Shumian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. arXiv preprint arXiv:2404118532, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Shenghang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568 9978, 2024.
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang. Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models. arXiv preprint arXiv:2406. 1230, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision languae model's perception the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jachong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for mul- timodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529 2024c.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thouuht prompting elicits reasoning in large language models. NIPS '22 2024. ISBN 978171387108
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Siwei Wu, Kang Zhu, Yu Bai, Yiming Liang, Yizhi Li, Haoning Wu, JH Liu, Ruibo Liu, Xing- wei Qu, Xuxin Cheng, et al. Mmra: A benchmark for evaluating multi-granularity and multi-image relational association capabilities in large visual language models. arXiv preprint arXiv:2407.17373 2024.
An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, et al. List items one by one: A new data source and learning paradigm for multimodal llss. arXiv preprint 4rXiv:2404 16375, 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv: 04840, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan. Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modes large language model with modality collaboration. arXiv preprint arXiv:231 1,04257, 2023.
13

---

Published as a conference paper at ICLR 2025
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. arXiv preprint 1xxiv:2306.135443 2023.
Andrew Zaldivar, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji. Lucy Vasserman, M. Mitchell, Parker Barnes, Simone Sanoian McCloskey Wu, and Timnit Gebru (eds.). Model Cards for Model Reporting, 2019. URL https://cll acm. org/ citation, cfm? ?id=
3287596
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm- Ilms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.3601, 2024a.
Zicheng Zhang, Haoning Wu, Erli Zhang, Guangtao Zhai, and Weisi Lin. benchmark for multi- modal foundation models low-level vision: from single images pairs. arXiv preprint arXiv:2402.07116 2024b.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2025
ITamy.... 3.r..--Traaa Traest
Forewater Imove" 1. I'nerpprtt
wertill
solitinal expionatoo
Generating Semandd Concegt Accociation Choin
Advarced MLUM
asvodiation step
Oweet
Libel
Origie
Vinge
dap
Carnett Dhaingushi
Image Resolution Fiber
MLUU Verttication
Human Expert Eveluation
Data Quullity Control
Figure 7: Annotation-free construction method. The left transfers supervised data for the asso- ciation dataset. The right is the expansion of three strict data quality control steps, including image resolution filter, MLLM verification, and human expert evaluation.
A IMPLEMENTATION DETAILS
A.1 IMPLEMENTATION OF DATA REFINEMENT
In our experiment, we comprehensively develop three steps to consolidate the data quality our tasks, including Image resolution filter, MLLM verification, and Human expert evaluation, seen in Figure]
Image Resolution Filter. In this work, we concentrate on the object within the image, yet the current object detection and object concept learning datasets typically represent only partial content of the images, posing significant perception challenges. Hence, the Image resolution filtering step excludes all images with fewer than 50, 000 pixels, ensuring sufficient visual quality.
MLLM Verification. While an image resolution filter ensures that images contain sufficient in- formation for downstream tasks, there remain existing raw annotations, and the other is perception ability in difficult categories. We propose an MLLM verification step that further filters input samples with erroneous annotations those requiring ad- vanced perception capabilities. This step does not introduce bias, as the filtering method relies on a widely used public dataset with only insignificant shortcomings. Specifically, we employ a question- answers strategy to check whether the model identifies the existence of one concept. This process begins with Gemini-1.55 Fls-aahh then GPT4-V once Gemini-1.55 1.5-Falh is unable to reach decision.
concerns. One is the correctness of
Human Expert Evaluation. The first two steps ensure the correctness of our benchmark, which reduces the image with low quality confusing annotation. We continue to develop online testing interface for human testers, as shown in Figure g which further filters the images with potential confusion ethical concerns from the human perspective. We have strictly followed the ethical review, which described in section H of the supplementary.
A.2 IMPLEMENTATION MEMORY BASE
In our proposed association tasks, whether involving single-step, synchronous and asynchronous associations, we developed three memory strategies emulate human intelligence: Structure Mem- ory (StructM), Natural Language Memory (NLM), and Chain Memory (ChainM). As in Table 3 the prompt in the association task can divided into three parts. The first component is the memory context, including the memory instructions and base. Next is the question content, comprising the question instructions and the questions themselves. Finally, the output instructions guide the model to ensure the generated output aligns with specified requirements.
In detail, for each association step. NoM operates without any memory content. StructM employs simple memory instructions that incorporate memory and structured dictionary knowledge from
15

---

Published as a conference paper at ICLR 2025
example,
Report
Choose the related atribute. ABl froit saad,, cocet sube Sary ron
Solecct the image with the attribute the original
beit paitec fary,
Proncison
Taperhuw
This one correct
Roadyy
Figure 8: The human evaluation interface in data refinement is designed to filter low-quality data and flag potential ethical issues. The evaluation process consists of two stages: first, preview phase to familiarize evaluators with sample data (Left), followed by the main testing phase (Right).
Type
lyst
Mensey Insrection
Mexsory
Queation Instratiin
Qacitias
Osipal lootuciio
NoM
Teromine the wilskoedpp hetweee original winnr with the same entheesiiomtdeeeiiiiii oepiinal image.
Oripiaal truge wwegeeiiegooo cclaage Imaget cilaaee-ce
Your respones doorld te diocc excludeelly follawing Oprine (Iragel, Image?)
Smash
NLM
Refire lave leart thai reliee pidtuco lave Sollowngg uffeclaee:
('recedP;; zr atfentance
Chism
breccall- Seal- blsked
Dedactioon
Step
Genrrine common affordanee
betwenn the orgginal inege and oeiecced inage.
Cheptall Sclecied image: <mmggeo
affendanc hellwwnn Opticett beak: 'cart. "kloe lur. lppen. pad. imprrn
Table 3: The prompt format in our association benchmark, which includes the association step and deduction step. The association step makes the selection according to current observations and prior practice. We devise three memory strategies in the association step. In the deduction step, MLLM deducts the underlying concept with the original image and correctly selected image.
prior tasks. Both NLM and ChainM utilize more detailed memory instructions, with NLM rep- resenting prior knowledge in everyday language, while ChainM organizes the memory task- oriented sequence. Furthermore, the deduction step includes a detailed question description and corresponding output instructions, without relying any memory content.
In the experiment, we set the repetition weight wr and forgetting decrement d, are 1.0 and 0.2 for memory base attention in StruetM and NLM in all cases, respectively.
A.3 IMPLEMENTATION OF EVALUATION SETTING
Transferring from the single-step synchronous and asynchronous steps, we evaluate the asso- ciation capability in dynamic sequential environment. This involves iteratively processing the association step and deduction step and exiting when the association makes an error. In this setting, evaluate the max mean step, which is the maximum step and average step across multiple round tests, respectively. The related evaluation setting can refer to Algorithm I
To memory during synchronous and asynchronous steps, developed three distinct strategies for transferring experiences in the deduction step to update association memory. These strategies encompass memory storage, deep memory retention, and forgetting mechanisms.
16

---

Published as a conference paper at ICLR 2025
Algorithm Synchronouu/Asyycchooooo Association Evaluation
1: procedure MAIN
2:
for all attr E options do for epoch
3:
N do Reset_ Memory)) for i max do
4:
5:
6:
if Asynchronous and mod 5 then Switch
7:
Change to the other attribute in Asynchronous.
8:
end if
9:
query_img + previous
10:
correct img, false img Get Next Images(attr) correct answer t- Random Random_Order()
11:
Randomly set Imgl Img2 correct image response + Query-MLLM _MLLM(mmmorr, query jim,, correctimm,, false false_img) if response j correct_answer then Record_Step.Lennge break end if if mode * "nomem" then
12:
query
14:
16:
18:
StotecuionStepqqueryy jmg,, correct_img)
19:
end if end for end for
20:
21:
22:
Calculate Statisticx(steps)
Calculate Max|Mean step for all epochs.
23:
end for
24:
end procedure
B DEEPLY COMPARISON WITH EXISTING BENCHMARK
Our paper is the first to conduct comprehensive investigation into sequential concept association MLLMs. As summarized in a recent benchmark survey Li al. (2024b), MLLMs' bench- mark mainly concentrates on the design of more complex tasks and evaluating nuanced correlations between input samples. More deep comparison, our benchmark also needs general perception ca- pability LLaVA-Benehs Liu et al. (2024b), nuanced features within images as Compbench Kil et al. (2024). and cooperation across different modalities as SpatialRGPT [Ceng et al. (2024). Furthermore, benchmark needs exceptional abilities beyond existing work. First, bench- mark builds on adjectives and verbs that need deeper perception than nouns in MMVP Tong et al. (2024). Second, our benchmark sequential images beyond the images the existing work of MILEBENCH (Song et al.] (0024). Finally, our benchmark breaks the closed reasoning into open scenarios, which evaluate an inductive and deductive process rather than ground-true 'yes/no' in existing work as MARVEL Jiang et al. (2024).
In summary, our benchmark involves the basic capability existing in the previous benchmark, in- cluding general perception, nuance features, and cooperation across different modalities. Simulta- neously, our benchmark involves exceptional abilities, including deep concept perception, sequential image tasks, and larger solution space.
C DEEP ANALYSIS OF MLLM INFERIOR HUMANS
We speculate three possible reasons contributing to the significant gap between MLLMs and hu- mans. The related content can be referred to Figure I0
The benchmark needs more precise locationality. Compared to existing methods Wu et al. 2024) that rely language questions with explicit information retrieve related image content, our benchmark includes comprehensive memory that stores structured knowledge from prior expe- riences. The key information relevant to the model's predictions is often distributed across different
17

---

Published as a conference paper at ICLR 2025
Perception & Understandin
Association
Cogeitive Reasoning
[Pramps] letweeen
Eomn need
sossersco on
aenotel
with attricute Image
puroos flrrrt
Can you contimm hae greste dih comp and feet
about thss image?
perception
Origial imape card imagisi Imagel
Image!
The unsvel espect the Image
Comprsharaine qusation arteeds alth-tench li (2000S)
Omagez, trags?).
Gonor Soutaa Rousernn Sports RCPT Ching atall (3024)
[MLLM output]: Iroget
Success! Step =c
benchmart aeedcadjectin
Deduction stop
Ourbanchnark Iesequeetal two imagre).
[Prompt] Genorets the common attrbbue belwwsnn the orginallmae image and aducceed Image-i In
this
endentanding.
Origiel irape
Selected orepe:
No. there
Nout ropprre doo onty indace shored dlricaite the Vokkwing optionelle
Optione: (Turryy metal Trah; cooked, "ratuni, 'taa' ppanted neryy..
images?
Ree-grined
buth
[MILMM
MILEBEECH Soney (2024)
Muttirm odiil Long contest
concectrnte
Whwhh bir more gee on the breek?
features
reanoad
Traintes Takel Xart
naty: Itar,
Menory
Oriinal Curd ulsa
Association
[ML Outpan]
Dedaction step
Curput]
dasse to open
setting (use
from
nels ny pas?
mure wihithe
the
PAYS
Memory
Asseciatinn Shop
Compbenen Klietal (1924)
Fusty' (uart (taier
(paintect: (lemp kcarl
(MILLM Outpat:
Image
Error!
Print (step)
instedd of doest
ability
Absiractoon Reasconing lbng (P024)
Original
Cardistes
aeowering.
Our Sequential Concept Association
Figure 9: Comparison with existing work. Existing work can be roughly divided into perception & understanding and cognitive & reasoning. Those all target the design of more complex tasks or nuanced evaluation. Conversely, our work concentrates evaluating sequential concept association between different objects.
locations and represented as relatively local, subtle elements within the context. This highlights the need for more advanced attention in our benchmark.
The association needs more common sense-based reasoning. The association is an implicit link between the different objects that are not explicitly shown in the query, requiring a deeper common sense understanding. For instance, existing benchmark [Song et al. (2024) infer the weather in the image, they directly use the word in the query to retrieve the part in the image. contrast, our benchmark only specifies the direction of the link, requiring the model to think of the underlying link within the prompted scope.
The association needs short-term memory, which is underexplored by existing work. Our sociation benchmark is sequential concept links between different objects, which inherently need the memory mechanism to retain the prior experiences. This aspect of memory remains underex- plored in existing benchmarks, with current designs falling significantly behind the sophistication of human cognitive systems. This inspires the future study in memory design.
D PRELIMINARY INVESTIGATION ON CONCEPT PERCEPTION
To thoroughly access the MLLM's ability in concept perception, systematically design multiple task complexities across multiple tunning-free methods. Specifically, we involve four task complex- ity settings across two types in all models, including individual and combination perceptions. The individual perception includes Task Instruction (TaskInstr) and Meta Instruction (Metalnstr). How- ever, combination perception consists of Task Instruction with Deduction (TaskInstr w/ DedPo) and Meta Instruction with Deduction (MetaInstr w/ DedPo), which involves another deduction step com- pared with individual perceptions. The distinction between TaskInstr and Metalnstr lies in whether the prompt includes question instruction that describes the recision-making explanations of the task. In addition, with Deduction (w/ DedPo) compared to normal, is whether we only access percep- tion or evaluate deduction ability. On the other hand, enhancing perception capabilities can be approached from several directions, including expanding foundational knowledge and incorporating practical examples. Hence, we include several prompt engineering skills for tunning-free meth- ods such as common knowledge (ComKnow), one-shot, and Ch Thought (CoT). ComKnow aims to provide foundational knowledge of object-related concepts, while One-shot and CoT focus
18

---

Published as a conference paper at ICLR 2025
Pypkit
What wwweteeeeeettter A. it's Suuies & it's fagay
CrAs ame ineeatsmment
deall pick & thepe reupis
maning heoudy. das
penled?
cooeeneeed??
Previous
V.S.
MARRA We of (2024)
Given them theronond Taary Thusk eago nhe rudiitie ancees onlict imagr the alurtance orignal
V.S.
(2020)
Withost Mimorry
Por Memary
Memory (Prie Experiencaj:
daperddd
Ours
parted EM
Spointed ('aieri, Pearl
Implicit
"nerall harl Mar
Output irage
benw whe
Openry
cardd DilL
Ours
the sngiaal anagr
and candair
Dulpst
rape wirthh
eije miph
candidere Ing!. cancilate ing21
Theireto thrstativn bonnbils stundoeed
Mamory:
Tpartted: tell pat Ler,
Our bendhmart needs predie Locationality.
burbbrchaatt neads nore common sense- based reasoning.
Our benchmork reed short-term memory mechanism. what is ande soplooed by couting work.
Figure 10: Deep analysis of MILMM inferior humans. We speculate there exist three possible reasons for this gap, including salience attention, common sense-based reasoning, and human-like memory.
on enhancing performance through few-shot learning. Compared to One-shot, CoT introduces an additional reasoning step to emulate human- like thought processes.
We carry out experiments on three open-source MLLMs. QWen- Bai et al. (2023b) uses QWn- 7B Bai et al. (2023a) as the initialize of LLM and OpenCLIP's ViT-bigG Cherti al. (2023) as the visual encoder. LLa VA-NeXT-7B Liu al. (2024a) is constructed the LLM of Mistral-7B 7B Jiang et al. (202)) and vision encoder of CLIP's ViT-L/14 Radford et al. (2021), while LLaVA-Nea NeXT- 13B
uses the same vision encoder but is constructed LLM of Vicuna 13B Zheng al. (2024).
Concept Perception Task Complexity (Success Ratio)
Model
Strategy
Individual Perception
Combination Perception
TaskInstr
Metalnstr
TaskInstr w/ DedPo
Metalnstr w/ DedPo
Random
0.5
0.5
0.25
0.25
Normal
0.515
0.690
0.506
0.542
ComKnow
0.594
0.512
0.502
0.419
QWen-VI VL
One-shot
0.610
0.638
0.425
0.436
CoT
0.518
0.657
0.440
0.433
Normal
0.494
0.565
0.433
0.415
LLaVA-NeXT (Mistral-7B)
ComKnow
0.568
0.498
0.411
0.374
One-shot
0.641
0.631
0.453
0.475
CoT
0.664
0.667
0.494
0.519
Normal
0,510
0.587
0.278
0.420
LLaVA-NeXT
ComKnow
0.597
0.532
0.408
0.381
Vicuna-13B)
One-shot
0.598
0.605
0.411
0.421
CoT
0.590
0.613
0.427
0.430
Table 4: The success ratio OCL for open-source MILMs the concept perception. We include prompt-engineer methods ComKnow, One-shot, CoT. The best results are shown in bold.
Table summarises the result the OCL dataset. In all models, we include four different task complexity settings: Task Instruction (TaskInstr), Meta Instruction (Metalnstr), Instruction with Deduction (TaskInstr w/ DedPo), and Meta Instruction with Deduction (Meta w/ DedPo). For com- parison, we include the Random and Normal as baselines, as well as general prompt-engineer skills Common Knowledge (Comknow), One-shot, and Chain-of-Thought (CoT).
19

---

Published as a conference paper at ICLR 2025
Open-source MLLMs perform poorly in concept perception, reflected in Normal having close performance to Random in all cases. The prompt engineering skills show a slight improvement in some cases. The biggest improvements for ComKnow, One-shot, and CoT are 0.130, 0.147, and 0.170, respectively. While they demonstrate improvements in certain cases, they exhibit weak stability. For instance, in the CoT strategy employed in the TaskInstr w/ DedPo setting, the LLaVA- NeXT-13B has shown an improvement of 0.170 w.r.t Normal whereas Qwen-VL exhibits a negative effect. Combining all results, we conclude that current MLLMs have limited capability in this task.
E DETAILED RESULTS
E.1 DETAILED SINGLE-STEP ASSOCIATION RESULT
Single-Step Stepe--Sepppaa Attribute Concept Association Chain
Type
Model
Memory
fuury
fresh
cooked
ripe
pointed
rusty
Avg.
NoM
57. 63
75.21
75.78
76.09
51.19
86.83
58. 19.19
93.21
75.52
StrectM
57.08
79.73
83.23
78:26
53.72
89.22
57.10
98.61
78.11
L.aVA.Onevisionn
NEM
60.15
$3.52
87.58
78.26
51.82
92.22
93.70
96.94
80.40
ChainM
60.90
$1.80
86.90 46.96
65.22
03.41
68.02
88. 98
96.95
77.73
NoM
60.08
76.61
85.09
71.71
53.88
87.13
78.71
97.
76.36
SinccM
61.31
87.51
83.23
78.26
55.78
79.04
88.98
87.84
77.74
QWea2-VL
NLM
65.26
93:06
96.27
97.83
62.61
96.41
92.91
98.65
86.63
Association
ChainM
63.90
91.59
92.55
93.48
55.15
97.01
93.70
1.00
73.55
Success Ratio
NoM
52.72
49.68
63.35
45.65
49.60
50.28
56.69
55.41
54.05
StrucM
61.38
75.16
63.01
62.63
79.53
72.67
mPL.IG-WW13
64.71
86.31
71.74
51.19
63.85
75.59
56.49
73.33
ChainM
61.72
62.42
83.23
65.22
67.05
83.23
02:20
90.54
70.70
NoM
59.11
71.01
82.
84.78
55.85
89.02
68.95
91.66
75.40
StrctM
68.90
91.80
93.70
97.80
58.40
92.70
95.20
100.0
$7.30
Gemini- 4.5-Flash
NLM
74.90
92:90
96.90
95.50
57.50
94.80
18.90
100.0
68.39
ChainM
70.30
92.60
87.
90.90
56:40
79.90
94:40
98.6
63.78
NoM
58.06
80.07
96.13
90.47
52.67
95.73
95.12
98.65
$4.49
SinccM
74.11
98.98
99.39
97.83
44.95
99.40
96.00
100.0
88.83
GPTT-o
NLM
78.35
96.98
96.20
97.83
53.77
95.83
100.0
100.0
89.87
ChainM
72.13
90.59
81.82
100.0
46.15
61.48
66.67
100.0
79.86
LLaVA-Onevision
71.73
69.37
90.55
12.
19.38
QWen2-VI.
63.10
89.75
51.55
63.04
30.62
59.88
50.84
58.11
58.11
Deduction
Success Ratio
mPL.IG-OW13
66.76
61.49
84.78
32.40
47.31
71.65
81.08
61.21
Commini-1.5-Flach
74.44
02.15 18
79.76
84.78
62:10
6.70
91.60
75.00
65.82
GPT-4o
59.39
86.34
94.19
73.81
90.77
52.80
$6.99
80.82
78.14
Table 5: The success ratio of single-step attribute association, which includes the association step and deduction step. The association step evaluates MLLM's capabilities in decision-making based on observation and corrected prior practice. The deduction step accesses MILM's capability to generate the common concepts that the association step includes three memory strategies and baseline NOM. The best and second results are shown in bold and underline, respectively.
the underlying rule in the association step. Additionally,
In the main text, we highlight three key points. First, the best performance of the closed-source MLLM, Gemini-1..-Flasn still shows a significant gap compared to human experts. Second, our proposed memory strategies result in noticeable improvements across all cases. Finally, the NLM strategy demonstrates the highest performance compared to our proposed other memory strategies. In this part, we continue to analyze the differences within each concept.
Table 5 7 summarise the results of single-step association on attribute, affordance, and action, respectively. For each concept type, assess the association and deduction success rates using three open-source MLLMs- BILLMs-LLaVA. LLaVA-Onel Vision, QWen2-VL. and mPLUG-Owl3- as well as one conducted across three memory
clored-source MLLM, Gemini-1.55 1.5-Fll.. These evaluations strategics- StrreggicsSStruuMM,,, NLM, and ChainM-aloo with a baseline, NoM. We have selected eight cat- egories for each concept, and provide detailed analysis of the results below.
20

---

Published as a conference paper at ICLR 2025
Single Step Aflordance Concept Association Chain
Type
Model
Memory
imprint
push
carry
open
break
Avg.
NoM
77.06
66.74
80.29
58.73
$3.33
51.83
78.30
65.57
StrucIM
81.18
73.43
85.53
62.20
80.44
48.39
63.96
71.99
73.39
LLaVA, Onevision
NLM
87.65
75.10
87.16
00.35
83.
58.06
88.68
68.17
76.13
ChainM
86.47
72.80
83.73
60.81
$2.00
53.22
90.57
68.58
74.38
NoM
81.76
67.78
79.39
62.54
85.54
72.58
81.13
70.08
75.10
StructM
83.53
QWen2-VL
NLM
$8.82
74.90
90.96
09.25
89.
48. 39
89.62
83.33
79.37
Asociiaion
ChainM
87.65
80.96
90.24
73.53
9/2 422-42
36.45
96.22
83.58
82.67
Success Ratio
NoM
58.24
08.33
53371
54.57
59.64
511.25
51.89
50.88
53.80
StreeM
81.76
61.51
74.86
68.09
74.24
06.13
83.96
73.63
73.02
mPLUG-OW11
NLM
78.82
61.72
67.99
60.81
79.89
46.77
78.30
68.03
67.79
ChainM
87.18
61.72
64.20
02.77
68
67.92
68.58
46
NoM
86.39
53.33
62.15
95.18
63.93
74.79
77.78
Gemini-1, Saaiii--.5-FFaaa 1.5-Flash
StracMM
97.02
70.72
91.77
73.99
98.
58.335
99.04
86.55
84.45
NLM
97.02
83.64
92.00
774.79
98. 20
05.17
98.06
83.44
86.40
ChainM
94.64
76.50
86.29
66.40
95.58
46.67
95.19
78.92
80.02
NoM
88.51
77.58
92. 17
75.61
98.
60.99
95.92
89,68
81.93
StrcctM
98.21
87.05
94.97
86.46
74.50
50.02
98.11
84.46
85.36
GPT-4o
NLM
97.65
100.0
80.82
ChainM
87.50
100.9
83.33
06
68.88
76.28
LLLVA-Ouevision
66:47
0.42
8.14
1.04
1.96
35:48
19.06
3.01
21.07
QWen2. VI.
21.76
14.23
1.63
1.50
1.24
4.84
50.19
2.46
15.98
Deduction
Success Ratio
mPLUUGOWW3
82,94
0.81
12:12
2.89
63.91
12.
84.91
5.87
33.30
Gemini-1,5-Flash
58.93
0.85
24.40
36.02
53.01
5.00
57.60
28.71
33.08
GPT-40
63.54
17.50
4.05
Table 6: Same as Table 5 but for affordance single-step association.
Comparison of Different Models. From the whole perspective. Gemini- 1.5-Flash consistently outperforms the other models across all cases. Specifically, for association success ratio, Qwen2- VL leads in several categories each concept. For instance, metal and cooked with NLM strategy attain the highest success ratio in attribute single-step association. Among the open-source MILMs, Qwen2-VL VL demonstrates the strongest performance across three concepts, with LLaVA-One Vision performing moderately well but trailing behind Qwen2-VL, and mPLUG-OW13 showing the lowest performance. Furthermore, different MLLMs exhibit consistent trends in concept understanding. For instance, LLaVA. VA-One One underperforms natural attribute association, the same pattern also observed with QWen2-VL VL and mPLUG-OW13 models. Additionally, there is significant dis- parity in deduction success rates among the different MILMMs. For example, Gemini-1.55 1.5-Fash per- forms well with "push", "ccary"", and "break" affordances, while the other MI.LMs show relatively weaker capabilities. Conversely, Qwen2- excels in "imprint" affordance, and LLaVA-One Vision demonstrates strength in the 'clean' affordance, whereas Gemini-1.5- struggles.
Comparison of Different Categories. Although single-step association shows consistent im- provements across various concepts, performance differences emerge in specific categories. For instance, fresh, ripe, and rusty attributes gain an association success ratio close to 1, but natural and ripe attributes with lower performance compared the random success ratio of 0.5. This phe- nomenon is more pronounced in the deduction success ratio. Several model implementations exhibit strong deduction capabilities across different categories. For instance, Semmnni-155-Faas achieves 91.6 6 deduction success ratio in the 'painted' attribute. However, some categories still demonstrate lower performance, highlighting insufficient object understanding.
Comparison of Different Memory Strategies. As demonstrated in other subsection analyses, NLM exhibits an overall more powerful performance than StructM and ChainM. But there also exists some observations that violate this finding, which further highlight the instability of the current MLLM. For instance, in the case of Gemini--1.5-Flas with "natural" and "painted" attributes, the success ratio of StructM strategy outperforms NLM. They demonstrate professional-level ability in
21

---

Published as a conference paper at ICLR 2025
Single-Step Action Concept Association Chain
Type
Memooy
drive
dress
cooking
huild
shalke
cut
Avg.
NoM
74.91
79.48
58.15
74.05
82.21
58.53
StrueMM
72:10
17.23
77.
75.36
85.63
78. 61.
79.95
56.67
75.41
AlaVA-OOeVissoo Vision
NI.M
78.10
77.50
82.50
75.36
58.03
79.99
87.39
60.44
78.66
ChainM
73.89
71:19
80.53
75.36
87.13
79.56
85.36
59.01
76.92
NoM
83.32
80,52
82.50
05.22
87.70
74.08
87.61
05.80
78.43
StreetM
80.71
79.29
81.01
75.36
87.89
87:22
91.82
70.50
82.10
QWea2-VL
NIM
89.72
66.97
88.66
79.71
96.31
91.87
96.17
75.70
88.01
Association
CbainM
85.99
87.00
76.24
Success Ratio
NoM
56.
62.26
08.02
62
06.49
63.34
57.43
65.24
57.94
StreetM
71.28
69.55
80.13
60.87
56.30
77.49
84.68
61.04
73.92
mPLUG-OW1B
NLM
63.
62.41
76.45
63.77
78.15
73:21
81.98
61.16
70.04
ChainM
61.04
62.14
78.61
50.72
82.57
73.39
8.1.91
59.55
69.50
NoM
85.96
88.96
80.23
71.88
94.16
81.94
92.05
030.4
84.21
Gemini- Bominn--1.5PPa 1.5-Flash
StructM
86.49
89.81
85.71
77.97
99.60
911.20
97.72
74.33
88.10
NLM
93.07
96.
93.78
97.28
73.65
89.58
ChainM
86.46
98.41
73.42
87.58
NoM
86.75
69.24
92.86
75.56
96.34
82.56
96.95
76.51
86.07
StructM
81.87
96.39
91.62
73.52
99.42
88.44
100.0
78.53
98.72
GPT-4o
NLM
87.37
89.41
80. 12
81.97
99.00
80.90
90.50
70.74
86.13
ChainM
85.00
90.40
91.00)
67.65
08.99
86.29
97.99
60.85
85.90
LLaVA- VA-Onevisiion
38.87
18.97
73.35
31.78
96.85
3.96
3.60
52,
46.61
QWen2-VI.
60.90
67.41
02.32
89.78
7.64
39.26
49.99
Deduction
mPLUG-OWB
33.99
74.52
96.74
7.72
74.62
Gemini- 1.5-Plushh
$6.99
97.79
19.96
GPT-4o
38.50
70.20
83 33.50
38.82
02.82
100.0
21.83
31.00 06
53.
57.28
Table Same as Table 5 but for action single-step association.
specific cases but cannot maintain consistent stability. We speculate that similar to the prominent research area of hallucinations, there is still significant room for improvement.
E.2 DETAILED SYNCHRONOUS ASSOCIATION RESULT
Model
Memory
Synchronous Attribute Coccept Association Chain (Max. Mean Step)
Farry
metal
fresh
cooked
natural
rige
painted
misty
Avg:
NeM
15
1.30
.1 13.20
:30
3.85
38
3.13
11
1.05
40
5.09
701
6.68
140
17.47
48.01
55:30
LLaVA-OneNision
StruetM
15 1.38
30 4.00
41
4.75
32
1.04
13
1.21
51
6.51
65
9:223
235
15.24
610.6
5.80
NLM
5.24
48
3.90
9.88
30.G4
33.88
ChainM
NoM
3:34
9:22
4.52
10.98
45.00
10000
Qwent-VV VL
StrrcM
NLM
15
1.74
1.61
58
12:30
6.77
89
3.35
8.50
25
36
2.03
4.52
22
11
1.34
1.27
109
35
11.90
3.10
162
37
3.42
12.50
1222
500
64.50
9.51
122.5
37.9
3.37
13.96
ChainM
18
1.64
20
4.388
42
7.53
32
4.04
10
1.18
77
9.70
(88
8.50
1.00
22.00
666.1)
7.30
NoM
11
1.06
12
1.08
13
1.59
15
1,27
12 1.04
17
1.16
13 1.28
16
1.54
13.0
1.27
IPLIIG-WWI
StreeM
NLM
2.354
9.34
2.885
5.07
4.48
391
ChainM
1.69
1.77
18
1.92
4.63
24
2.65
31.5
3.37
Expert-X
22
121
500
500
10
500
500
500
3511.0
Expert-Y
500
Expen Z
361.8
Max Mean
21.3
600
377.77
500
500
337.33
5001
500
500
3:00
1500
381.8 347.5
Table 8: The Max] Mean Step of synchronous attribute concept association across open-source MLLM and human experts. We select eight categories to reduce the complexity of the associa- tion and ensure the data quality. For open-source MLLMs, we involve three memory strategies, StructM, NLM, and ChainM, along with baseline strategy, NoM. The best and second results are shown in bold and underline, respectively.
As summarized in the main text. significant gap exists between current MLLM and human intel- ligence in synchronous association. Below, we provide a detailed analysis of individual concept as- sociations across each category, focusing OCL attributes, affordances, and the actions of Pangea.
22

---

Published as a conference paper at ICLR 2025
Synchreneus Afferdance Contepp Assnciation Chain (Max
MMan Step)
Model
Memory
impeint
push
canyy
clean
open
Augg
NoM
40 4.17
47
[4:34
11:49
12.006
ALaVA-OneVinn
SinucMM
NLM
50
3.70
4.63
30
2.88
46
45
[5.65
6.16
22
26
1:86
1.05
45 6.73
[7.41
2.07
95
73
10.95
8.64
2.61
12.80
42.6
4.88
ChainM
37
4.76
24
2.82
51
5.72
13
1.59
07
0.82
19
1.333
22
8.00
201 2.32
38.5
4.22
NoM
36
4.17
21
2.76
36
13.96
15
1.88
74
7.15
18
1.68
39
4.44
36 2.76
31.6
3,80
Quen2-VI.
StructM
NIM
51
48
6.12
3..00
91
21
2.75
1.229
225
35
12.00
4.42
25
19
2.01
1.96
68 6.94
255
2.65
16F
1.90
1.71
86
6.57
3.300
32 3.57
18 2.17
206
38.6
2.50
4.54
ChainM
0.08
37
4.81
22.59
7.75
43.45
37.5
NoM
1.42
323
14
1:14
1.22
1.72
mPL.UG-OW13
StnucIM
3.94
27
3:44
21
13:14
NLM
23 8.18
21:
21 11:88
11:39
2.224
3.91
ChainM
2:04
1.56
21
2.668
24 1.06
35 3.79
1:48
19 2.34
42 1.25
25:4
2.663
Expert-X
77
105
500
500
500
500
500
500
307
Human
Expert
Expor-Z
49
93
78
500
500
500
500
73
500
136
500
600
Max Mean
77 56.0
(02.0
(543.7
500
500.0
500.0
500
578.7
500
307.8
131.0
Table 9: Same Table but for synchronous affordance concept association.
Synchronous Action Concept Association Chain (Max
Mcan Step)
Model
Memory
run
hit
drive
dress
cooking
huild
shake
cut
Avg.
22
31
3.50
22
78 7.05
36
13
LLaVA-OneVision
StractM
23
2.57
25
2.90
32
9.61
27 2.52
35 6.49
30
3.11
34
4.71
15 1.73
30.1
3.447
NLM
25
2.95
27
3.66
32
3.84
22
2.36
56 0.61
29
2.58
63 5.86
16
1.71
33.8
3.73
ChainM
22
2.60
27
3.05
31
3.50
25
2.36
52 5.37
25
2.55
43 5.366
18
1.69
30.4
3.36
19
5,40
42
5.04
48
4.90
21
2:37
79 1.35
35
10.85
24
5:11
Qwea2-VL
StrecM
19
2.57
23
2.71
30
9.15
17
2:31
46
5:41
31
3.62
119
12.56
100
2:311
38.1
4.:33
NIM
48
6.27
35
4.94
64
5.57
30
3.76
162
16.48
49
5.59
197
24.07
24
2.86
76.1
8.69
ChuinM
3.47
12.09
45
19.07
39.1
7.12
NoM
12
11223
11
1.17
14
1.51
13
1.14
201 2.20
15 1.77
16 1:26
16
1.19
14.6
11:47
mPLUG-OW13
StructM
NIM
23
23
1.70
2:20
16
18
1.63
2.01
24
24
2.40
3.29
18
19
1.97
39 3.80
64
6.17
34
30
2.94
3.90
24 1.96
21
39
40
4.00
5.00
90.8
26.3
2.588
33.31
1.94
1.71
37
3.10
14
1.79
4.61
28
3.12
22 1.90
2.79
Experi-X
14
70
94.88
Human
Expert-Y
Expor-Z
22
7
22
14
10
5
187
10
500
500
37
15
31
26
90
17
76.8
91.1
Max Mean
22
14.3
70
35.3
30
15.0
37
|1.3
21.3
500
500.0
37
27.7
77
| 44.7
90 | 42.0
107.9
Table 10: Same Table 8 but for synchronous action concept association.
Table 8 9 10 shows the results of synchronous concept association on attribute, affordance, and action, respectively. For each concept, we include the same eight categories as in the single-step association. We include three open-source MLLMs in synchronous association, including LLaVA- VieVinion, QWen2-VL, and mPLUG-OW13. Additionally, we also integrate this setting into the human test interface. In this setting, we use Max Mean Step as the primary metric.
Comparison of Different Models and Memory Strategies. We find that QWen2-VI. VL in the at- tribute and action concepts outperforms other models in almost all cases. But, LLaVAAOL Visioon exhibits extraordinary capabilities in affordance association. This demonstrates the inconsistency of MLLM's concept understanding, which can also taken as a reference for future improvement of MLLM's ability. Interestingly, memory strategies perform worse than NoM in certain instances. For example, in the "cooked" attribute concept association under the QWen2-VL. model, NoM achieves mean-step of 4.52, while other memory strategies fall below this. We speculate that this attribute may have encountered deduction errors, as shown in Figure in the main text. The more remarkable situation QWen2-V VL with StructM strategy, in which memory prevents the utilization of correct input context. This leads to it attaining low performance compared to NOM.
Comparison of Different Categories. Notably, the MLLM demonstrates similar pattern to hu- man intelligence, as both humans and open-source MLLMs achieve low mean-step scores in the "furry" and "natural" attributes, as shown in Table g More specifically, different categories within each concept exhibit different capabilities, which are also inconsistent with human experts. For in- stance, in the synchronous association of individual affordances, human experts take average of fewer than 100 steps to complete the "sit" affordance, which presents certain challenges for testers. In comparison, QWen2- VI. achieves average of 6. steps, though this exceeds the steps required
23

---

Published as a conference paper at ICLR 2025
for the "carry" affordance. Notably, human experts are less likely to make errors when performing the "carry" affordance.
E.3 DETAILED ASYNCHRONOUS ASSOCIATION RESULT
Asynchronous Attribute Concept Association Chain (Max
Mcan Step)
Model
Memory
fresh-cooked metry
natural-ripe
painted-rusty
Avg.
NoM
11
1.15
20
3.07
0.99
43
5.39
23.0
2.65
StructM
10
1.24
36
3.52
17
1.11
59
6.63
30.5
3.13
LLaVA-OneVision
NLM
17
1.36
32
4.24
15
1.19
83
8.97
36.8
3.94
ChainM
32
3.47
1.08
7.65
3.35
NoM
13
1.24
29
4.47
1.19
44 6.22
25.5
3.28
QWen2-VI.
StructM
12
1.33
19
2.55
1.43
38
3.75
21.8
2.26
NLM
16
1.63
43
1.69
19
1.49
97
9.91
43.8
4.43
ChainM
10
1.42
32
4.50
1.19
6.67
28.5
3.45
1.03
1.26
1.04
16
1.23
13.3
1.14
mPLUG-OW13
StructM
11
1.44
3.26
1.16
4.28
25.0
2.54
NLM
12
1.46
24
3.07
11
1,25
25
2.62
18.0
2.10
ChainM
12
1.32
23
2.68
22
1.13
24
2.77
20.3
1.98
No
8
1.11
28
1.06
19
1.05
56
7.05
27.8
3.32
StructM
1.25
42.8
MoE
NLM
19
1.53
39
5.26
1,26
137
15.91
53.0
5.99
ChainM
12
1.36
39
4.40
12
1.17
82
11.38
36.3
4.58
No
7
1.68
18
5.38
8
1.84
15
3.35
12.0
3.06
StrucsM
2.46
76
13.8
2.46
186
49.5
69.3
17.1
Gemini-1.5- -.5
31
2.30
305
97.3
88.3
27.8
ChainM
1.53
24
5.30
2.43
51
14.9
22.8
6.04
No
7
2.07
53
12.1
8
2,93
121
24133
47.3
10.4
StrueMM
15
4.33
39
8.63
6
3.10
151
37.3
52.8
13.3
GPT-4V
NLM
3.07
10.5
2.33
168
45.5
59.0
15.4
ChainM
7
2.35
9.70
1.95
122
38.2
47.5
13.1
Expert-X
16
70
16
500
150.5
Expert-Y Y
21
80
500
163.8
Human
Expert-Z
295
Max
Mean
79
38.7
70
15.3
80
35.7
500
431
182.3
140.2
Table 11: The Max Mean step on paired attribute categories in asynchronous association setting. We involve three open-source MLLMs, cutting-edged MoE based on open-source MLLMs, two close-sourced MI.LMs, and human experts. three memory strategies, SturetM, NLM, and ChainM, as well as one baseline strategy NoM. The best and second results are shown in bold and underline, respectively.
open-source and close-source MILM,, we involve
Paired Concept Association is the most complex task in our work. We find that all max-step and mean-step decrease in this setting compared individual concepts association the main text. In this subsection, we detailed analysis of the paired categories association on adjectives and verb concepts, i... attribute and affordance OCL, and action in Pangea.
Table I2 13 summarise the results of paired categories concept association attribute, af- fordance and action respectively. In this section, we include the closed-source MLLMs Gemini- 1.5-Flash and GPT-4V, along with the open-source MILMS L.L-VA-One ViLeAAOOneiision, QWen2-VI, and mPLUG-Owl: as well as an MoE that integrates open-source MILMMs and human expertise. We include eight categories from each concept of attribute, affordance, and action.
Comparison of Diffferntt Models and Memory Strategies. All the max and mean steps decrease on each paired category by comparing with individual concept association, which consistence with the whole results. We speculate that this stems from weak ability to transfer knowledge between concepts, as human testers also require some time to adjust to new rules. Additionally, MoE outperforms open-source MLLMs in some paired categories, where there also exists some cases which MoE attains a balanced performance compared to one of the highest MLLMs. For instance, for paired categories of "sit-imprint" and "open- break", MoE outperforms all open-source MLLM in all cases, while "push-caryy" and "cut-clean" open-source MLLM leads the performance when comparing the open-source setting. On the other hand, different memory strategies remain
24

---

Published as a conference paper at ICLR 2025
Asynchronous Affordance Concept Association Chain (Max
Mcan Step)
Model
Memory
sit-imprint
push-caryy
cut-clean
open-break
Avg.
NoM
29
3.13
31
3.26
18
3.02
15
3.02
23.3
3.11
StructM
41
2.41
2.92
3.82
26.3
3.14
LLaVA-One Vision
NLM
34
4.09
2.70
3.45
4.12
21.5
3.59
ChainM
37
3.56
12
2.65
3.19
3.77
21.3
3.29
NoM
26
3.30
16
2.58
22
3.70
19
2.93
20.8
3.13
@Wen2-VI.
StructM
28
2.58
17
1.89
18
2.13
17
2.51
20
2.28
NLM-3
29
2.98
3.29
4.18
23.3
3.63
ChainM
25
3.54
2.79
3.55
3.93
3.45
NoM
13
1.10
1.23
1.27
12.3
StructM
20
2.75
22
1.90
21
3.43
21
3.19
21.0
2.82
mPLUG-Ow1
NLM
15
2.19
14
1.59
20
2.64
17
2.56
16.5
2.25
ChainM
15
2.19
2.05
3.08
2.70
16.8
2.51
NoM
26
3.08
19
2.44
21 3.10
15
3.25
20.3
2.97
MoE
StrucIM
27
2.41
3.16
3.84
3:20
NLM
38
6.41
20
2.93
16
3.66
24
4.42
24.5
4.36
ChainM
36
3.65
2.95 16
20 3.62
21
4.17
23.4
3.60
NoM
26
3.36
11
2.56
14 3.12
14
4,26
16.3
3.33
Gemini-L,5-Flaa
StructM
32
4.67
2.37
3.73
68
6.93
31.8
4.43
NLM
2.30
3.07
13.0
3.43
ChainM
11
3.67
10
2.93
2.10
15
3.27
11.3
2.99
NoM
17
6.00
2.67
3.43
3:00
13.3
3.78
StructM
29
4.70
7
1.70
13
4.00
19
4.61
17.0
3.75
GPT-4V
NLM
23
6.177
9
2.21
3.57
24
4.45
17.5
4.10
ChainM
16
2.43
3.36
12.0
3.91
Expert-X
199
19
29
134
95.25
Expert-Y
152
47
17
500
179.0
Human
Expert-Z
96
13
32
500
160.3
Max Mcan
26.3
26.0
500
378
194.5
144.8
Table 12: Same
Table
11
but for asynchronous affordance association.
inconsistent tendencies that they improve performance in different cases. For instance, Gemini- 1.5-Flash with StuctM paired fresh cooked attribute association leads other MLLM and other memory strategies, while QWen2-VL with ChainM in shake-cut action association makes leading- This further highlights the varying comprehension abilities of MLLMS when presented with the same input contexts, which also provides insight for improving the understanding of language specific dimensions.
Comparison of Different Categories. Different paired categories exhibit specific abilities in asynchronous association. For instance, the paired categories of "fresh-cookedd and "painted-rusty" have the mean-step of asynchronous step open-source MILMMs lower than 2.0, while the "fresh- cooked" and "painted-rusty" have the mean-step of asynchronous step greater than 3.0. We spec- ulate that this is due to two different aspects reasons. One is the insufficient perception of specific categories within one concept, and the other is the lack of consciousness to convert the concept.
E.4 DETAILED ABLATION STUDY
In this section, we supplement the experiment with the ablation study from the main text, concluding that the memory strategy with three examples shows most comprehensive performance across all experiments. We below analyze the differences between different categories.
Table 14 shows the detailed results of the ablation study of example sizes. We observed that while the average max and mean step slightly differ between different example sizes, there are notable with paired
gaps within specific paired categories. For instance, in the case of LLaVA-OneVisse attribute fftal""mtaa",, StructM-3 outperforms StructM-5 by approximately the mean-step of 1.95. Furthermore, while an average mean step of 3 outperforms in most cases, the example size of and 5 also leads in certain cases. For instance, with QWen2-VL on "painted-rusty" affordance and LLa LLaVA-One VA-One Vision on "fresh-cooked" attribute, the example size of 5 outperforms the others.
25

---

Published as a conference paper at ICLR 2025
Asynchronous Verb Concept Association Chain (Max
Mean Step)
Model
Memory
run-hit
drive-dress
cooking-build
shake-cut
Avg.
NoM
2.74
2.65
29
5.44
2.78
22.5
3.40
StructM
2.39
2.24
35
4.16
2.59
22.5
2.85
LLaVA-OmeVision
NLM
2.87
2.85
4.96
3.05
28.0
3.43
ChainM
2.22
23
2.39
23
4.54
2.87
22.5
3.01
NoM
2.56
5.99
4.73
27.5
4.48
StructM
2.51
2.34
28
4.52
24.0
3.32
QWen2-VL
NLM
4.97
3.94
63
88.1
5.43
44.5
5.66
ChainM
28
4.06
20
3.34
44
7.80
31
5.79
30.8
5.25
NoM
1.17
20
2.20
15.0
1.56
StructM
1.75
28
4.84
2.74
2.76
mPLUG-OW13
NLM
10
1.46
1.37
4.17
2.86
19.3
2.47
ChainM
16
1.59
1.43
28
4.43
2.95
21.3
2.60
33.15
35
6.39
3.89
29.3
4.39
StructM
2.74
4.75
27.5
3.37
MoE
NLM
4.40
3.50
39
6.55
4.24
34.8
4.67
ChainM
3.55
27
3.19
40
6.93
4.29
31.8
4.49
NoM
5.40
3.90
15
4.86
3.03
18.3
4.30
Gemini-1.5-Flash
StructM
75
10.6
31.0
5.67
NLM
3.90
2.70
11.7
18.0
5.63
ChianM
5.50
2.90
18
5.07
18.5
4.67
NoM
1.44
20
4.60
1.90
14.5
3.35
StructM
35
6.80
3.85
22.0
4.68
GPT-4V
NLM
6.70
3.60
39
8.11
1.28
25.8
5.42
ChainM
16
3.75
2.50
26
4.30
3.40
15.0
3.49
Expert-X
64
12
39
12
30.5
Expert-Y
10
32
26
59
30.5
Human
Expert-Z
19
17
77
35
39.5
Max
Mean
64
31.0
32
20.3
77
47.3
59
35.3
58.0
33.5
Table 13: Same Table I1
but for paired action synchronous association.
Paired Attribute Concept Association Chain (Max
Mean Step)
Model
Memory
furry-metal
fresh-cooked
natural-ripe
painted-rusty
Avg.
StructM-1
11
1.28
36
3.41
27
1.18
47
7.02
30.3
3.22
StructM-3
41
3.42
13
2.41
22
2.92
29
3.82
26.3
3.14
StructM-S
12
1.47
34
3.61
17
1.36
67
6.24
32.5
3.17
NLM-1
14
1.27
32
4.08
11
1.07
76
9.26
33.3
3.92
LLaVA-One Vision
NLM-3
17
1.36
32
4.24
1.19
83
8.97
36.8
3.94
NLM-5
16
4.27
1.15
86
8.86
36.5
3.90
ChainM-1
12
1.27
28
3.30
1.06
61
7.73
28.3
3.34
ChainM-3
11
1.21
32
3.47
17
1.08
67
7.65
31.8
3.35
ChainM-S
11
1.21
22
3.18
15
1.01
83
7.99
32.8
3.35
StructM-1
13
1.26
21
2.63
19
1.19
33
3.13
21.5
2.05
StructM-3
12
1.33
19
2.55
18
1.43
38
3.75
21.8
2.26
StructM-5
15
1.36
19
2.39
18
1.37
34
3.26
21.5
2.09
NLM-1
4.59
1.24
10.18
36.8
4.36
Qwen2-VL VL
NLM-3
16
4.69
1.49
43.8
4.43
NLM-5
13
1.54
4.69
1.35
10.52
38.8
4.53
ChainM-1
11
1.38
29
4.19
1.14
6.94
27.8
3.41
ChainM-3
25
3.54
14
2.79
20
3.55
32
3.93
22.8
3.45
ChainM-5
11
1.27
26
4.51
17
1.20
46
6.58
25.0
3.39
Table 14: Ablation study of the example sizes in synchronous and asynchronous association, which we compared by the results of Max Mean Step. {Memory arategy]-X indicates the X example samples involved in the preview of the association. In this setting, we are implemented in open- source MILMSs second results are shown in bold and underline, respectively.
PrrA--Oeiiiio and QWen2-Vi VL with three memory strategies. The best and
26

---

Published as a conference paper at ICLR 2025
Paired Action Concept Association Chain (Max
Mcan Step)
Model
Memory
Arush_hiirdivg
clap-hug
hands--t
smoke-eat
Avg.
NoM
10
1.02
1.14
9
1.04
10
1.07
10.0
1.07
QWen-VL
StructM
1.03
1.15
1.17
10
10.0
NLM
10
1.04
11
1.15
9
1.11
10
1.04
10.0
1.09
NoM
15
6.00
10 6.00
10
4.20
5 1.80
10.0
4.50
Gemini-Pro- ProviiPPooV
StructM
10.5
8.44
1.20
14
21.5
7.19
NLM
8.10
8.10
5.70
31 8.60
22.0
6.25
NoM
19
6,20
5
2.60
12
4.30
9 2.90
11.3
4.00
GPT-4V
StructM
23
12.0
37 11.0
17
7.70
14 4.10
22.8
8.70
NLM
15.6
6.60
3.10
19
20.3
Expert-X X
45
33
82
17
44.3
Expert-Y Y
182
94.5
Human
Expert-Z
172
108
28
88.0
Max
Mean
172 92.7
182
107.7
82
54.7
86
43..7
130.5
74.7
Table 15: The Max Mean Step paired action concept association from HMDB dataset in asyn- chronous association setting. We implemented at open-source MLLM QWen-VL close-sourced MLLM Gemini Proni Vision, and GPT4-V, as well as human experts. The best and second results shown in bold and underline, respectively.
F COMPARISON WITH MORE CONCEPTS
While we provide comprehensive experiments to benchmark the MLLM's performance in asso- ciation tasks, we further expand experiment verb concepts attain solid perspective on association tasks. Specifically, we conduct asynchronous association actions from the HMDB dataset, which implemented in the open-source MLLM QWen- VL and closed-source MLLMs Gemini-Pro- Vision and GPT-4V. Simultaneously, we have integrated HMDB datasets into our hu- man test interface. Furthermore, we include two memory strategies in this experiment, StructM, and NLM, well as compare them against the baseline NoM.
Table 15 summarises the result of asynchronous association on the action from the HMDB dataset. Actions in the HMDB dataset are relatively easier than the actions in the Pangea dataset, which reflected the comparison of the mean-step at human experts between the different datasets in Table 1 and Table 15 Additionally, the open-source MLLM QWen VL has significant gap between closed-source MLLMs Gemini-Pre Vision and GPT-4V. We speculate that this is due to the earlier version of QWen-VI., which had limited capabilities in multi-image perception. This has been effectively improved in the new QWen2-VI. version, as demonstrated by the comparison between Table 13]and and Table
G COMPARISON WITH UNFILTERED RAW DATA
MLLM Verification
Individual Attribute Concept Association Chain (Max
Mean Step)
Model
NoM
StructM
NLM
ChainM
Avg.
26.8
3.05
27.1
3.36
25.9
3.05
29.1
3.10
27.2
3.14
LLaVA-One Vision
48.0
5.30
60.6
5.80
75.4
8.91
60.8
7.03
61.2
6.76
Table 16: Comparison of whether MLLM verification on the individual attribute association. We are implemented in the LLaVA-One Vision with three memory strategies and one baseline. The compar- ison of "Yes" and "No" indicates the effectiveness of the MLLM verification in data refinement.
It is noted that involve three comprehensive steps in data refinement ensure the data quality met in the association task. We below involve the experiment for comparison of the effectiveness and difference with and without the MILM Verification step.
Table 16 and summarise the comparison of results for individual concept or paired concepts association, respectively. For comparison of individual concept association, utilize LLaVA- One Vision with four memory strategies. Additionally, for paired concepts association, we provide a broader implementation that includes open-source MI.LMs, LLLaVA-O VA-One Vision and QWen-VI.,
27

---

Published as a conference paper at ICLR 2025
Model
Concept
MLLM Verification
Paired Concept Association Chain (Max Mcan Step)
NoM
StructM
NLM
ChainM
Avg.
No
20.0
2.12
22.3
2.47
2.78
25.8
2.43
23.3
2.45
Attribute
Yes
23.0
2.65
30.5
3.13
36.8
3.94
31.8
3.35
30.5
3.27
LLVA-OnnVisioa
Affordance
No
13.5
1.91
16.0
2.09
11.8
2.06
12.5
2.05
13.5
2.00
23.3
3.111
26.3
3.14
21.5
21.3
3.29
23.1
3.28
No
18.0
2.60
18.8
2.73
20.5
2.67
17.3
2.70
18.7
2.68
Action
22.5
3.40
22.5
2.83
28.0
3.43
22.5
3.01
23.9
3.17
Attrihute
22.3
2.47
2.01
23.0
23.8
2.62
Yes
25.5
3.28
21.8
2.26
43.8
4.43
28.5
33.45
29.9
3.35
QWen2-VL. VL
Affordance
No
16.5
2.13
15.3
1.87
23.3
2.60
17.0
22663
18.0
2.311
20.8
3.13
3.12
No
21.5
3.19
19.8
2.74
27.8
3.76
23.8
3.67
23.2
3.34
Action
Yes
27.5
4.48
24.0
3.32
44.5
5.06
30.8
5.25
31.7
4.68
Same as Table I6 but for comparison on the Paired categories attribute association. No- tably, we expand the model to LLaVA-One Vision and QWen2-VL with three different concepts.
Table 17:
with three different concepts. this section, we mainly compare the results of with or without MLLM Verification, i.e., comparing the line of "Yes" to the line of "No".
From the result, we easily find that the result with MILM verification outperforms without MILM verification, i.e., the line of "Yes" larger than "No". This demonstrates that MLLM Verification effectively reduces the sample with potentially confusing annotations or those requiring powerful advanced perception abilities beyond the current capabilities of MLLMs. More interesting, while the MLLM Verification step reduces the complexity of perception on object concept understanding, also observe similar pattern between with and without MLLM verification, i.e., the tendency consistent when comparing each line.
H ETHIC REVIEW
As shown in Figure g in our human evaluation protocol, we have implemented a comprehensive Ethic Report mechanism to proactively address and mitigate potential ethical concern.. The user interface incorporates dedicated options for participants to report issues related to privacy or other ethical considerations. This facilitated through structured reporting system comprising cate- gorical buttons and an open-ended text field for additional context. This approach enables active participant engagement in ethical oversight during the evaluation phase, fostering a collaborative approach to responsible AI development. We prioritize transparency by empowering participants to articulate concerns about potential privacy infringements, algorithmic bias, or instances where the system may induce discomfort or exhibit opaque behavior. This methodology aligns with the best practices delineated by Zaldivar et al. (2019) Kennedy-May Gord (2024), which empha- sizes the criticality of integrating transparency and user feedback mechanisms to ensure fairness and accountability in machine learning systems.
Besides the ethical review implemented within human testing demo, it is crucial to emphasize that our approach builds upon the previous ethical review of the original datasets. They have un- dergone rigorous ethical review process, particularly concerning data sourcing, privacy considera- tions, and bias mitigation. This prior evaluation also sets a strong foundation for ethical safeguards.
28


Reviews:
[{"review_id": "komBHzm6RE", "reviewer": "Reviewer_Lt2g", "summary": "This work proposes a new benchmark testing the zero-shot association ability of MLLMs. Association origins from object concept learning, where the task is to connect observations with previous practice memory by identifying the underlying principle. For example, images of fresh apples,  oranges, vegetables could be connected through the adjective \"fresh\". It is a fundamental capability for humans. The proposed benchmark leverage previous datasets of object concept learning and is created in an annotation-free way. Basically, the labels in datasets of object concept learning directly provide the underlying principle (concept) that could connect objects.\n\nThe authors designed different settings to test MLLMs' zero-shot association ability: single-step vs multi-step, synchronous vs asynchronous. According to the reported results, all the leading MLLMs show a gap in terms of the association ability, compared to humans.", "questions": "1. I don't see a clear argument on why sticking to zero-shot setting? As there is a practice memory involved, is it also naturally similar to few-shot?\n2. Why does the study only focus on MLLMs? Is it because the object concept learning dataset is for multi-modal initially? How hard is it to make a text-only corresponding benchmark?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**originality**\nThe originality is good. The work proposed and will open-source a new benchmark for MLLMs, by transforming a previous ML benchmark into a LLM benchmark.\n\n**significance**\nIt shows all MLLMs have an obvious gap vs humans. In this sense, the benchmark is able to evaluate and push an overlooked capability towards AGI.", "weaknesses": "**quality**\nThere are two aspects that improvements could be made on. First, regarding error analysis, more insights are preferred. For example, by \"limited perception capability\", is it due to the limitation of image encoder/resolution or something intrinsic to LLMs. Most public MLLMs are composed of a separate image encoder+adaptor and the main LLMs. Some ablation studies on this aspect are preferred. Second, checking the correlation between this new benchmark and existing benchmarks is preferred. For example, if the performance on this new benchmark is strongly correlated with a weighted sum of those on some existing benchmarks, we could better know how to improve such a capability. If no correlation, this work might point out an overlooked dimension, which could also motivate more related benchmarks to be created.\n\n**clarity**\nThe presentation could be improved a bit, by adding more clear examples. For example, Figure 7 in the context is better in explaining the exact task than Figure 1. And adding figures for the annotation-free transformation from object concept learning datasets can better explain the exact dataset building process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0d6YW1UWCj", "reviewer": "Reviewer_8My5", "summary": "This paper proposed an evaluation benchmark, which evaluates multimodal large language models' performance on predicting association in three scenarios: single-step association, synchronized multi-step associations, and asynchronized multi-step associations.", "questions": "1. Could the authors explain the input and expected output of the single-step \"deduction\" task? Since the benchmark is claimed to evaluate MLLMs' ability in association prediction, whether deduction is part of the benchmark? In addition, could the authors explain the relation between the \"association\" and \"deduction\" tasks?\n\n2. For the results in Figure 3, since the human expert could achieve hundreds rounds of association, could the authors explain how the length of association is evaluated? Specifically, is the benchmark contains the ground truth of hundreds rounds of associations? Is each prediction on the next association necessarily requires all the previous memories? Are those MLLMs prompted with such long-term association or with only current inputs?\n\n3. What would be the prompting format for such tasks. Are those images concatenated as a single image or separately input into the prompt? Are those images interleaved with proper textual instructions, so that MLLMs could understand the intention correctly? In addition, is the proposed tasks formatted as multi-choice question-answering tasks, where generated answers could be well mapped to ground-truth?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. This paper proposed a novel task and a perspective on MLLMs.\n\n2. Various MLLMs are tested on the benchmark.", "weaknesses": "1. The definition of the task \"deduction\" in Table 1 could be better explained and illustrated like the \"association\" task in Figure. 2.\n\n2. It would be essential to include human performance in Table 1 to understand current gap between MLLMs and human. In addition, the authors might consider including more recent and powerful GPT-4o performance in Table 1.\n\n3. For Table 1, since MLLMs' performance on the association task is rather high (around 80\\% accuracy), there could be concerns about the potential improvement MLLMs can achieve for future works. To understand such, human performance might be a valuable baseline to refer to.\n\n4. The evaluation settings in the synchronous association task in Figure 3 could be difficult to understand. The authors might consider better explaining their inputs and expected outputs and their metric calculation as some pseudo-code.\n\n5. The performance comparison in Figure 3 is not very insightful, which might lacks proper explanations about why MLLMs are significantly inferior to human judgement. Such significant gap could raise concerns about the effectiveness of the proposed evaluation protocol.\n\n6. The illustration of Figure 6 is not very self-explainable, while its textual explanations in context are also very brief, which could hinder readers understanding of the major concerns in current MLLMs on such tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9TRRxPQNWr", "reviewer": "Reviewer_6DyH", "summary": "This paper proposes a novel benchmark aimed at evaluating the association capabilities of Multi-modal Large Language Models (MLLMs), a crucial yet often overlooked aspect of human intelligence. The authors formulate a specific association task based on adjective and verb semantic concepts and introduce an innovative annotation-free method for constructing these tasks, minimizing the reliance on expensive data annotation. They implement a rigorous data refinement process to enhance dataset clarity and present three levels of association tasks: single-step, synchronous, and asynchronous. The investigation covers a wide range of MLLMs, including both open-source and closed-source models, exploring various memory strategies and involving human experts. Findings reveal a significant performance gap between current MLLMs, including state-of-the-art models like GPT-4V, and human capabilities in association tasks. It implies that this benchmark could advance future MLLM research.", "questions": "1. **Rationale for Focus**: What is the rationale behind selecting adjectives and verbs as the focal point for your semantic concepts in the association tasks?\n\n2. **Bias Mitigation**: Regarding equation (1), where \\(z_{ij}\\) is constrained to 0 or 1, could this introduce bias from human evaluations? How can this bias be effectively removed from the assessment?\n\n3. **Overlooked Capability**: Can you elaborate on why you believe the association ability of MLLMs has been overlooked in previous research?\n\n4. **Novelty Clarification**: How does your proposed benchmark differentiate itself from existing benchmarks assessing association and reasoning capabilities in language models?\n\n5. **Bridging Performance Gaps**: Given the performance gap observed, what advancements do you foresee as crucial for future MLLMs to approach human-level performance in association tasks?\n\n6. **Resource Considerations**: How can researchers with limited resources effectively utilize your benchmark, considering its resource-intensive nature?\n\n7. **Model Selection Criteria**: What criteria informed your selection of specific MLLMs for this study, and how do you view these models in the context of the current state of MLLM research?\n\n8. **Future Implications**: How do you envision your findings influencing the design of future MLLMs and the benchmarks used for their evaluation? What next steps do you recommend in this line of research?\n\n9. **Evolving MLLM Performance**: As MLLMs continue to evolve and improve in association tasks, how do you anticipate the relevance of your findings will change?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **Originality**:\n   - The benchmark specifically targeting association capabilities represents a unique contribution to the MLLM literature. While benchmarking is prevalent, focusing on association adds a novel dimension to the evaluation of language models.\n\n2. **Quality**:\n   - The authors' annotation-free construction method for association tasks is a practical innovation that alleviates the common challenges associated with extensive manual data labeling, enhancing the quality and usability of the benchmark.\n\n3. **Clarity**:\n   - The paper is well-structured and clearly articulated, with straightforward definitions and explanations of association tasks. This clarity facilitates comprehension and accessibility for a broad audience.\n   \n4. **Significance**: \n   - By highlighting the substantial gap between MLLM performance and human intelligence in association tasks, the paper underscores the importance of developing models that can better mimic human cognitive capabilities. This sets the stage for future research efforts in enhancing MLLMs.", "weaknesses": "1. **Limited Novelty**:\n   - The benchmarking of association capabilities is not entirely novel. Similar efforts can be found in previous research focusing on common sense reasoning, such as \"CommonSenseQA\" and \"HellaSwag,\" which also evaluate reasoning and associative capabilities.\n\n2. **Subjectivity in Tasks**:\n   - Association tasks can be influenced by subjective interpretations, yet the paper does not sufficiently address how such subjectivity is mitigated. Discussion around the prompt design for MLLMs—especially concerning analogy tasks—could enhance the validity of the results.\n\n3. **Resource Intensiveness**:\n   - The comprehensive nature of the study, involving multiple models and memory strategies, raises concerns regarding reproducibility. The resource demands may hinder wider adoption among researchers with limited computational resources.\n\n4. **Experimental Design**:\n   - The rationale for selecting specific MLLMs could be more thoroughly explained, and comparisons with existing benchmarks would strengthen the justification for the new benchmark's necessity.\n\n5. **Discussion Depth**:\n   - The discussion section could delve deeper into the implications of the findings, particularly regarding practical applications and theoretical contributions to AI and cognitive modeling.\n\n6. **Performance Gap Exploration**:\n   - While the performance gap between MLLMs and human intelligence is significant, the paper could provide more in-depth analysis of the causes and potential paths toward bridging this gap.\n\n7. **Bias in Dataset Annotation**:\n   - The paper lacks clarity on how biases in the initial dataset annotations are addressed, which could affect the robustness of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ANgFoi86R", "reviewer": "Reviewer_hJvy", "summary": "The paper proposes a benchmark, using an annotation-free construction method to transform general datasets. The benchmark includes three levels of association tasks: single-step, synchronous, and asynchronous associations. The authors conduct extensive experiments involving multiple open-source and closed-source MLLMs, including state-of-the-art models like GPT-4V, and compare their performance with human experts.", "questions": "How might the authors' findings on the associative capabilities of MLLMs influence the design of future models, particularly in terms of memory and reasoning architectures?\n\nAre there any specific areas of application where the authors believe the current gaps in associative capabilities are particularly problematic, and thus, warrant immediate attention in research?\n\nHow did you ensure that the limitations of the original datasets did not significantly affect the results?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers a fresh approach to assessing MLLMs by focusing on their associative abilities, which is a novel contribution to the field. The annotation-free construction method for association tasks is innovative and has the potential to simplify the creation of benchmarks in this area.", "weaknesses": "The evaluation is limited to MLLMs’ zero-shot ability in association tasks across adjectives and verb semantic concepts. This may not fully capture the complexity of real-world scenarios where MLLMs are expected to perform.\n\nWhile the paper analyzes failure cases in the association process, it could have provided a more in-depth understanding of why these failures occur. \n\nThe paper focuses on single-step, synchronous, and asynchronous associations, which are complex tasks. However, it might not fully capture the nuances of human associative learning, which often involves more gradual and contextually influenced processes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)