Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
Authors: Yuansen Liu
Haiming Tang
Jinlong Peng
Jiangning Zhang
Xiaozhong Ji
Qingdong He
Donghao Luo
Zhenye Gan
Junwei Zhu
Yunhang Shen
Chaoyou Fu
Chengjie Wang
Xiaobin Hu
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Multimodal Large Language Models
Primary area: datasets and benchmarks

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks. 
However, their capacity to comprehend human-centric scenes has rarely been explored, primarily due to the absence of comprehensive evaluation benchmarks that take into account both the human-oriented granular level and higher-dimensional causal reasoning ability. 
Such high-quality evaluation benchmarks face tough obstacles, given the physical complexity of the human body and the difficulty of annotating granular structures.
In this paper, we propose Human-MME, a rigorously curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric scene understanding. Compared with other existing benchmarks, our work provides three key features:
**(1) Diversity in human scene**, spanning 4 primary
visual domains with 15 secondary domains and 43 sub-fields to ensure broad scenario 
coverage.
**(2) Progressive and diverse evaluation dimensions**, evaluating the human-based activities progressively from the human-oriented granular perception to the higher-dimensional multi-target and causal reasoning, consisting of eight dimensions with 19,945 real-world image question pairs and an evaluation suite.
**(3) High-quality annotations with rich data paradigms**, constructing the automated annotation pipeline and human-annotation platform, supporting rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. Our benchmark extends the single-person and single-image understanding to the multi-person and multi-image mutual understanding by constructing the choice, short-answer, grounding, ranking and judgment question components, and complex question-answer pairs of their combination. The extensive experiments on 20 state-of-the-art MLLMs effectively expose the limitations and guide future MLLMs research toward better human-centric image understanding and reasoning. Data and code are available at [https://github.com/Yuan-Hou/Human-MME](https://github.com/Yuan-Hou/Human-MME).

Published as a conference paper at ICLR 2026
HUMAN-MME: A HOLISTIC EVALUATION BENCH- MARK FOR HUMAN-CENTRIC MULTIMODAL LARGE LANGUAGE MODELS
Yuansen Liu Haiming Tang**
Jinlong Peng"*
Jiangning Zhang²
Xiaozhong Ji
Qingdong He? Donghao Luo2
Zhenye Gan
Junwei Zhu²
Yunhang Shen
Chaoyou Fu Chengjie Wang?
Xiaobin Hu
Shuicheng Yan
National University of Singapore
Tencent Youtu Lab
Nanjing University
ABSTRACT
Multimodal Large Language Models (MI..Ms) have demonstrated significant ad- vances in visual understanding tasks. However, their capacity to comprehend human-centric scenes has rarely been explored, primarily due the absence of comprehensive evaluation benchmarks that take into account both the human- oriented granular level and high-quality evaluation benchmarks face tough obstacles, given the physical com- plexity of the human body and the difficulty of annotating granular structures. In this paper, we propose Human-MME, rigorously curated benchmark designed provide a more holistic evaluation of MLLMs human-centric scene under- standing. Compared with other existing benchmarks, our work provides three key features: (1) Diversity in human scene, spanning 4 primary visual domains with 15 secondary domains and 43 sub-fields to ensure broad scenario coverage. (2) Progressive and diverse evaluation dimensions, evaluating the human- -aaaddd--ddddd ac- tivities progressively from the human-oriented granular perception to the higher- dimensional multi-target and causal reasoning, consisting of eight dimensions with 19,945 real-world image question pairs and evaluation suite. (3) High- quality annotations with rich data paradigms, constructing the automated an- platform, supporting rigorous manual la-
causal reasoning ability. Such
notation pipeline and numan-annotatioo beling by expert annotators to facilitate precise and reliable model assessment. Our benchmark extends the single-person and single-image understanding to the multi-person and multi-image mutual understanding by constructing the choice, short-answer, grounding, ranking and judgment question components, and com- plex question-answe pairs of their combination. The extensive experiments on 20 tfate-of-the-as MLLMs effectively expose the limitations and guide future MILMS research toward better human-centric image understanding and reason- ing. Data and code are available https://gttub..co//
INTRODUCTION
Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in perceptual understanding and reasoning for general comprehension tasks. Among various types of scene data, human-centrie images represent particularly critical domain due to their prevalence in real-world data (Wang et al., 2024; Gkioxari et al., 2018). Compared to general image understanding, human-centric image understanding imposes greater challenges on models. These tasks require not only fine- iineigninedd perception (e.g., eyebrow, accessories) and recognition of physical complexity, but also highly sophisticated causal reasoning abilitics (Xiao & Yamasaki, 2024; Yang al., 2022). Consequently, thorough evaluation of the capabilities and limitations of existing MLLMs within this domain is critical. Such an investigation is fundamental to progress in
Yuansen Liu, Haiming Tang, and Jinlong Peng are co-first authors Xiaobin Hu corresponding author

---

Published as a conference paper at ICLR 2026
Understanding
Body Understanding
Human- Interaction Understanding
O Wiich one purceiic the Fucanclacc insicioniiirap
Highteer healing piva na Rig stait fouiing poou
D06, S08 490 EX5
Pleaae poeide bounding boe the otteet with rnnecctin
Reswnn bobo
Young tice
Pasas lics
pwentr Judo bndourtfoom
delecc ned sake waaking
Gvep th
e Pewien para the sig Bud www.deee. hoodl retling with
Asswer: Panch
Mutt--mage Underatanding
Reasoning
Wast
Paxss bo boundin
proedde baned on the couit. of linues
deser piin of huraaveeaz inuraciion
tw peaan
too of motio bay
aiteglass
Our Wherigght
(g run in paraarn
e EE then semecse
light tie plaas
nod foo chose gen the
orkonen
(111, RLA
2 The dentavr risectort
Toa everyne in the ima
dapxi Kave
A-C--b-k
chiid
Intention Discrimination
Emetion Discrimination
Causust Discrimination
Eiattrnory bytttty througg deliberzte
arthualism...
@ Moves setra the and vadyae
kmocis for sorwore image
The of intecton the mntrure belibente
through
The mee sppewirng looce
Adeip of oecer fooaed
y soinding phapages
The cisdall aegpesse ng. cniinniion
emprding with natanal
cnoppctd chalange...
Thaindikkall tusi aounkaly cauped instummett...
settngg.
test
inabals et iisandoo for
The coppereinn
reporeed
contennent entreednga anlovdend
of cenemm Proorvatio tnyinng proeess
and
inea...
Arsweer Pustie futtra iud
Question Component
Choice
Short-Answer
Bounding Box
Ranking
Judgment
Evaluation Metric
Acourary
Figure I: Overview of Human-MME: The progressive and diverse evaluation dimensions can be divided into eight aspects from the human-oriented granular dimension perception (e.g.. face, body, human-object interaction understanding) to higher-dimension reasoning (e.g.. multi-image and mulli-pprson understanding, intention, emotion, cause discrimination).
both theoretical and the evolving human-oriented framework of MLLMs. However, existing bench- marks rarely attempt explore the fine-grained human-centric image understanding, but predom- inantly focus on the general content comprehension. These benchmarks usually suffer from three key limitations in centric scenes: (1) Overly simplistic evaluation settings that inadequately represent the full spectrum of human-centric activities. (2) Lack of comprehensive dimensions to take into account both the granular level and higher-level spatial and reasoning perception. (3) Low annotation quality and limited question-answe ticated and diverse reasoning tasks. Such deficiencies preclude holistic evaluation of the MLLMs' inherent capacity for human-centric scene understanding.
paradigms to handle broader spectrum of sophis-
To address these limitations, we propose Human-MMin the first comprehensive benchmark for eval- uating MLLMs toward human- image scene understanding via progressive and diverse eval- uation dimensions from granular dimension perception and higher-dimension reasoning, as shown in Figure 1. Compared to the existing benchmarks, benchmark distinguishes itself through three key innovations: (1) Diversity in human scene. The benchmark consists of 43 distinct and fine- grained visual scenarios to support the comprehensive human scene perception. (2) Progressive and diverse evaluation dimensions. The evaluation is structured to progressively assess MLLMs' ca- pabilities from granular human-oriented perception to complex spatial and causal reasoning, which is quantified across eight dimensions via dataset of 19,945 real-world image-question pairs and a comprehensive evaluation suite. (3) High-quality annotations with rich data paradigms. Human- MME advances beyond single-image analysis to multi-image, multi-person mutual understanding and introduces suite of tasks: choice, short-answer, ranking, grounding, identifying, and judgment. These high-quality tasks are facilitated by our annotation platform, which enables expert annotators to efficiently verify and correct automatically annotated labels through intuitive operations.
To assess Human-MME effectiveness, we perform a comprehensive benchmarking using 20 state- of-the-art art open-source source and proprietary MLLMs. Our benchmark reveals several interesting findings in Section 3.3. We hope that benchmarking results, evaluation findings, and benchmark itself will inspire and guide future research toward developing more capable and robust human- oriented MLLMS. In summary, our key contributions are follows:
2

---

Published as a conference paper at ICLR 2026
2.1 Data Collection
Free stock media sites pexels
Openssurce Datasel: HICC-DET
More than person
Image set to Be processed
pixabay
YoloV11
Image hash De-duplicate
Head Detection Body Octection
2.2 Automated Annotation Pipeline
2.3 Manual Adjustment De-duplication automattic
Stup Body Bux
Step Object Box
Correction
nice
di Douncing
FlLLl
HOI Cijjett
DONO
beffee Extimation
Focs Attibate
Sounding
Face Aass Buden
Facial
highest readaiiiyy
Herd Modily Annotation
Step Attribute, Weasing and Object
Pace Landrorks
Para Rar
Bourding
Vip
Qwen2. 5.VL
Gorend Attrisutes
highest complectty
Delete Annotation
farn cabk Faine
Whoring K411
Stop Ereation, Intention Cassee
Oienn2.5V Oheen3
Emition
(ersciate)
HQ3
descrpt01 Von
descripions
Cousc
Figure 2: Curation of Human-MME consists of: (1) Data collection provide images for anno- tation and QA generation (Section 2.1): (2) Automated annotation to provide the original feature for each person in image (Section 2.2); (3) Manual adjustment ensure annotation quality for
final answer construction (Section 2.3); (4) Construction of question- question-answer answer pairs using the features extracted in the image (Section 2.4, not shown in the figure).
We introduce Human-MME, novel human-centric image benchmark for MLLMS that empha- sizes progressive assessment across eight dimensions, from fine fine- grained human comprehension to higher-level intricate spatial and causal reasoning perception.
We build up user-friendly annotation pipeline and platform facilitating rich and high-quality data paradigms, covering 43 fine-grained visual scenarios and 19,945 real-world quege-quetioon pairs. The type of data paradigms breaks the restriction of single-image, single-perse perle- and single- question paradigms, and extends to multi-image mutual understanding and sequential complex questi pairs.
To the best of our knowledge, we are the first to conduct a holistic evaluation MLLMs on human-centric image perception in progressive manner that considers both fine-grained percep- tual dimensions and higher-level reasoning. Extensive experiments on 20 ofate-of-the-arr MLLMs effectively expose current limitations and provide guidance for future models toward improved human-centric image understanding and reasoning. All data, the annotation platform, and the code will be publicly released to support reproducibility.
CURATING HUMAN-MME BENCHMARK
2.1 DATA COLLECTION
We collect a total of 54,122 images from Pexels and Pixabay, addition, the HICO-DET (Chao et al., 2018) dataset, which contains total of 47,776 images, has not been reported as training data in any of the models we evaluated, which makes suitable for use in our benchmark.
For images from free media sites, we use pre-trained YOLOv11 (Jocher & Qiu, 2024) models to detect face and body bounding boxes and only retain the ones with detection results. To reduce redundancy, the extracted images are further de-duplicated using image hashing (Buchner, 2021). For the HICO-DET (Chao et al., 2018) dataset, we select images containing more than one person and compared the original annotations with automatically generated ones for further filtering. After further annotation and de-duplication, we obtain 8,010 images from the free media sites and 3,755 images from the open-source dataset, with total of 16,765 high-quality images.
3

---

Published as a conference paper at ICLR 2026
2.2 AUTOMATED ANNOTATION PIPELINE
After collecting the image data, we complete the automated annotation of these images in five steps as shown in Figure 2. This automated annotation pipeline produces 13 different types of bounding boxes, 42 binary facial features, and four person-level attributes. It also enumerates eight categories of clothing worn by the person (covering details such as color, type, and name) and captures inter- actions between the person and objects (including the interacting body part, action, object name). In addition, it extracts three types of higher- digee--dimeniional features. The symbolic representation and interpretation of the extracted features can be found in Table 8. Here we explain step by step how each feature of person in the image extracted. Detailed information about automated annotation pipeline can be found in the Appendix B.
Face bounding box, body bounding box and body parts bounding boxes. In step 1, we use a pre-trained YOLOvI detector (Jocher & Qiu, 2024) produce candidate body and face bound- ing boxes. In parallel, we apply DWPose (Yang et al., 2023) to each image to obtain whole-body pose estimates per person instance with 134 keypoints. After establishing the correspondence be- tween pose estimation and face/body bounding boxes using geometric relationships, for each pose estimation, align the body and face boxes and extract boxes for both hands and both feet.
General attributes, wearing and human-object interactions (HOI). In step 2. we use the matched bounding box obtained from step 1 to isolate each person instance within the image. For each instance, we query Qwen2.5-VL-772 A set of factual attributes including age group, gender, race and emotion.
(Bai et al., 2025b) following a JSON template to extract: (1)
(2) Each clothing item with their types, colors and names. (3) Interaction relationship with objects including: body parts that conduct the interaction, the verb phrases of the actions and names of the objects.
Bounding boxes for HOL. In step 3, the original image and the names of HOI objects are given to Grounding DINO (Liu et al., 2023), which predicts the bounding boxes for HOI objects. The bounding box of an object is also combined with the bounding box of body parts to refine the body part information of the HOI.
Facial attributes and bounding boxes for facial parts. In step 4, for each individual. the corre- sponding face region is passed FaceXFormer (Narayan et al., 2024) for facial attributes and land- marks recognition. Facial attributes from FaceXFormer are 40 binary values, for example "Bags Under Eyes" and "Bangs", in the same format as CelebA (L.iu et al., 2015). In addition, two binary values are extracted from the head pose prediction. Bounding boxes for facial parts including eyes, eyebrows, mouth and nose are extracted from the 68 landmarks predicted.
Intention, Emotional analysis, cause (past) and consequence (future) narratives. In step 5, each person appearing in the image is sequentially highlighted and queried by Qwen2.5-VLL VLL-2B (Bai et al.. 2025b) with two prompts. The first prompt requests a detailed analysis of the individual's emotions and thoughts to produce the intermediate output. The intermediate output is provided to Qwen3 (Yang et al., 2025) yield an identity-neutral emotional analysis. The second prompt seeks a comprehensive description of the person's behaviors, interactions, and any plausible intentions, resulting in behavior description. This description is passed Qwen3 (Yang al., 2025) to produce the final intention analysis, past-cause description and future-consequence description. This helps prevent same-mode bias by using an independent text-only model for final text generation.
2.3 MANUAL QUALITY REVIEW AND ADJUSTMENT
To refine the automatically generated annotations, we design a custom Gradio-based (Abid et al., 2019) interface supporting cluster-level de-duplication and instance-level correction. A detailed description of the interface and workflow provided in Appendix C.
At the cluster de-duplication stage, experts select representative and diverse samples for each group of similar images that have been auto-annotated. They are expected to find the images with best au- tomatic annotation quality, best image quality and highest complexity. This step helps maximize image diversity while better utilizing annotation results on similar images. At the instance correction stage, experts adjust bounding boxes and attributes with real-time visualization. Experts can decide whether to accept or discard each image and make detailed modifications to the automatic annota- tion results, especially the features that rely on Owen2.5-VI-77B to annotate. This step contributes
4

---

Published as a conference paper at ICLR 2026
to higher annotation quality and eliminates potential same-model bias (Panickssery et al., 2024) of Qwen models which are among the models evaluated.
2.4 QUESTION-ANSWHH DESIGN
We design 21 question types based on the annotated features, covering eight dimensions: Face Un- derstanding (FU), Body Understanding (BU), HOI Understanding (HU), Multi-Image Understand- ing (MIU), Multi-Person Reasoning (MPR), Intention Discrimination (ID), Causal Discrimination (CD), and Emotion Discrimination (ED). The answer formats include Choice, Bounding Box, Short- Answer, Ranking, Judgment, as well as composite forms such as Judgment combined with Short- Answer, Judgment combined with Bounding Box, and Short-Answer combined with Bounding Box. Here we explain how to construct each of the five question components. Detailed construction logic and examples for eight dimensions and 21 question types are provided in Appendix D)
Choice questions, except for Causal Choice questions, have only one correct answer. The distinctive feature of Causal Choice questions is that they require selecting one past event (the cause) and one future outcome (the consequence) from four options, When constructing incorrect options, we prioritize confusing patterns. For example, in Wearing Choice, we select only the same type of wearing as incorrect options. In HOI Choice, we deliberately confuse left and right hands as body parts conducting interaction. In Emotion Analysis Choice, we choose emotion analyses from images where the subject's mood is broadly similar as incorrect options.
Bounding Box questions require MILMs to provide bounding boxes coordinates for facial parts, body parts, or HOI objects within images. While dedicated detection models exist to address such problems, these tasks serve as the most direct measure of MLLMs' image-text grounding capability. Furthermore, many questions in this benchmark like Judgment Bounding Box and Identify Open HOI involve complex or indeterminate conditions that open-set object detector cannot handle.
Short-Answer questions open-ended. asking for brief responses primarily regarding the names and colors of clothing items, the names of HOI objects, and general attributes such as gender, age group, emotion, and race. The ground truth for these nouns or adjectives is derived from automated labeling and manually refined by experts to minimize same-model bias (Panickssery et al., 2024).
Ranking questions provides three features and four images, and requires the model rank the im- ages according to the number of features present in the people shown. Multi-Face focuses on facial features, while Multi-Wearing focuses clothing. This tests the model's ability to simultaneously cross-check multiple features across multiple images.
Judgment questions are combined with other types of questions and require MLLMs to proceed with an answer only if certain conditions are met; otherwise, it should refuse to answer. For example, a Judgment Bounding Box question asks the model to check whether there is person in the image with specific feature; if so, it must provide the bounding box of certain body part of that person; not, it should return [-1, -1, -1, -1] as the answer. Such questions are essentially about true/false judgments, but here they are designed to target hallucination of MILMS on human-centric problems.
Different evaluation metrics are used for each question component. We evaluate the perfor- mance of the models using metrics tailored to each component of question (detailed definitions are provided in Appendix G). For Choice questions, accuracy is reported: for Short-Answer questions, composite score combining BERT Fl (Zhang et al., 2020), embedding cosine similarity (Wang et al., 2020), and keyword coverage used; for bounding- box questions, Intersection-over cereetioonove-nnniioo (loU) is used; for Ranking questions, Kendall's Tau (Kendall, 1938) is reported; and for judgment questions, Fl score balances precision and recall.
2.5 STATISTICS AND ANALYSIS
Image Statistics. Human-MME contains a diverse collection of images representing wide range of real-world scenarios. Figure 3(a) shows sunburst chart of image content, organized into four main domains: Daily Life, Work, Study, and Entertainment, with subcategories covering typical situations. For example, the Office subcategory under Work includes Meetings, Desk Work, and Reports. Figure 3(b) presents the distribution image resolutions, with over 30% of images having 4K (3840x2160) resolution higher. Figure 3(c) shows the distribution of the number of people
5

---

Published as a conference paper at ICLR 2026
secnchh dog umbre
130
bench rotor cyc
15 1%
35.75
$196
1712
Snowboard
knife e boo donun
hand
kite top chatt
700 200p
5.%
Balow
skis Cell O cake
20.14
508
bicye Katbebar
bottle an bothbrushh h
Resolutionn
Narber Penaan
Cloud
1150
Bacy tectentt witk
16
Reasing
1350
Muli Pertar Roats
piggenrrt sounding
Undersarndrgg
150
Muli lmoupe indrrsund
250
2300
143
291
290
Cazgerica
Disribuuiion
Diaribationn
worda)
Figure 3: Human-MME demonstrates rich diversity: Images: (a) shows a sunburst chart of image content across four domains and subeategories; (b) indicates image resolutions ranging from below 480p to over 4K; (c) presents the number of people per image, from single- to multi-person scenes; (d) illustrates word cloud of HOI objects, covering interactions with hundreds of distinct objects.
QA Pairs: (e) illustrates multiple question types distributed across eight reasoning dimensions; (f)
shows the lengths of questions, capturing variability question complexity.
Table I: Comparison to the existing benchmarks involving human features.
Benchmark
Modality
4QA
Formats
Fine-grained grounding
Face Features
Body Features
Human-Otjeet Interaction
Multi- image
Multi-
High-level abstract features
MMBench
Image
Choice
X
X
MME
Image
2.8K
TF
X
X
X
V
X
Images
X
X
V
X
J
X
IIV MY-MMBench
Video
8.7K
ChoicoOppenendeel
X
X
X
X
J
V
Human' VBench
Video
2.1K
Choice
X
V
X
X
Face Human Bench
Image
2.7K
Choice
X
X
/
/
HumaniBench
32K
Choicoopee-endeei
X
X
X
/
/
ChoicelOpen ended/
Hunsn-MME (Ours)
Image
TF/BBooRankingg
image, dominated by single-person scenes, with some multi person scenes as well. Figure 3(d) illustrates word cloud of HOIs, depicting interactions between humans and hundreds of different objects, such as skateboards, books, laptops and knives.
QA Statistics. Figure 3(e) demonstrates the distribution of question types across different dimen- sions. Among all dimensions, Multi-Person Reasoning contains the largest number of questions and the richest variety of answer formats. Choice questions are the most frequent answer format, appear- ing in all eight dimensions, with each question type containing at least 340 questions. Figure 3(f) presents the distribution of question lengths, reflecting variability in question complexity across dimensions. Questions in high high-reasoning dimensions, such as Intention Discrimination. Causal Dis- crimination, and Emotional Discrimination, are generally longer, typically exceeding 100 words; in particular, Causal Discrimination questions have average length of 200 words.
Table compares Human-MME with existing human-related benchmarks. Human-MME provides larger scale and more diverse tasks, with richer formats and fine-grained annotations. It covers face and body features, HOI, multi-image and multi-person scenarios, and high-level abstract reason- ing. Unlike Face-Human- (Qin et al., 2025), which lacks fine-grained body details such as spatial grounding and has only one question format, and HumaniBench (Raza et al., 2025), whose bounding box tasks are searse-grained and overlooks detailed human features, Human- MMEE the first human-centric benchmark to offer truly comprehensive and fine-grained evaluation. Further comparison methodology of the benchmark construction can be found in Appendix A.

---

Published as a conference paper at ICLR 2026
Table 2: Human-MMEE scores by eight dimensions and five question components. Bold indicates the best. underline indicates the second place. Closed-souree models are ranked separately. Detailed results are provided in Appendix H.
FU
BU
HU
MIU
MPR
ID
CD
ED
Bounding Box
Choice
Short- Answer
Ranking
Judgment
Model
Avg.
23
GLM-4.5V
61.6
77.4
82.5
79.2
71.5
83.9
85.4
66.6
76.0
66.3
70.8
83.5
86.2
68.3
GLM-4. IV-9B
55.2
74.1
69.5
71.8
64.3
82.7
76.0
58.8
69.1
49.7
68.0
80.7
82.5
66.3
Qwen2.5 Qwen2.5-VL-7225 VL-
61.1
70.2
70.6
75.4
65.2
88.1
86.3
65.3
72.8
50.8
70.4
81.7
83.9
71.3
Swen2.5-VL-3222
56.2
73.3
65.3
70.7
58.2
82.9
81.1
64.9
69.1
44.9
67.9
72.7
82.4
67.0
Qwen2.5-VL-77
49.4
68.4
61.4
61.0
46.3
84.1
72.
60.9
63.0
31.7
60.1
71.0
70.7
56.5
Intern-S1
41.0
65.2
65.5
79.8
59.3
82.9
83.2
68.3
68.2
22.1
72.6
82.0
86.6
68.9
50.7
74.6
71.4
76.4
59.4
83.7
82.5
66.4
70.6
46.3
68.9
81.3
84.0
57.3
VL3-77
67.9
67.2
78.6
68.9
20.9
44.6
72.6
64.6
75.0
53.8
86.9
78.0
65.6
67.6
30.6
67.9
80.7
82.6
62.0
Llama-4-Scout
27.3
50.6
49.4
48.9
33.9
66.5
57.1
50.4
48.0
6.4
47.9
69.5
71.0
38.6
LLaVA-NeXI-72B
38.0
66.8
59.3
58.2
72.8
Ayn-vision-32B
30.9
57.2
57.1
67.9
42.8
76.2
71.8
57.4
57.7
8.9
57.7
78.5
75.7
53.8
Gemma3-278
35.1
59.9
61.2
65.3
45.1
81.5
73.0
60.1
60.2
13.8
59.4
78.4
75.4
54.5
Kimi
37.3
63.1
52.6
56.7
72.1
MiniCPM-N-445
38.9
62.6
62.4
73.5
52.1
81.5
67.8
63.3
62.8
20.2
65.0
80.7
84.0
57.9
Phi-4
29.5
48.1
48.6
34.6
29.6
62.9
38.1
46.4
42.9
5.5
45.5
62.4
54.4
19.8
Gemini-2.5-Pro
42.4
66.5
70.0
83.6
48.6
79.4
86.1
64.5
67.6
23.5
72.4
83,9
90.9
72.0
GPT-4o
28.8
58.8
59.8
74.7
34.4
79.2
76.2
52.7
58.1
11.5
57.6
78.3
83.8
48.6
GPT-5
34.4
67.8
71.1
75.8
43.1
82.3
89.2
42.6
63.3
25.2
65.8
77.5
85.8
41.7
GPT-5-mini
30.6
66.3
67.4
76.4
41.3
79.4
81.7
39.9
60.4
21.1
63.4
76.9
87.4
49.1
3 EXPERIMENTS
3.1 EXPERIMENTAL SETTING
Evaluated MLLMs. We conduct evaluations on several vision-language models, including GLM- 4.5V (Hong et al., 2025), GLM-411V-9B (Hong et al., 2025), Qwen2.5-VL. (72B, 32B, 7B, (Bai al.. 2025b)), VL3-788 (Zhu et al., 2025). Intern (38B. 241B) (Wang et al., 2025), Intern-S1 (Bai et al., 2025a), MiniCPM-V-4.5 (Yao et al., 2024), Gemma3-27B (Kamath et al., 2025), LLaVA-NeXT-722 (Chen Xing, 2024), Aya-vision-328 (Dash et al., 2025), Kimi-VI. A3B (Du et al., 2025), Llama-4-Scout (Meta AI, 2025), and Phi-4 (Abouelenin et al., 2025). More details of these open-source models are provided in Appendix E, In addition, we also evaluated four closed-source models: Gemini-2.5-Pro (Comanici et al., 2025), GPT-44o (Hurst al., 2024), GPT-5, and GPT-5-mini (OpenAI, 2025).
We use VLLM (Kwon et al., 2023) to deploy the open-source MLLMs under evaluation, and for the closed-source MILMMs we call their official APIs. To ensure structured and consistent outputs from the evaluated MILMs, we employ format-enforcing prompts for each question type (see Appendix F). Model outputs are parsed using regular expressions to extract the final answers. This will also minimize the bias introduced by the generation style of MLLMs.
Implementation Details.
3.2 RESULTS
Table 2 compares performance across the eight evaluation dimensions. Models trained with explicit grounding data consistently lead on perception-orienter tasks. In particular, GLM-4.5V achieves the highest scores on most dision-heavy dimensions such as Face Understanding, Body Understanding, HOI Understanding and Multi-Person Reasoning, reflecting the value of its precise visual element localization. Qwen2.5-VI. VL--2 performs almost well in these areas and surpasses all others on higher-level reasoning tasks such as Intention and Causal Discrimination. By contrast, models without explicit grounding training, such as Gemma3-27B, Aya-vision-32B and Llama-4-Scout, lag far behind in localization-hent heavy tasks. An interesting exception is Intern-S1, which, despite lacking fine-grat fine-grained grounding ability, reaches the top score in multi-image understanding and emotion discrimination, indicating strong high-level understanding abilities. Among the two closed-source models tested, Gemini-2.55Prr consistently outperforms GPT-40. Moreover, Gemini-2.5. 2.5-PPo shows

---

Published as a conference paper at ICLR 2026
Bounding Box
Choice r= 0.78, p 0.0138
Short-answer
r= -0.00, p 0.991
0.64, p 0.0.66
0.725
0.825
0.6
Composite Score
0.700
Accufacy
0.775
0.4
0.650
0.750
0.3
0.625
4
0.725
0.2
0.600
10a
101
Model (B, log scale)
(B, log scale) Judgment
Model (B, log scale)
Ranking 0.75, p 0.0188
0.23, 0.557
0.85
0,70
Medel family
Kendall's tau
Score
0.65
Intern
Owen
4
0.60
0.70
10
10
10'
10a
Model size (B, log scale)
Model size (B, log scale)
Figure 4: Correlation between model size and performance in differrnt question components. To minimize the influence of differences model architecture and training strategies, only the models ranking in the top half of the overall performance table (Table 2) are selected for this analysis. These models all belong to the GLM, Qwen, and Intern families.
pattern very similar to Intern-S1: it lacks fine-grained image grounding ability but demonstrates strong high-level understanding abilities.
Table 2 also breaks down the results by five question components. GLM-4.5V leads in Bounding Box and Short-Answer related questions, confirming its advantage spatial localization and con- sistent recognition. Meanwhile, Intern-S1 and VL3-78 excel at choice, short-answer and ranking questions but perform poorly on Bounding Box questions, highlighting their strengths in consistent human feature recognition and limits in fine-grained spatial alignment. Qwen2.5-VL-7275 achieves the highest Judgment score, shows reliable selective answering, reflecting a strong ability prevent hallucination. Among the closed-source models, Gemini-2.5-Pro excels in all aspects except for Bounding Box related questions, matching or surpassing the best open-source models.
3.3 FINDINGS HUMAN-MME
We have seven findings from this benchmark. which are listed here. More detailed analysis and discussion of these findings can be found the Appendix
Stronger scaling effects in Choice and Ranking tasks. Figure 4 shows that performance on Choice and Ranking components have stronger correlation with model size than other metries. Larger mod- els can process and integrate multiple visual features simultaneously. which enhances their ability to evaluate candidate options and order items effectively. For details, see the discussion Appendix I..
Training data influence on grounding tasks. Table 10 in Appendix E and Figure 4 show that Bounding Box performance is influenced more by the relevance and composition of training data than by model scale or model architecture. Models whose training sets explicitly include human- centric grounding examples, or that place emphasis on grounding-related supervision, achieve no- ticeably higher and more stable results. In addition, models that apply structured ourput-format alignment, such as using normalized or JSON-based bounding box representations with special- ized tokens, further benefit from improved precision and consistency. These patterns indicate that grounding fooundeng-occused focused data and clear furmut-format constraints are central factors in obtaining strong visual grounding performance. For details, the discussion in Appendix I.
8

---

Published as a conference paper at ICLR 2026
Owen2.5
Liaand
Boly Grounding
Mrend-
442
BiHand
606
6.
642
0.
6M
Gront
L.
L-Rye-
6-1
644
645
0.39
6-4
0.06
014
8:6
043
Fate Grounding
8:08
609
650
8:02
639
030
601
0.M0
0.80
8:08
038
MARTY
122
642
Mhuth 628
AdB
642
Marhh
5.22
0.2
221
80L
624
L.bre
4.599
8d
Figure 5: Confusion matrices for body and face grounding tasks. This figure presents the con- fusion matrices of three MILMs the Body Grounding and Face Grounding tasks. All images containing any overlap or ambiguity between left and right hands or feet are removed advance ensure that the evaluation focuses purely the models' ability to distinguish left-right body and facial parts.
Table 3: R, F1 of different models in choosing whether to answer abstain questions with Judgment component and their performance compared to similar task without Judgment component.
Precision
Recall
Bounding Box
J Bounding Box
Short-Answer
Short-Answer
O
GLM-4.5V
54.7
90.9
63.80-2.0)
81.6
80,16-1.5)
GLM-4.1V-99
53.0
88.7
66.3
45.0
1.7)
79.1
76.96-2.2)
Qwen2.5.VL-722
71.3
80.7(=0.9)
Qwen2.5.VI -22
53.5
89.4
67.0
46.4
43,21-3.1)
34.3
66.1(-81)
Owen2.5.VL-78
69.4(-4.4)
Intem-S1
$7.3
86.5
63.9
22.6
19.9t-2.7)
79.3
78.5(-0.8)
VL5-78L3378B
61.6
81.9(+2.4)
IntemVV1...-3388
47.1
90.8
62.0
28.7
25.8(-3.0)
77.9
78.2(+0.2)
Scoont
33.3
50.1
38.6
6.6
1.8(4.8)
62.4(.5.5)
LLaVA-NeXT-72B
36.9
89.8
52.3
27.2
24.8(-2.4)
68.8
71.1(+2.3)
Aya-rision-32B 32B
38.9
88.1
53.8
8.7
6.30-2.4)
75.8
74,40-1.4)
Gemma33-278
37.8
98.1
54.5
15.5
15.3(-0.2)
77.7(+0.7)
Kimi VL-A3B
41.4
67.0
50.4
19.5
19.6(=0.1)
73.1
72.26-0.9)
MiniCPM-V-4.5
78.6(-0.8)
Phi-4
42.2
16.9
19.8
4.1
0.7(-3.4)
66.9
59.96-7.1)
GPT-4o
6.90-3.9)
70.60-5.3)
Gemini-2.5.Po
60.1
89.8
72.0
23.8
19.1(-4.7)
82.2
80.4(-18)
Challenges in left-right discrinination for body parts. Figure 5 shows that all evaluated models experience consistent difficulty in distinguishing left from right for hands and feet, often confusing the two sides across a range of poses and viewpoints. In contrast, their left-right discrimination on facial features, including eyes and eyebrows, is markedly more reliable. This discrepancy suggests that the spatial configuration of facial components provides more stable and unambiguous cues during training and inference, whereas the variable placement and articulation of limbs introduce greater ambiguity for MLLMs. For details, see the discussion in Appendix I.
Precision-recal tradeoff in Judgment tasks. Table 3 shows that models generally reach high call and relatively low precision, often failing to abstain when no valid target exists. This indicates a tendency toward hallucination. Models with stronger mechanisms to prevent hallucination improve precision but reduces recall and sometimes leads to over-cautious refusals, reflecting a tradeoff be- tween cautiousness and faithful instruction following. For details, see the discussion in Appendix I.

---

Published as a conference paper at ICLR 2026
Extra Judgment component reduces task performance. Table 3 in Appendix I further shows that adding a Judgment component to questions lowers performance. The need to match two specified features before answering increases the complexity and reduces accuracy despite extra hint provided. For details, see the discussion in Appendix I.
Hierarchy of discrimination difficulty. Table 2 in Section 3.2 shows that accuracy generally fol- lows the pattern Intention > Cause > Emotion, reflecting an increasing level of abstraction and difficulty across these discrimination tasks. For details, see the discussion in Appendix I.
Table 4: Total refusal rate (%).
Table 5: Refusal rate due to policy (%)
FU
BU
HU
MIU
MPR
ID
CD
ED
FU
BU
HU
MIU
MPR
ID
CD
ED
Model
80
A2
Avg.
Model
Avg.
Gemini-2.5.Po.
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Gemini-2.5-Pro
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
GPT-4o
GPT-4o
0.0
0.0
0.0
GPT-5
5.1
0.0
0.0
0.6
2.6
1.2
0.0
16.6
3,3
GPT-S
2.1 0.0 0.0
0.3
1.7
0.0
0.0
7.3
14
GPT-5 mini
10.2
0.0
0.6
6.1
4,0
0,2
25.5
5.8
GPT-5-mini
3.2
Table 6: Bounding box IoU comparison between the proprictary models, best open-source model (GLM-45VV), and slightly weaker open-source model (Intem for face and body parts.
Body
Model
Left
Right
Left
Right
Whole
Left
Right
Left
Right
Nose
Mouth
Whole
hand
band
foot
foot
body
eye
eye
brow
brow
face
GLM
0.497
0.513
0.394
0,400
0.964
0.540
0.317
0.447
0.513
0.594
0.719
0.887
Intern VL3,5-38B
0.259
0.360
0.330
0.393
0.640
0.253
0.184
0.074
0.022
0.077
0.133
0.415
Gemini 2.5 Pro
0.129
0.144
0.066
0,056
0.688
0.066
0.058
0,049
0.034
0.160
0.121
0.474
GPT-4o
0.052
0.047
0.021
0.037
0.346
0.000
0.005
0.006
0.088
0.016
0.024
0.242
GPT-5
0.176
0.195
0.090
0,100
0.739
0.023
0.028
0.014
0.051
0.062
0.085
0.547
0.109
0.104
0.596
0.026
0.025
0.030
0.044
Proprietary models underperform strong open-source systems. As shown in Table and Table 5, safety alignment affects GPT-5 and GPT-5-mini in particular, but this factor alone cannot account for the overall gap. The main weakness lies in bounding box tasks, where proprietary models handle whole-face face and whole-body regions adequately but fail sharply on fine-grained facial and body-part localization, revealed by Table This pattern suggests limited grounding capability at detailed spatial scales, possibly due to imbalanced training data. GPT-4o is further constrained by its older model, causing limited performance (Fu et al., 2024b; Lu et al., 2024; Zhang et al., 2024b). For details, see the discussion in Appendix I.
4 CONCLUSION
In this work, we presented Human-MME, comprehensive benchmark specifically designed to evaluate the human-centric perception and reasoning abilities of multimodal large language models. Our benchmark integrates a rich spectrum of tasks, spanning from fine-grained facial and body understanding to high-level intention, causal, and emotional reasoning. By coupling automated annotation pipeline with rigorous manual review process, we ensure both scalability and the high fidelity of annotations, while supporting diverse question-answer formats such as multiple-choice, short-anse anwwer grounding, ranking, and judgment- based tasks. The extensive experiments on 20 state- of-the-art MILMs effectively expose the limitations and guide future MLLMS toward better human- centric image understanding and reasoning. We hope that the proposed benchmark, analyses, and insights will serve foundation and catalyst for the next generation of multimodal systems that more deeply and reliably comprehend human scenes and behaviors.
5 REPRODUCIBILITY STATEMENT
We have already elaborated all the models algorithms proposed, experimental configurations. and benchmarks used the experiments in the main body or appendix of this paper. Furthermore. declare that the entire code used in this work will be released after acceptance.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Data Privacy and Licensing. The Human-MME benchmark utilizes images from Pexels, Pixabay, and the HICO-DET dataset. We strictly adhere the licensing terms of all data sources. Images from Pexels and Pixabay are used under their respective Content Licenses (Pexels, 2026; Pixabay, 2026), which permit free worldwide usage, modification, and distribution for academic and com- mercial purposes without mandatory attribution. In accordance with these licenses, we ensure that: (1) the images are redistributed sold standalone copies without sigmificant creative modifi- cation; (2) identifiable individuals are not portrayed in an offensive, immoral, or misleading manner. For HICO-DET, we comply with its original distribution policy (Chao al., 2018). To further safe- guard personal privacy, we will promptly remove specific images from the benchmark upon receipt of valid request from the depicted individuals.
Annotation Process. The manual correction and quality assurance process was conducted entirely by the authors of this paper. Utilizing our expert knowledge multimodal reasoning and human- centric computer vision, performed rigorous review of the automatically generated labels. This internal annotation strategy ensured a high level of consistency and professional judgment in handling complex scenarios, such disambi fine-grained body parts and interpreting higher- dimensional causal relations. No external crowdsourcing platforms or underpaid labor were involved in the creation of this benchmark.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62320106007) and the NUS Start-up Grant 10-0000006000-00..
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. Gra- dio: Hassle-free sharing and testing of ml models in the wild. arXiv preprint 2019.
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary. Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of- arXiv preprint arXiv:2503.01443, 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.086: 087744, 2023.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint urXiv:2305. 20400 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Opentlamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01130, 2023.
Lei Bai, Zhongrui Cai, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, Yongkang Chen, et al. Intern-sl: scientific multimodal foun- dation model. arXiv preprint 157753330...1763, 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877- 1901, 2020.
Johannes Buchner. imagehash: A python perceptual image hashing module, 2021. URL https: //github. http//Johannnnneheeeae imagehash.
Yuxuan Cai, Jiangning Zhang, Zhenye Gan, Qingdong He, Xiaobin Hu, Junwei Zhu, Yabiao Wang. Chengjie Wang, Zhucun Xue, Xinwei He, et al. Hy-mmbench: Benchmarking milms for human- centric video understanding. arXiv preprint arXiv:2507.00 04009, 2025.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. arXiv preprint
2018.
Lin Chen and Long Xing. Open- next-: An open-source implementation of llava-next series for facilitating the large multi-modal model community., 2024. URL hwtpt:tt////tt.t /t com/ xiacachen98/ 2aaa/ppenniee Ltwaa/opeen--aa---aaeaa
Yinan Chen, Jiangning Zhang, Teng Hu, Yuxiang Zeng, Zhucun Xue, Qingdong He, Chengjie Wang, Yong Liu, Xiaobin Hu, and Shuicheng Yan. Ivehench: Modern benchmark suite for instruction- guided vidco editing assessment, 2025.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguiiin tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, et al. Vicuna: An open-source chat- bot impressing gpt-4 with 90%* * chatgpt quality, 2023. URL https: / //ws////m.//////////////ss//ssss org/blog/ 1023-03-30-00 cuna/.
12

---

Published as a conference paper at ICLR 2026
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):
1-113, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Saurabh Dash, Yiyang Nan, John Dang, Arash Ahmadian, Shivalika Singh, Madeline Smith, Bharat Venkitesh, Vlad Shmyhlo, Viraat Aryabumi, Walter the frontier of multilingual multimodality. arXiv preprint Sidi::2505.0851,, 2025.
et al. Aya vision: Advancing
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied mul- timodal language model. arXiv preprint arXiv:2303.0333 2023.
Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, et Mme: comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2024a.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal lims video analysis. arXiv preprint arXiv:2405.21075. 2024b.
Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vin- cent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05550 055000 2024.
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- object interactions. arXiv preprint 170400774. 2018.
Haozhen Gong, Xiaozhong Ji, Yuansen Liu, Wenbin Wu, Xiaoxiao Yan, Jingjing Liu, Kai Wu, Ji- azhen Pan, Bailiang Jian, Jiangning Zhang, et al. Med-emr: A fine-grained benchmark integrating visual evidence and clinical logic for medical complex multimodal reasoming, 2025.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang. Guobing Gan, Haomiao Tang. Jiale Cheng, Qi, Junhui Ji, Lihang Pan, et al. Glm-4.5v and glm-4. Ii-t-..v-thinning:: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv preprint arXiv:2507.01006 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Juntao Jiang, Jiangning Zhang. Yali Bi, Jinsheng Bai, Weixuan Liu, Weiwei Jin, Zhucun Xue, Yong Liu, Xiaobin Hu, and Shuicheng Yan. M3cotbench: Benchmark chain-of-thought of milms in medical image understanding, 2026.
Glenn Jocher and Jing Qiu. Ulrraytics yololl, 2024. ultralytics/ulle wtw.c///utaalytiiiooooo
URL https://github. com/
Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, et al. Gemma technical report. arXiv preprint arXiv:2503.19788 2025.
Maurice George Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 06 1938.
13

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. arXiv preprint arXiv:2309.060 00..000090,,,, 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. In Advances in Neural Informa- tion Processing Systems, volume 36, pp. 71683-71702, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the Inter- national Conference Machine Learning, pp. 19730-19742, 2023b.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint 0rXiv:2403.00231, 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26306, 2024a.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint S.Ii:2333.054999 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Pruceedings of the European Conference on Computer Vision, pp. 216-233, 2024b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2015.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi. Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. In Proceedings of the International Conference on Learning Representations, 2024.
Meta AI. The llama 4 herd: The beginning of new era of natively multimodal ai innovation, 2025. URL https:://// /ai www.. cww/ haanm//lo////// alama- 4 mult moultimodal-ai
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf. et al. Crosslingual gen- eralization through multitask finetuning. arXiv preprins arXiv:221 1. 1.0/786, 2022.
Kartik Narayan, Vibashan VS, Rama Chellappa, and Vishal M Patel. Facexformer: unified trans- former for facial analysis. arXiv preprini arXiv:2403. 12960, 2024.
OpenAl. Gpt-5 system card. Technical Report System Card, OpenAI, August 2025. URL https: //con. openai. wtt.cccmcccpppottt 5- s--yyttem- card. car.
Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Lim evaluators recognize and favor their generations. In Advances in Neural Information Processing Systems, volume 37, pp. 68772- 68802, 2024.
Pexels. Pexels License What is allowed?, 2026. URL https: httpp::://wwwwpeelle../ com/ license/.
14

---

Published as a conference paper at ICLR 2026
Pixabay. Pixabay Content License Summary, 2026. URL license- suneeee-uumrr.....
Lixiong Qin, Shilong Ou, Miaoxuan Zhang, Jiangning Wei, Yuhang Zhang, Xiaoshuai Song, Yuchen Liu, Mei Wang, and Weiran Xu. Face- saice-human-bench A comprehensive benchmark of face and human understanding for multi-modal assistants. arXiv preprint arXiv:2501.01243, 2025.
Shaina Raza, Aravind Narayanan, Vahid Reza Khazaie, Ashmal Vayani, Mukund Chettiar, Aman- deep Singh, Mubarak Shah, and Deval Pandya. Humanibench: A human-centric framework for large multimodal models evaluation. arXiv preprint arXiv:2505. 250i::2550..11454 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B Hashimoto. Alpaca: A strong, replicable Stanford Center for Research on Foundation Models, 3(6):7, 2023.
model.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.099: 0,9888, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding. localization, and dense features, 2025. URL
https://arxte org/ /250 14786.
Jing Yi Wang, Nicholas Sukiennik, Tong Li, Weikang Su, Qianyue Hao, Jingbo Xu, Zihan Huang, Fengli Xu, and Yong Li. A survey on human-centric llms. arXiv preprint 2.ii::241..14493 2024.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, volume 33, pp. 5776-5788, 2020.
Ling Xiao and Toshihiko Yamasaki. Attribute-guided multi-level attention network for fine- ine-grained fashion retrieval. IEEE Access, 12:48068-48080 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Linyi Yang, Zhen Wang, Yuxiang Wu, Jie Yang, and Yue Zhang. Towards fine-grained causal reasoning and qa. arXiv preprint arXh::2204.07400 2022.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210-4220, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.8 01 800, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: ration. In Proceedings the the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
ow12: Revolutionizing multi-modal large language model with modality collabo-
Shukang Yin. Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12):nwae403 2024.
15

---

Published as a conference paper at ICLR 2026
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint XrXiv:2308. 0et..22000.04000, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoming benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, al. Mathverse: Does your multi-modal Ilm truly the diagrams in visual math problems? In Proceedings of the European Conference on Computer Vision, pp. 169-186, 2024a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with bert. arXiv preprint arXiv: 19040097,9 2020.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang, et al. Mme-rcalworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that difficult for humans? arXiv preprint arXiv:2408. 13257, 2024b.
Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li. and Ying Shen. Humanvbench: Exploring human-centric video understanding capabilities of milms with synthetic benchmark data. arXiv preprint arXiv:2412.17522 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En hancing vision-language understanding with advanced large language models. arXiv preprint DrXiv:2304.1059. 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXhv:2504.10470 2025.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
1
Introduction
2
Curating Human-MME Benchmark
2.1
Data Collection
2.2
Automated Annotation Pipeline
2.3
Manual Quality Review and Adjustment
2.4
Question-Answer Design
2.5
Statistics and Analysis
3
Experiments
3.1
Experimental Setting
3.2 Results.
3.3
Findings of Human- -MME
Conclusion
10
5
Reproducibility Statement
10
6 Ethics Statement
11
Related Work
19
Automated Annotation Detail
20
C Manual Annotation User Interface
22
Question-Answer Detail
23
Model Detail
31
F Prompt Formats
32
G Evaluation Metrics
32
H Full Experiment Result
34
I Detailed Discussion and Findings
35
Further Case Study and Findings
38
K Difficulty Analysis
39
17

---

Published as a conference paper at ICLR 2026
L The Use of Large Language Models
39
18

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Multimodal Large Language Models. The emergence of large language models (LLMM) (Touvron et al., 2023; Achiam et al., 2023) has driven substantial progress in the development of multimodal large language models (MLLMs) (Bai et al., 2025b; Chen et al., 2024; Hurst et al., 2024; Li et al., 2023b; Liu et al., 2024a; Georgiev et al., 2024; Ye et al., 2024) for visual- visual-language understanding. The progress is boosted from the BERT-based language decoders and progressively integrating de- velopments in LLMs (Yin et al., 2024). Multimodal large language models (MLLMs) demonstrate improved capabilities and performance, largely due to end-to-end training techniques that leverage advanced LLMS such as GPT series (Achiam et al., 2023; Brown al., 2020), L.LMMA (Touvron et al., 2023), Alpaca (Taori ct al., 2023), PaLM (Chowdhery et al., 2023; Anil et al., 2023), BLOOM (Muennighoff et al., 2022), and Vicuna (Chiang et al., 2023). Recent model advances--inllddnn Flamingo (Awadalla et al., 2023), PaLI (Laurençon et al., 2023), PaL.MEE (Driess et al., 2023), BLIP- 2 (Li et al., 2023b), Phi-4 (Abouelenin al., 2025), Otter (Li et al., 2025), MiniGPT-4 (Zhu al., 2023), MiniCPM-V (Yao et al., 2024), LLaVA-NeXT (Chen & Xing, 2024), Qwen- VL (Bai et al., 2025b), GLM-4V (Hong et al., 2025), and Intern VL (Wang et al., 2025) introduce novel approaches to obstacles such as improving instruction-following abilities, addressing alignment problems, and scaling up the pretraining model. Nevertheless, the performance of these models in fine- ane-grained human-centric image perception often remains underexplored.
Multimodal Large Language Model Benchmarks. In the field of MLLMs, numerous bench- marks have been developed to evaluate models' capabilities in both perception and cognition. As these MLLMs (Liu et al., 2024b; Yu et al., 2023) have demonstrated exceptional performance in general perception tasks, benchmarks regarding scientific understanding (Li al.. 2024), multi- modal mathematical reasoning (Lu et al., 2024; Zhang et al., 2024a), and multi-disciplinary (Yue et al., 2024: Fu et al., 2024a; Li al.. 2023a; Chen et al., 2025; Jiang et al., 2026; Gong et al., 2025) capabilities have drawn increasing attention. Among these benchmark, there exist some at- tempts to analyze the temporal and dynamic of human-centrie video understanding (Cai al., 2025; Zhou et al., 2024), which assign great attention to temporal perceptions and ignore fine-graine human-centric perceptions. There are also benchmarks focusing on human-centric image under- standing (Qin et al., 2025; Raza et al., 2025). However, (Qin et al., 2025) has a question format that is monotonous and lacks fine-grained features, and its scale is limited, while overly focusing on facial aspects. Raza et al. (2025) lacks questions related to facial features. And both of them do not have fine-graie greigra grounding features. Figure shows comparison between our work and previ- ous related benchmarks. To address this gap, we propose a dedicated benchmark focusing on human-centric image comprehension, which systematically evaluates model performance from granular human comprehension to high-order intricate spatial and causal reasoning perception.
Table 7: Methodology comparison to the existing benchmarks involving human features.
Benchmark
Scale
Data source
Labeling
QA curation
Quality assurance
MMBench
3.2K
public darasets and Internet
QA are constructed directly
No labeling.
Human
Human verification after multiple L.Mss filter
MME
2.8K
public darasets
are constructed directly
No labeling.
Human
Not mentioned
captions of datasets and
Sead-Bench
19K
public datasets
text generated by foundation models
GPT- generate
Homan verification
HV-MMBennch
8.7K Internet
Qwen2.5
Qwen2.5 generate
Human verification after Qwen2.5 filter
Human
2.1K
Internet
Video-MI.J J.MsM Mss gonerate
L.MSs generate
Human verificatiin
Face- Human Bench
2.7K
public datasets
captions of datasets
templates
Human verification
HumaniBench
32K
Internet
GPT-40
templates
Human verification
Human-MME.CO (Ours)
20K
public dazasets various specialized models LLMss Internet
templates
Human filtering and correction
As shown in Table 7, our benchmark design offers several advantages. First, by focusing specifi- cally on the domain of human images, we can rely on broad set of reliable expert models tailored for human-centric analysis rather than depending heavily on manual annotation, preexisting dataset labels, or LLM -e.eereeeetted labels. Manual annotation is constrained by cost, existing dataset labels risk information leakage and limit flexibility in question construction. and LLM annotation may introduce bias for certain question types and cannot reliably assess abilities such as fine-grained
19

---

Published as a conference paper at ICLR 2026
grounding. Second, by extracting rich portrait features using multiple specialized models, ob- tain highly structured and comprehensive representations of each individual. This enables integrated human-centrie reasoning that is not restricted to cettaset-rroiddee labels or to the limited set of at- tributes observed by a single MLLM. Consequently, we can construct complex multi-step questions that combine multiple attributes from the same image, for example "What is the age group of the person who is wearing beige jumpsuit". Because the data are structured, we do not rely on large language models for QA construction; instead, templates allow efficient generation large volumes of category-specift QA pairs. Third, we exploit the fact that the data souree contains many visu- ally similar images. By aggregating and comparing labels across similar instances, we selectively retain high-quality annotations, which greatly improves the efficiency of human verification while ensuring diversity across the dataset.
B AUTOMATED ANNOTATION DETAIL
VL2.5
Gender: Famale
Race:
Happy
Clothing:
Top:
Red.
Shirt
Ancessory: Silver
Brocolet
Interaction with Objectat
Single holding.
hawl
Stop
Body Bounding Boxes
Seaudion wit eaucbity some
Stop teveniing wew andinng tain image aioar poodre can oip, anee deattiin coseccttionn cotin oiject b actio
General Attributes, Wearing and Objects
dew arow while to net retures inyee we whw.
tanbo and 2ot
weth eptreetin ans, alowing sach sochstte amportes en anoce westhehhhee on sdonth
these teat carsa ri to emaring be
Stop 3:
Object Bounding Boxes
Stop 4:
Face Actribures and Bounding Boxes
Stop &
Emotion, Intontion and Cause
dreunsiing Dhee the sus sit
Passchermes tate foom anso streed hoceniing to initie 1)
Qaee M6d.a fartr pepeosee desured analyle portere
yiu Calibbk danest.
theee bnsyy of rntrrctiionn in "aanglo an will be spplace
Juncher butther properaed
Mort Oweed proce seas the desccee rerooc ary urte
tutriigh and lithiihht rowi.
Gatt eg the data envctionn onteec and raint bast
Pinaly desoriatonn ofeech person perprrteeenie beteeee
Quenti which askad pasaibbl everrs supporead burics
VL2.5
Aggrogated Bish Bohavioo
Altributes:
Emotion
Wett hand', 'nolding]] Object: bowl of fruits
Baare
inoviig encic opaetiee
No Baak-ttar
The peroor stargh
Eyebrows
Yun (uff]
Pull -25
Erctton:
3
aeteode acd cane puling acc
Asubee
Bounding Scooss
Figure 6: Annotation Pipeline before Manual Correction and QA Generation, corresponding to
Section 2.2.
Table 8: Overview of extracted feature sets and their meanings.
Symbol Bi
Description
Bounding box of the full body of person 7
F.
Bounding box of the face of person i
piame
Bounding box of specific body part (e.g., left hand, right foot) of person General attrihute (e.g., gender, age group, race) of person
Abame
W
Wearing item of person i, including type, color, and name
O
Object interaction of person including object and bounding box, action and body part Facial attribute of person (e.g., expression, hair style, presence of beard) Region-specife facial part box of person (e.g., nose, mouth, left eyc) Identity-neutral emotional analysis of person Identity-neutral intention inferred from the behavior of person i Scene-level cause (past) and consequence (future) narratives
name
P Piame
E,
4
Cpast fiuuture
20

---

Published as a conference paper at ICLR 2026
Figure 6 shows details about the automatic annotation pipeline applied to the collected dataset. The automatic annotation result will be used to generate question-anse answer pairs after manual selection and correction.
In Step 1 of Figure 6, we begin by applying DWPose (Yang et al., 2023) to each image to obtain whole-body pose estimates per person instance with 134 keypoints, including 18 body keypoints, 21 keypoints for each hand, 68 facial keypoints, and keypoints for each foot (each with a confidence score). In parallel, a pre-trained YOLOvI1 detector produces candidate body and face bounding boxes.
For each DWPose instance, align the body box by selecting the detection that has the greatest overlap with the minimum bounding rectangle (MBR) of the keypoints, Formally:
B: arg max (BU MBR(K:)) BEB
Here, B is the set of YOLOv11 (Jocher Qiu, 2024) body boxes; K; is the keypoint set of the i-th DWPose (Yang et al., 2023) instance; B; denotes the matched body box for person i. Face matching is analogous (replacing B by the face set F) and yields a matched face box F,. When a reliable body-pose match cannot be established, the image is discarded at this stage; when face match is unavailable, we simply retain body-only or face-only cases to accommodate close-ups and occlusions.
For body parts, if the minimum confidence over hand/foot keypoints is high, we derive part-level boxes from their keypoints' MBRs. We denote any such part box for person by Pname, where name is one of "left hand", "right hand", "left foot", or "right foot".
In Step 2 of Figure 6, we use the matched body bounding box B, obtained from Step 1 to isolate each person instance within the image. For each instance, we query the vision-language model Qwen VL2.5-72B Bai et al. (2025b) extract a set of factual attributes. The prompts used for this querying process follow a templated format, detailed in the appendix.
The model outputs are parsed obtain three categories of information:
General Attributes: These include perceived properties such as gender, age group, and race. We denote the general attributes person as A Agender A Aemotion and Arace,
Wearing Attributes: These describe the clothing and accessories worn by the person, in- cluding their type (e.g., "top"), color (e.g., "red"), and name (e.g. "shirt"). Each identified item represented as W(O), indicating the j-th wearing item for person i.
Object Interaction: The model also predicts interactions between the person and nearby objects. Each interaction is expressed as triplet O! where the object name (e.g., "bowl of fruits"),, the interacting body part (e.g.. "single hand"), and the type of interaction (tex- tite.g., "holding") are all inferred.
In Step 3 of Figure 6, the original image and the corresponding object name O(1) are input to Grounding DINO (Liu et al., 2023), which predicts the bounding box for each object. If the body part assigned to O.1) in Step 2 is "single hand", it is further refined at this stage. Since DWPose provides keypoints for both the left and right hands in K,, we compute the average distance from each hand's keypoints to the object bounding box. The hand with the smaller average distance is then used to replace "single hand" with either "left hand" or "right hand".
In Step 4 of Figure 6, for each individual, we first enlarge the face bounding box F, by a factor of two and crop the corresponding face region from the image. The cropped region then passed to FaceXFormer (Narayan et al., 2024) for facial feature recognition, landmark localization, and head pose estimation. FaceXFormer predicts 40 binary facial attributes, all derived from the CelebA dataset (Liu et al., 2015). We define two probability thresholds: high threshold and a low threshold. If the predicted probability for an attribute exceeds the high threshold, the face is considered to possess the corresponding attribute; if it falls below the low threshold, the attribute is considered absent; otherwise, the attribute is marked as uncertain. The extracted facial attributes are denoted as F Alame, where name refers to the specific attribute (e.g., "goatee")).
21

---

Published as a conference paper at ICLR 2026
Among the head pose parameters provided by FaceXFormer, focus pitch and yaw, which define the face orientation in degrees. Using a threshold of +15" we discretize pitch into up or down and yaw into left right, where the latter indicates the side of the image toward which the face is oriented (rather than the subject's own left or right). These head pose attributes are also incorporated into FA," F F.ddww, F FA and F FA as part of the facial attributes.
In addition, FaceXFormer outputs 68 facial landmarks. Based on prior knowledge of these land- marks, we can derive bounding boxes for specific facial regions regions-includng including the nose, mouth, left eye, right eye, left eyebrow, and right eyebrow. The bounding box from FaceXFormer outputs will also be compared with the the facial part of DWPose results K; in Step to see if they aligned well enough. When FaceXFormer results have low IoU compared DWPose result, the bounding boxes will not be recorded. These region-specific bounding boxes are denoted as F prume, where name refers to the corresponding facial part.
In Step 5 of Figure 6, the objective is to extract higher-level semantic attributes such as Emotion, Intention, and Cause. Each person appearing in the image is sequentially highlighted and queried by Qwen2.5-VL-725 with two prompts. The first prompt requests a detailed analysis of the individual's emotions and thoughts to produce the intermediate output E using Agmotion as reference. The second prompt seeks comprehensive description of the person's behaviors, interactions with other people and objects, and any plausible intentions, resulting in a behavior description denoted as 1,*
Subsequently, E is provided to Qwen3, which removes any information that could reveal personal identity, such gender age, yielding purely identity-neutral emotional analysis Eg. In parallel, T,t is passed to Qwen3 to perform two operations: first, to remove all identity-related details, and second, explicitly extract any inferred intentions of the person. This process produces the final intention analysis Li-
Finally, all behavior descriptions I,- from every person in the image are aggregated and submitted to Qwen3 to infer the broader context of the scene. Qwen3 generates two high-level narratives: C'+past describing possible events that may have occurred prior the captured moment, and ++ pre- dicting events likely to unfold afterward. Each of these narrative outputs is further processed by Qwen3 to remove description Cffuurre and future-consequence
information, resulting in the final past-cause description C'ast
The complete set fine-grained features extracted for person i can be expressed as:
T ={ Bi, F,, (Phane) name:
(A name
(w) (0()))) }3 ; (F name)
{ Paame
name }
name,
name:
C MANUAL ANNOTATION USER INTERFACE
Boundin Dox Nootr
(a) Manual Selection and Deduplication.
(b) Manual Correction.
Figure 7: Manual Review Software Interface.
We developed a custom software tool based on Gradio (Abid et al., 2019) to facilitate both data deduplication and annotation refinement after automatic annotaation.
22

---

Published as a conference paper at ICLR 2026
Because the de-duplication in Section 2.1 was relatively permissive, the automatic annotation pro- cess described in Section 2.2 inevitably produces redundant annotations for many nearly identical images. To address this, we first extract feature vectors with YOLOv11 and cluster the images accordingly. The subsequent annotation workflow then first operates at the cluster level.
As illustrated in Figure 7a, the first stage is Selection and De-duplication. Within each cluster of visually similar images, annotation experts are asked to select subset that maximizes diversity of subjects and complexity of content. For example, when multiple images contain the same indi- viduals, preference is given to those in which a HOI has been successfully annotated. Likewise, previewed bounding boxes should be accurate as possible, ensuring that the automatic annota- tions capture all important individuals. After selecting the subset, the expert proceeds to the next step by clicking the button at the bottom of the interface, where each image is reviewed and corrected individually.
The Manual Correction phase involves detailed inspection and refinement of the automatically generated annotations for every detected person. This includes removing adjusting entire body bounding boxes when they are incorrectly assigned. These operations are performed the red area of Figure 7b. Wearing attributes w:(1) are examined and edited in the yellow area by directly modifying the corresponding JSON text. HOI attributes O such as associated objects, body parts, or actions can be corrected in the blue area. Finally, bounding boxes can be adjusted in the green region within the red area, and all modifications are immediately visualized in real time in the same green region.
Once the corrections yield satisfactory results, the expert saves the manually refined annotations. the automatic annotations too poor to be corrected effectively, the expert can either return to the previous step to reselect images or discard the problematic image altogether. After completing all images in the current cluster, the process repeats for the next cluster until the entire dataset has been reviewed.
D QUESTION- DETAIL
The Question-Answer Design stage leverages the rich of features extracted during data anno- tation to construct a comprehensive evaluation framework. We design total of 21 question types spanning eight dimensions: Face Understanding, Body Understanding, HOI Understanding. Multi- Image Understanding, Multi-Person Reasoning, Intention Discrimination, Causal Discrimination, and Emotion Discrimination. The corresponding answer formats cover seven distinet forms: sin- gle choice, ranking, short-answer, bounding box, judgment combined with short-answer, judgment combined with bounding box, and short-answer combined with bounding box. Representative ex- amples of each question type are provided.
Face Understanding applies to images containing only single person i, we define two categories of questions: Face Grounding and Face Choice. In Face Grounding, the model is required to predict the bounding box of specified facial region. The target region is randomly selected from the full face box F, or one of the finer-gras greined subregions Fimmouth, FP F eye FP Fl cyebrow or F cyebrow In Face Choice, a single facial attribute is chosen from F F, as the correct answer. Three distractor attributes, absent from F FA,, are also provided, and the model must identify the correct one.
Body Understanding also focuses on images with exactly one person i and includes two question types: Body Grounding, Wearing Choice and Wearing Short-Answer. Body Grounding asks the model to predict the bounding box of a body region randomly selected from the full body box B, or hand,
one of the keypoint-based limb regions plett hand pright hand platt foot or pright foot Wearing Choice
evaluates the model's understanding of clothing attributes: one wearing item is randomly sampled from W; the correct answer, while three incorrect options are generated by altering its color and name. The question specifies the item's 's type, and the model must identify the correct wearing from the image. The replacement names and colors for the distractors drawn from annotations of wearing items of the same type in other images. Wearing Answer provides the type of a existing W; and ask model to provide its color and name.
23

---

Published as a conference paper at ICLR 2026
HOI Understanding has three types of questions: HOI Choice, HOI Short-Answer, and HOI Grounding. For any person in the image, one of their object interactions Og is selected as the basis of the question. In HOI Choice, each option contains an object name, an action, and body part; only one option is entirely correct, while the object name or body part in the remaining options is randomly altered. The model is required to select the correct option. In HOI Short-Answer, the action and body part are provided and the model must answer with the corresponding object name. In HOI Grounding, only the action and body part are given, and the model is expected output the bounding box of the relevant object.
Multi-Image Understanding consists of three question types: Multi-Face, Multi- Sulti-Wearing,, and Multi-HOI. Multi-Face presents four images together with three facial attributes that may appear in FA. Among the four images, one contains a face that satisfies all three attributes, one satisfies two, one satisfies only one, and one satisfies none. The model is asked to rank the four images according to how many of the three attributes are satisfied by at least one face. Multi- Mualii-eering follows the same logic but uses three clothing items that may appear in W; the model must order the images by the number of those clothing items present. Multi-HOI differs in that it provides a description including body part, an action, and an object name; in the four candidate images some may have mismatched objects or body parts, and the model must identify the image that best matches the description.
Multi-Person Reasoning covers a broader set of question types and focuses on images containing multiple people. Questions typically target a specific person i with feature set Tr, while the other people in the image are indexed by j and have feature sets Tj.
Identify-related questions: Identiffy Short-Answer, Identify Bounding Box, Identify Open HOI, and Identify Choice. They are constructed by first selecting feature from Ti 1 "T that unique to person i, ensuring that the model can localize the correct individ- ual. For Identify Short-Answer, a second feature is chosen from T, (such as clothing, HOI, or general attributes) and the model must provide the answer to a direct question about it. Identify Bounding Box instead selects a facial or body bounding box from T, and requires the model to output the corresponding box. Identify Open HOI chooses an HOI from O and provides its action and body part; the model must return both the object name and its bounding box. Identify Choice selects one feature from To as the correct option and three features from T, Te as distractors; the model must select the feature that matches person i.
Judgement-based questions: Judgement Short-Answer and Judgement Bounding Box. They require the model to refrain from answering if no individual satisfies the specified criteria, and to proceed only when a suitable person exists. A unique feature a is first drawn from T, - T.. and another feature b is drawn from Ti U TT,, which may may not belong to person i. If b E Tr, the question has a valid answer; if E Tr the model is expected explicitly decline to answer. The model is instructed locate and focus only on a person who satisfies both features and b, and then provide the requested response, or state that no such person exists. Judgement Short-Answer subsequently asks about a feature from Ti (such as clothing, HOI, or general attributes). Judgement Bounding Box asks the model output facial or body bounding box from T, These types of questions are specifically designed to detect model hallucinations.
Finally, Common Choice is a single-choice question in which the correct answer is a fea- ture drawn from n Te, i.e., feature shared by all people in the image. The three dis- tractor options are features unique to individual persons. The model must select the feature that is common to every person present.
Intention Discrimination contains a single question type, Intention Choice. In this setting, an image features one person whose identity-neutre intention is denoted by T,. Using CLIP, three visually similar images are retrieved, and the identity-neutral intentions from these three images are used as distractor options. The original image is presented to the model, which must select the intention description that best matches the depicted scene.
Causal Discrimination also includes only one question type, Causal Choice, which is a dual- selection task. For each image, there are two scene-level narratives: the cause Crest and the con-
24

---

Published as a conference paper at ICLR 2026
sequence future, CLIP is employed to find visually similar image, from which Crpast and C*funure serve as incorrect alternatives. The model is required to determine which option correctly describes the past cause of the scene and which option correctly predicts its future consequence.
Emotion Discrimination comprises single question type, Emotion Analysis Choice. Given an image with one person and that person's identity-neutae emotional analysis E,. CLIP is used cate visually similar images in which at least one person shares the same raw emotion label _ The identity-neutral emotional analyses of those individuals with matching raw emotions taken as distractors. The model is then asked to choose the emotional analysis that most accurately reflects the emotions of the target person in the original image.
Representative examples for all the questions types are shown in Table 9,
Table 9: Examples of all question types.
Question Type
Example Image
Example QA
Question: Resolution of the image provided is 4096x2160, Please provide the bounding box of the facial part "mouth" of the main person in the image.
Face Grounding
Answer: [2773, 1582, 3558, 1923]
Question: Please select the facial features of the person in the image from the following options (only one selection is allowed):
A. Wavy hair
Face Choice
B. Rosy cheeks
C. Has goatee
D. Mouth slightly open
Answer: D
Question: Resolution of the image provided is 1920x1080, Please provide the bounding box (in xyxy format) of the "right hand" of the main person in the image.
Body Grounding
Answer: 1522,326.604.4444
Question: Please select the wearing the
person in the image from the following options
(only one selection is allowed):
A. Black dress
Wearing Choice
B. Olive witch hat
C. Black bonnet
D. Black graduation gown
Answer:
Question: Please give name and color of the clothing item of type "headwear" that the main person is wearing the image.
Wearing Short-Answer
Answer: Party hat in pink and white
Continued next page
25

---

Published as a conference paper at ICLR 2026
Question Type
Example Image
Example QA
Question: Please select the option that best describes the interaction between the person and object (only one selection allowed):
A. body part: left hand action: holding object:
beads
B. body part: right hand action: holding object:
HOI Choice
beads
C. body part: right hand action: holding object:
flower branch
D. body part: left hand action: holding object:
flower branch Answer: D
Question: Please name the object that have interation "body part: left hand, action: holding" with the main person in the image.
HOI Short-Answer
Answer:
cardboard box
Question: Resolution of the image provided is
4096x2160. Please provide the bounding box (in хyxy format) of the object that have interation "body part: left hand, action: holding" with the main person the image.
HOI Grounding
Answer: [1119,701,2404,4477
Question:
Narrow eyes
Wearing earrings
No beard
Listed are some facial attributes that appeared in the four images above. Please give the sequence of four images by the maximum count of facial attributes that appears in one single person. someone in specific image is showing all three facial attributes, it should be the first image your answer, and if none of three facial attributes are present, it should be the last image. Please provide explicit sequence of four images by their letters.
Multi-Face
Answer: A C
Continued on next page
26

---

Published as a conference paper at ICLR 2026
Question Type
Example Image
Example QA
Question:
Gray trousers
Gold necklace
White sneakers
Listed are some clothing items that appeared in the four images above. Please give the sequence of four images by the maximum count of clothing listed that appears in one single person. someone in a specific image is wearing all three clothing items, should be the first image in your answer, and if none of three clothing items are present, it should be the last image. Please provide a explicit sequence of four images by their letters.
Multi- Meati-Wearing
Answer: D
Question: This is description of a human-object interaction: "body part: body, action: lie on, object: surfboard"
Multi-HOI
Which one of four images listed above best represents this interaction? Provide your answer with the corresponding image letter.
Answer:
Question: There one person the image that meets the following condition:
race: black
Identify Short-Answer
What is name and color of the clothing item of
type "headwear" that the person is wearing?
Answer:
Crown hat in red and green
Continued on next puge
27

---

Published as a conference paper at ICLR 2026
Question Type
Example Image
Example QA
Question: Resolution of the image provided is
3840x2160. There is one person in the image that meets the following condition:
Have interaction with object:"body part: body,
Identify Bounding
action: sitting on, object: chair"
Box
Please provide the bounding box of the person's s
face in xyxy format.
Answer: 1195,,133,2220 1864]
Question: Resolution of the image provided is
1920x 1080. There is one person in the image that
meets the following condition:
WWearing Gray sweater
Identify Open HOI
Please provide the name and bounding box in xyxy format of the object that have interation "body part: right hand, action: holding" with the person.
Answer: hiking poles [1005,696 6,,4, 1115,1076)
Question: Resolution of the image provided is 4096x2160. There is one person in the image that meets the following condition:
-Wearing Sari in blue.
Ignoring other persons, please select the option that best describes the referred person.
Identify Choice
A. Face turned left side of image
B. Face turned to right side of image
C. face in the bounding box
[2377,842 042,8009,999,,, 1452] (xyxy format)
D. Wearing White necklace
Answer: B
Question: There might be one person the
image that meets the following two conditions:
Has beard
Lookking downward
Judgement Short-Answer
Please answer the following question if there
such person. Or else, please provide
"unknown" as answer:
What is the gender of the person?
Answer: male
Question: Resolution of the image provided is 1920x 1080. There might be one person the image that meets the following two conditions:
-Has goatee
Judgement Bounding Box
-Face turned to left side of image
Please provide the bounding box of the person's 's face in xyxy format if there is such a person. Or
else, please provide [-1,-1,-1,-]] as answer.
Answer: [1264.65, 14466.367
Question: Please select the option that fits most
or all of persons in the image:
A. Wearing T shirt in white and black
B. gender: male
Common Choice
C. race: black
D. Not blond hair
Please provide the option letter of the most possible answer.
Answer: D
Continued on next page
28

---

Published as a conference paper at ICLR 2026
Question Type
Example Image
Example QA
Question: Please select the best analysis of intention for someone appearing in the image:
A. The individual is expressing enthusiasm and engagement in a celebratory or performance-relatee activity through deliberate posing and vibrant attire
B. The individual is preparing for an outdoor adventure with companions and pets by organizing gear and coordinating plans while positioned near a vintage van mountain road C. The individual is expressing joy and relaxation while engaging with natural outdoor setting in comfortable and effortless manner
Intention Choice
D. The individual is expressing joy and contentment while embracing relaxed and natural setting with confident ease
Answer: A
Question: Please select the best analysis of what happened in the past and what will happen in the
future:
A. The individual sets down the instrument, takes deliberate breath, then flips through the pages with focused intent, adjusting finger placement and refining the rhythm before resuming with renewed precision.
B. The figure moved steadily along a winding trail, the earth beneath foot soft with fallen leaves and moss, drawn by the hush of the water and the distant whisper of reeds, pausing only steady breath and release the weight of the day before stepping into the clearing.
C. The individual had carefully arranged the performance setup. positioning the instrument precisely on the surface, aligning the written material for optimal visibility, and connecting the digital device to the audio output, all with deliberate intent to begin a structured and focused session.
Causal Choice
D. The hand moves with deliberate calm, tracing lines that breathe life into the stillness, each stroke a quiet declaration of presence, as the mind, unburdened and attuned, translates the essence of the moment into something tangible and enduring.
Please provide the option letters of the most possible answer separately for past and future. Answer: past: C future: A
Continued on next page
29

---

Published as a conference paper at ICLR 2026
Question Type
Example Image
Example QA
Please select the best analysis of emotion for someone appearing in the image:
Question:
A. A deep sense of joy and contentment radiates from within as the individual experiences pure happiness and emotional ease in the moment
B. A deep sense of joy and contentment is present accompanied by a feeling of ease and fulfillment in the moment
Emotion Analysis Choice
C. A deep sense of accomplishment and fulfillment washes over with the realization of hard-earned success bringing joy that both intense and enduring
D. A deep sense of joy and contentment is evident through broad smile that reaches the eyes and relaxed, upright posture suggesting inner peace and satisfaction with the moment Answer: C
Figure 8: Samples of images used by questions in Human-MME.
30

---

Published as a conference paper at ICLR 2026
E MODEL DETAIL
In this section, we provide detailed descriptions of the MLLMS evaluated in our experiments. Ta- ble 10 summarizes the architecture and design choices of each open-source model, including the vision encoder backbone, the underlying language model, and the total number of parameters. In addition, we indicate whether model incorporates image grounding training or alignment, which is an important factor for interpreting their performance in different tasks.
These details complement the experimental settings described Section 3.1. In terms of image grounding training and alignment, more specifically, GL.--455V (Hong et al., 2025) and GLM-4.1 9B (Hong et al., 2025) target image grounding by aligning their outputs to normalized xyxy-format bounding box, separated by dedicated tokens. Qwen2.5-VE (72B, 32B, 7B; (Bai et al., 2025b)) models are trained with corresponding data and restricts its outputs to JSON format. InternVL3-78B (Zhu et al., 2025) and VL3.5
(Wang et al., 2025) are also trained with image grounding data. Kimi- VL-A3B (Du et al., 2025) is trained with image grounding data but focuses mainly on GUI tasks. By examining these architectural aspects, aim to facilitate more transparent comparison of experimental results and highlight how design choices impact model behavior.
Table 10: Information about evaluated open-source MI.LMs. Image encoding means how they map the pixels of images tokens in language models. Grounding training means the training data and format alignment related to vision grounding
Vision Encoder
Language Model
Model
Image Encoding
Grounding Training
GLM-4.5V
dynamic
xyxy- pox--fomma bounding box with dedicated tokens
AlMv2-Huge
GLM-4.5-Air
(106B)
tokenization
dynamic
xyxy-format bounding box
GL--4.1V-9B
AIMv2- Huge
GLM-4-9B
tokenization
with dedicated tokens
Qwen2.5-VI-
Redesigned ViT
dynamic
xyxy-format bounding box
Qwen2.5-72B
72B
tokenization
with ISON format alignment
Qwen2.5-VI
Redesigned ViT
dynamic
xyxy-format bounding box
Qwen2.5-328
32B
tokenization
with JSON format alignment
Qwen2.5-VL-
Redesigned ViT
dynamic
xyxy-formal bounding box
Qwen2.5-7B
7B
tokenization
with ISON format alignment
with image grounding training
Intern VL3-78B
InterVVi--6B
Qwen2.5
448x448 256
data
Intern VL.3.5-
448x448 > with image grounding training
Intern ViT-6B
Qwen3-32B
38B
data
Intern Intern VL3.5-
with image grounding training
inT-mVViT-B
Qwen3 --3BB
448x448 + 256
BB
data
Intern-S1
Intern ViT-6B
Qwen3-2333
448x448 + 256
None
(241B)
MiniCPM-V-44.5
SigLIP2-
Qwen3-8B
448x448 256
None
(8B)
400M
Dec-only
896x896 - 256
Gemma3-27B
400.IPP400MM
None
Transf.
(whole image)
SigLIP2-
Aya Expanse
Aya-vision-322
364x364 + 169
Nonc
400M
32B
LLaVA-NeXT- 72B
CLIP-Large
Qwenl.5-722
336x336 + 576
None
dynamic tokenization dynamic
Llama-4-Scout
MetaCIIIP
Ilama MoE
None
Kimi-V VL-A3B
With GUI-focused grounding
Moon VIT
Moonlight MoE
(16B)
tokenization
training data
384x384 - 729
Phi-4 (6B)
SigLIP-400M
Phi-4-Mini
None
(whole image)
31

---

Published as a conference paper at ICLR 2026
F PROMPT FORMATS
For each question type, the model is guided by structured prompt prefixed with "Your answer should follow this format strictly:" " to ensure that the outputs are consistent and easily parseable. Table 11 summarizes the prompts used for all question types.
Table 11: Prompt formats for different question types. SC: Single-Choice, DC: Double-Choice, R:
Ranking, BB: Bounding-box, SA: Short-Answer, Judgement.
Analyze: <ouur analysis>
Analyze: <your analysis>>
SC
BB
Answer: A/B/C/D
Answer: [xI.yI.x2.2]]
Analyze: <your analysis>
Analyze: <yourr analysis>>
DC
Past: A/B/C/D
SA
Answer: <your final answer in short and concise expression>>
Future: A/B/C/D
Analyze: <your analysis>
<your analysis>> Answer: <your final answer in short and concise expression>> if match, answer unknown
Analyze:
First: A/B/C/D
R
Second: A/B/C/D
J+SA
Third: A/B/C/D
Fourth: A/B/C/D
Analyze: <your analysis> Answer: [x1.y1xx2.2]] no match, answer [-1,-1,-1,-1]
Analyze: <your analysis>>
J+BB
SA+BB
Name: <name of the object> Box: [x1.yI.x2.y2]
G EVALUATION METRICS
We evaluate the performance of the models using metrics tailored to each type of question.
Choice Questions
Accuracy is used to measure correctness:
Number of correct selections
Accuracy
Total number of questions
Short-Answer Questions We assess semantic correctness using three complementary measures:
1. BERT F1 Score (Zhang al., 2020): compute token-level Fl between predicted answer and ground truth:
2. A.EER RBERT
BERT F1
PBRR + RBERT
where PBERT and REERT are precision and recall computed over BERT-token matches.
2. Cosine Similarity of Embeddings (Wang et al., 2020):
Vpred Vgt
CosineSim
|| |
where prdd and Vgi are the embedding vectors of the predicted and ground-trute truth answers.
3. Keyword Coverage:
KeywordCoverage Keywords pred O Keywords,
Keywords
The three measures combined into composite score enhance robustness:
Composite Score 0.5 BERT F1 0.3 CosineSim + 0.2 KeywordCoverage
32

---

Published as a conference paper at ICLR 2026
Ranking Questions true ranking:
Kendall's Tau (Kendall, 1938) measures agreement between predicted and
C-D
2 ()
where C and D the number of concordant and discordant pairs among items.
Bounding-Boo Questions
Intersection over Union (IoU) is used:
Area( Bp 0 Bgr) IoU
Area( Bp U Bye)
where Bp is the predicted bounding box and Bgt is the ground-truth bounding box.
Judgment Questions Fl score balances precision and recall:
2 Precision Recall
F1
Precision + Recall
This comprehensive set of metries captures both exact correctness and semantic similarity across diverse question types.
33

---

Published as a conference paper at ICLR 2026
H FULL EXPERIMENT RESULT
Table 12: Full result Human-MME on 20 models, 8 dimensions, 21 question types and 5 question components.
Inderstancing
Body Understanding
HOT
Model
Face Grounding
Face
Body
Wearing
Wearing
HOI Grounding
HOI
HOI
Choice
Average
Grounding
Choicc
Short-Answer
Average
Choice
Showe-Answer Average
ToUl
Accuracy
ToU
Accuracy
Composite
InUl
Accuracy
Composite
GLM-4.5V
58.8
61.6
90.7
80,9
77.4
84.9
82.5
GV99
45.8
64.5
55.2
55.7
88.6
77.9
74.1
40.5
79.8
88.3
69.5
Qwen2.5
61.1
Qwen2.5 2 VL-32B
47.7
64.7
56.2
$6.0
86.8
77.1
73.3
29.2
81.2
85.5
65.3
Qwen2.5 VI.7B
38.5
60.4
49.4
45.9
81.9
77.5
68.4
28.0
74.3
81.9
61.4
Intern S1
14.8
67.1
41.0
23.3
90.0
82.2
65.2
24.9
82.3
89.3
65.5
L
32,3
69.2
50.7
54.2
90.3
79.4
74.6
39.0
84.5
90.7
71.4
VntcmVL3-788
73.2
43.4
93.7
82.9
G7.9
29.1
67.2
IntemVI.1.:388B
20.4
68.8
44.6
42.4
93.7
81.7
72.6
24.3
81.7
87.8
64.6
Llama-4-Scout
27.3
69.8
50.6
78.3
LLVA--NXXT7228
13.2
62.8
38.0
38.4
85.2
76.9
66.8
35.8
78.4
81.0
65.1
Aya-vision 32B
55.5
30.9
13.5
75.9
82.2
57.2
10.2
77.0
84.3
57.1
Gemma3-27B
62.1
35.1
16.1
83.1
80.5
59.9
81.2
86.2
61.2
Kimi-VL-A3B
15.2
59.4
37.3
29.7
83.3
76.2
63.1
11.1
75.3
65.9
50.8
MiniCPM-V-4.5
38.9
24.5
88.9
Phi-4
6.0
$3.0
29.5
12.1
59.2
72.9
48.1
8.5
72.3
64.9
48.6
Gemini
89.0
GPT-4o
5.7
$2.0
28.8
13.1
82.8
80.6
58.8
18.0
76.4
85.1
59.8
GPT-S
14.7
34.4
91.2
67.8
71.1
GPT-S-mini
104
50.8
30.6
25.4
91.0
82.6
66.3
29.8
85.4
87.1
67.4
Multi Person Remsoning
Identily Grounding
Identify Choice
Identify Short-Answer
Identity Open HOI
Judyment Grounding
Judgment Short- Bhowe-Answer
Commoo Choice
Average
loU
Accuracy
Composite
Composite
loU
FI
IoU
Composite
Accuracy
GLM-43V
63.8
62.3
81.6
83.8
72.6
68.3
63.8
68.3
80.1
71.5
71.5
G.M-4.IVV9B
45.0
59.9
79.1
84,0
66.5
69.2
46.8
63.4
76.9
62.9
64.3
Owen2.5 VL-72B
57.3
64.9
79.8
81.9
61.4
67.5
49.7
75.2
80.7
46.2
Qwen2.5 VL-32B
46.4
61.8
74.3
65.7
47.0
65.3
43.2
68.6
66.1
46.9
58.2
Qwen2.5 VL-7B
33,2
56.6
17.1
69.4
29.9
Intern-SI
22,6
64.1
79.3
86.2
26.6
70.3
19.9
67.5
78.5
74.3
59.3
VL3.52L3
47.2
58.1
82.9
61.8
58.2
79.0
40.1
V1.3788
28.2
$9.8
79.4
86.9
35.8
62.6
25.8
60.7
81.9
38.2
54.6
VL.355L.33..00 38B
28.7
57.4
77.9
83.8
42.4
66.0
25.8
58.0
78.2
35.2
53.8
Llama-4-Scout
6.6
31.7
67.9
74.0
5.1
28.9
48.3
62.4
21.1
33.9
LLaVA-NeXT-72B
27,2
43,1
68.8
75.0
39.6
53.0
24.8
51.7
71.1
34.0
47.2
Aya-vsion-322
82.7
Gemma3-27B
15.5
39.8
77.0
75.9
11.8
55.4
15.3
53.7
77.7
38.2
45.1
Ximi-VL-A3B
19.5
40.2
73.1
72.2
9.0
46.2
19.6
54.7
72.2
28.8
42.6
MiniCPM-V-4.55
24.5
49.3
79.4
84.4
21.3
58.9
20.5
$7.0
78.6
50.9
52.1
Phi-4
4.1
32.3
66.9
49.5
0.2
7.0
0.7
32.7
59.9
29.0
29.6
Gemini-2.5.8 2.5.Prr
23,8
56.9
82.2
86.4
20.0
69.9
19.1
74.1
80.4
74.6
58.9
GPT-4o
10.8
38.7
75.9
85,9
12.8
47.7
6.9
49.5
70.6
27.4
41.4
86.7
27.1
GPT mini
21.7
51.5
71.9
86.7
23.7
47.1
11.5
$1.2
68.0
48.8
48.3
Mult--Image Understanding
Intention
Cause Discrimination
Emotion
Discrimination
Discrimination
Model
Mulii Face
Mulii Wearing
Mulni HOI
Intention Choice
Emotion Choice
Average
Average
Accuracy
Accuracy (Future)
Tau
Tau
Accuracy
Accuracy
Accuracy
(Past)
GLM-433V
86.1
86.
63.4
79.2
85.9
85.6
85,1
85.4
66.6
GLM-4.IV.. IV-9B
76.8
76.7
62.0
71.8
82.7
75.3
76.8
76.0
58.8
Qwen2.5 VL-72B
81.5
60.6
88.1
85.4
87.2
86.3
65.3
Qwen2.5 VL-32B
75.3
78.5
58.3
70.7
82.9
80.3
81.8
81.1
64.9
Qwen2.5 VI.7B
6344
65.8
53.7
61.0
84.1
67.4
76.8
72.1
60.9
Intem-S1
85.1
87.2
67.1
79.8
82.9
83.1
83.3
83.2
68.3
In3emVLL3.5224BB
83.0
83.9
62.1
76.4
83.7
77.8
87.2
82.5
66.4
IntemVLL3778
86.7
IntemVL333-388
78.0
84.2
62.8
75.0
86.9
79.1
77.0
78.0
63.6
Llama-4-Scout
51.1
51.3
44.2
48.9
66.5
54.4
59.8
57.1
50.4
I.A.V.NeXXT-23
60.9
60.1
43.3
54.8
77.0
68.4
72.6
70.5
54.6
Aya-vision 32B
78.3
73.0
52.3
67.9
76.2
68.6
74,9
71.8
57.4
Gemma3-228
75.7
77.0
Kimi VL-A3B
23.3
7.6
51.0
27.3
$1.0
65.5
60.7
63.1
55.3
MiniCPM-V-4.5
Phi-4
45.6
42.5
30.6
39.6
62.9
41.4
34.7
38.1
46.4
Cemmin-2..-PPo
88.1
GPT-40
77.9
85.5
60.7
74.7
79.2
74.3
78.0
76.2
$2.7
GPTS
73.0
89.0
65.5
75.8
82.3
90.4
88.1
89.2
42.6
GPT:S-mini
73.7
89.6
65.9
76.4
79.4
82.6
80.8
81.7
39,9
34

---

Published as a conference paper at ICLR 2026
1 DETAILED DISCUSSION AND FINDINGS
Here provide detailed discussion for findings in Section 3.3.
Stronger scaling effects in human related Choice and Ranking tasks. According to Figure 4, per- formance on Choice and Ranking question components have the strongest correlation to model size. The significant positive correlations can be attributed to the fact that models with larger parameter counts are better able attend to and integrate a greater number of visual features simultaneously, In Choice tasks this capability allows the model to evaluate multiple candidate options in parallel and distinguish subtle differences among them. For Ranking, which involves reasoning across multiple images and combining numerous facial or clothing attributes, the ability to consider many features at once is even more critical. As a result, increasing model size directly enhances performance in these settings, leading to stronger correlations compared with other metrics.
Training data have a strong influence on human related grounding task. Figure 4, the perfor- mance on bounding box tasks shows almost no correlation with model size. We therefore examine range of factors that could potentially influence grounding capability, including the choice of vision encoder, the language model backbone, the method used to convert images into tokens, and the pres- ence of grounding-relese relevant training data and format alignment. Our analysis indicates that training data and output-fomm tailed comparison of model architectures is provided in Table 10, and the corresponding bounding box results appear in Table 2.
alignment provide the most substantial benefits for visual grounding. A de-
MiniCPM-V-44 and Aya-vision-33 -32B both employ the same SigLIP2-400M (Tschannen et al., 2025) vision encoder and neither reports any specialized training on visual grounding tasks. Despite having far fewer parameters, MiniCPM-V-4 achieves a clearly higher IoU than Aya-vision-32B. Gemmaß- 27B and Phi-4 also share a common vision encoder, SigLIP-400M (Zhai et al., 2023), yet their grounding performance differs substantially. Within the GLM, Qwen, and Intern families, models that use identical vision encoders still display large within-family variation. These observations indicate that the vision encoder is not the dominant factor affecting grounding outcomes.
Regarding the language model, MiniCPM-V-Ae uses Qwen3-8B, which has stronger general ca- pability than the Qwen2.5-7B backbone used by Qwen2.5-VL-7B. Nonetheless, MiniCPM- V-4.5 performs worse on grounding than Qwen2.5-VL-7B. Similarly, Intern-SI adopts the largest Qwen3- 235B language model in the comparison, yet its grounding performance is weaker than that of which relies on the smaller previous generation Qwen2.5-7B. These results in- dicate that the language model itself has limited influence on bounding box performance.
In terms of image tokenization, the best-performing GLM and Qwen models use dynamic tokeniza- tion rather than mapping fixed-resolution image tiles fixed token counts. However, Kimi-VL- A3B adopts a similar mechanism yet performs markedly worse than all GLM and Qwen models on grounding tasks. Substantial performance differences can also be observed among models that use the same image-token conversion strategy, showing no consistent trend.
The picture changes once training data are considered. The models with the strongest performance, namely GLM, Qwen, and Intern all explicitly report the inclusion of visual grounding data in their training corpus. Although Kimi-VL-A3B also mentions grounding-related data, its training is restricted to OS-agent grounding, whose domain differs substantially from the human-centric imagery used in our benchmark.
A further piece of compelling evidence comes from comparing Intern-S1 with VL3.5-VL3.35222 Apart from differences training data and the special tokens introduced in Intern-S1 enhance its scientific capabilities, the two models share an almost identical architecture. Nevertheless, their performance on human-centric visual grounding diverges substantially, with Intern-S1 performing markedly worse. This gap is consistent with the fact that Intern-S1 is designed as science-oriented large model whose training data are correspondingly biased, and its technical report does not explic- itly mention training visual grounding tasks.
Moreover, among models of comparable scale, grounding performance follows a consistent pat- tern: GLM outperforms Qwen, which in turn outperforms InternVL. This ordering matches the degree of format alignment described their technical reports. Intern does not mention any dedicated bounding box format alignment. Qwen states that it aligns bounding box outputs to JSON
35

---

Published as a conference paper at ICLR 2026
with absolute xyxy coordinates. GLM constrains outputs to normalized xуxy coordinates and in- troduces dedicated start and end tokens for bounding boxes. The progressively stronger alignment requirements correspond directly to progressively stronger grounding performance, reinforeing the conclusion that both training data and output-format alignment play a decisive role in achieving high-quality visual grounding.
Challenges in left-right discrimination for body parts. Figure 5 presents confusion matrices for three MLLMs of different architectures and parameter scales on the Face Grounding and Body Grounding tasks. Across all six matrices, it is evident that these models encounter notable difficulty in distinguishing between the left and right hands or feet, whereas their ability to differentiate left from right on facial features shows significantly stronger. Even Qwen2.5-VL-722 which overall appears relatively robust, exhibits clear tendency toward such confusion. A plausible explanation is that the left-right configuration of facial components remains fixed in image space: when person faces away from the camera the face simply not visible. is impossible for a person's real left eye to appear on the opposite side of the nose, unless we could somehow see their eye through the back of their head. In contrast, the human body does not impose such a constraint, and the left and right hands may appear on either side of the torso depending pose or viewpoint. This difference during both training and testing makes left-right discrimination of body parts more challenging for the models.
Judgment tasks have precision-recall tradeoff. Table reports the precision, recall, and F1 score of different models when deciding whether to answer abstain Judgment-type questions. Most models exhibit relatively low precision, indicating that they often fail to abstain when no suitable person is present in the image. This reflects a persistent tendency toward hallucination. By contrast, recall is generally high, showing that when the correct individual is indeed present, the models usually succeed in identifying the target.
Among all models, Qwen2.5 Swen2.5-VL-722. VL-72B achieves the highest overall Fl score, but its behavior reveals a notable tradeoff. Its recall is slightly lower than that of Qwen2.5-VL-322. suggesting that its stronger control over hallucination comes at the cost of missing some valid answers. A case-by-case inspection further highlights this effect: even on Body Grounding tasks that contain no judgment component, Qwen2.5-VL-722 frequently refuses to output the full body bounding box, citing in- complete visibility of the human body despite explicit instructions to annotate "all visible body". As a result, its final loU for body drops to only 0.31, far below the 0.89 achieved by Qwen2.5. VL-7B. By examining the pattern of refusal responses, we find that it is likely a byproduct of CoT alignment. This suggests that during the collection of CoT samples, the dataset may have overrep- resented answers to questions about objects that do not exist in the image. As a result, the model sometimes refuses to answer legitimate question by incorrectly claiming that "the requested object does not exist", particularly in certain specific scenarios. These observations suggest that mod- els with stronger hallucination prevention mechanisms may, paradoxically, follow instructions less faithfully because they act with excessive caution.
Extra Judgment question component reduces performance on original task. Table 3 also presents the effect of adding Judgment component on related tasks. The four columns on the right compare model performance on Identify Bounding Box versus Judgment Bounding Box, and Identify Short-Answer versus Judgment Short-Answer. In Identify tasks, the model must locate the correct individual based on a single distinguishing feature and then answer the question. In Judg- ment tasks, MLLMs must first identify person satisfying two specified features and, only when such a person exists, provide the answer. Across most models and metrics, the Judgment versions yield lower performance than their Identify counterparts, indicating that the added complexity of feature matching makes final question completion more challenging, although the extra feature provides additional cues for localization.
Intention discrimination is easier than cause discrimination, which in turn is easier than emo- tion discrinination. Table 2 shows that, across all evaluated models, accuracy almost consistently follows the pattern Intention > Cause > Emotion. Intention discrimination benefits from strong, visually grounded cues such as body posture, gaze direction, and surrounding objects, making comparatively straightforward for models to infer a person's 's likely goal or purpose. Cause discrim- ination requires imagining scene-level causes and predicting future consequences, which depend on higher-level commonsense reasoning and temporal context that are not directly observable, thereby lowering accuracy. Emotion discrimination proves to be the most challenging because emotional
36

---

Published as a conference paper at ICLR 2026
states are inherently subtle and subjective, facial expressions can be ambiguous culturally vari- able. This progressive increase in abstraction explains the observed hierarchy of model performance.
Poor performance of proprietary models on certain tasks. We observe that among the four proprictary models tested, even the strongest ones, such as Gemini-2.5-PHt and GPT-5, still fall behind the best-performing open-source model certain metrics. This pattern be explained by three factors: the safety alignment applied to commercial models, inherent limitations in their grounding abilities, and the general performance constraints of older model architectures. Table reports the proportion of refusal cases across eight evaluation dimensions. We identify a refusal by checking whether the output contains the word "sorry". Only GPT-5 and GPT-5- -mii exhibit such behaviors, which matches the emphasis on safety alignment described in their system card (OpenAI, 2025).
A closer, case-by-case examination shows that GPT-5 often refuses to provide detailed analysis of facial attributes because its safe-completion mechanism is triggered. In these cases, the model tends to abandon the original instructions or required format and instead proposes an alternative task, which is usually irrelevant or inappropriate. Table 5 shows the frequency with which these refusals mention keywords related to policy or privacy (keywords: "policy", "policies", "prohibit", "sensitive", "not appropriate", "privacy"). These instances are mostly caused by tasks that require inference about gender, race, or other rppearance-related features. The most affected dimensions include Face Understanding, Emotion Diserimination, and other categories that involve facial in- formation are also affected. This behavior appears only in the GPT-5 series; GPT-4o and all other open-source or proprietary models do not show similar patterns. The overly conservative alignment practices in GPT-5 therefore have clear negative impact on its evaluation scores.
Even after removing the influence of safety alignment for GPT-5, and even for models not affected by such constraints, such as 2emini-2.5-Pro or GPT-4o, we still observe a considerable performance gap relative to the leading open-source model. As shown in Table 12, this gap is concentrated in IoU metries, which correspond bounding-box tasks. We further break down IoU performance by bounding- box type and compare each proprietary model with the top open-source model, as shown in Table 6. A consistent pattern emerges. All proprietary models perform worse than GLM-4.5V overall, but the degree of underperformance varies. For ggaaine boxes such as whole face and whole body, the decline is moderate, roughly a 50% reduction, indicating that these models can images. For fine-grained body parts, however, the performance
still produce global annotations drop is dramatic, often falling to one-fifth -ifth or even one- hundeehdd of the open-source baseline. For comparison, another relatively weaker open-source model, Intern VL..55388, shows similar perfor- mance to the proprietary models on coarse-grained boxes, but remains significantly stronger than all proprietary models fine-grained facial and body-part annotations. This suggests a consistent cognitive bias in the proprictary models when dealing with human-image grounding. They tend to recognize entire faces or bodies clearly, but their understanding of specific body parts much less accurate. This pattern appears consistently across all four proprietary models we evaluated.
The most plausible explanation is the imbalance in training data. Open-souree datasets often con- tain many annotations for whole faces and whole bodies, while annotations for individual body parts are much rarer, which limits the models' ability to generalize to fine-grained grounding tasks. This gap may stem from the slower iteration pace of closed-source models, coupled with the fact that the requirements for fine-grained visual alignment in semantic tasks tend to favor scenarios where open-source models perform better than closed-source models. Common scenarios for fine-grained visual alignment in semantic tasks include large-scale data annotation and fine-tuning for down- stream tasks. Both of the scenarios more conducive to open-source models than closed-source ones. Consequently, the latest open- open-source models have also been optimized for the category of task accordingly.
Among the proprietary models, GPT-40 performs notably worse than the others. This observation aligns with findings reported in several previous studies (Fu al., 2024b; Lu al., 2024; Zhang et al., 2024b). GPT-4o is an older model released in mid-2024, and its weaker performance relative more recently released systems, including open-source models, is therefore expected.
37

---

Published as a conference paper at ICLR 2026
(b)
(C)
(8)
(d)
(e)
Figure 9: Images for case study on failures of best models.
J FURTHER CASE STUDY AND FINDINGS
By examining the errors made by state-of-the-art models (GLM-4.5V and Gemini-2.5-Pro. across different tasks, we identify several shortcomings that these models still exhibit. For example, GLM- 4.5V tends to commit to an answer prematurely and then craft its reasoning around that predeter- mined conclusion, rather than analyzing first and deciding afterward. Due to the autoregressive nature of LLMs, analysis should logically precede the final answer to achieve higher accuracy; how- ever, current models sometimes output answer before conducting full evaluation. In the task corresponding to Figure 9a), which asks the model to select the most accurate facial feature, GL-- 4.5V gives the following reasoning: "Looking the image, the person's 's face has a rounded and full appearance, which matches the description of a chubby face. The other options (big nose, narrow eyes, bushy eyebrows) do not accurately describe the facial features visible here". Here, the model implicitly locks onto the "chubby face"' option before performing a comprehensive comparison. In contrast, a step-by-step analysis followed by final decision often produces better results; Gemini- 2.5-Pro adopts this strategy on the same question and successfully reaches the correct answer.
GLM-4.5V shows limited discrimination ability for fine-grained clothing categories. In Figure 9b),, the model is asked to choose the option consistent with the person's clothing. It is misled by distractor describing a "light green fluffy jacket".. Although the individual does wear a jacket-like garment, it is not fluffy jacket. The correct option should be "a white and yellow shirt". The jacket is more visually salient but mismatched in category; the less salient inner shirt, which exactly matches one option, is overlooked. A similar issue appears in Figure 9()), where the model must identify a shared attribute of two people. One distractor claims both individuals are "wearing black shirts," yet the woman is clearly wearing tunic dress rather than shirt, so this option should be eliminated. The correct answer is that neither person has a double chin, which is visually apparent.
Despite its strong ability to predict bounding boxes, GLM-4.5V is less capable of inferring object identity directly from coordinates. In Figure 9(d), the model is asked whether a bearded person appears within the bounding box [961,90 .9, 129,355] in 1920x1080 image. GLM-4.5V incorrectly answers that such a person exists. Its reasoming shows that once it concludes the box lies "near the center-left", it immediately assumes the region corresponds the man on the left, even though the box is not actually on the left and the person inside is woman, not a bearded man.
Gemini-2.55 Pro also makes rrieate mistakes, particularly in distinguishing left from right when analyzing facial direction. Since closed-source models generally perform poorly on grounding tasks, is difficult to rely on IoU metrics to verify whether they truly understand left-right distinc- tions: nonetheless, other examples also suggest such issues. In a Multi-Face task requiring the model to rank face images by the number of attributes they satisfy, one attribute is "face turned to the left side of the image". In the example shown in Figure 9(),, both the viewing angle and facial cues clearly indicate that the face is turned to the left, yet Gemini-2.54. 2.5-Pro incorrectly states that the face is turned to the right, which leads an incorrect count of satisfied attributes.
38

---

Published as a conference paper at ICLR 2026
Question Distribution by Difficulty and Type
Correct Count Model Abllity Group and Difficuity Level
2048
2842
2140
50000
2000
1008
Medal Abollyy Group
548
18-211
Har
Figure 10: Difficulty analysis of 19,945 questions in Human-MME. The questions are categorized into five levels based on their accuracy rates. Models are grouped into five tiers according to their average scores across all dimensions. The figure presents the difficulty distribution of questions across eight different dimensions, as well as how the total number of correct answers changes for models of different capability tiers when facing questions of varying difficulty.
K DIFFICULTY ANALYSIS
We aggregate the responses of 20 models for each question and categorize the questions into five groups based on the number of models that answered them correctly: very easy (17-20 models correct), easy (13-16 models correct), medium (9-12 models correct), hard (5-8 models correct), and very hard (fewer than models correct). The stacked chart on the left side of Figure 10 shows the difficulty distribution both overall and across each individual dimension. Although different dimensions manifest distinct difficulty lendencies, once they are combined into the full Human- MME dataset, the diversity of question types results in remarkably balanced difficulty distribution. The difference between the largest and smallest question counts across difficulty levels is within 1000, ensuring sufficient discriminative power for evaluating models.
We then rank the models according to their average scores and grouped them in sets of four, dividing the 20 MLLMs into five tiers: Basic, Intermediate, Proficient, Advanced. and Top-tier. As shown in the bar chart on the right side of Figure 10, for each difficulty level, the number correctly answered questions steadily increases model performance improves. Moreover, the easier the questions are, the more substantial the improvement among weaker models; conversely, the harder the questions are, the larger the performance gap among stronger models. Within each model tier, we observe consistent pattern in which easier questions lead to higher number of correct answers. benchmark with balanced
Taken together, these observations demonstrate that Human-MME and fair difficulty distribution, providing meaningful discrimination across models of all capability levels.
L THE USE OF LARGE LANGUAGE MODELS
We use large language models solely for polishing our writing, and we have conducted careful check, taking full responsibility for all content in this work.
39


Reviews:
[{"review_id": "iyVid8cDrC", "reviewer": "Reviewer_cQVR", "summary": "The paper proposes Human-MME, a large-scale benchmark for evaluating multimodal large language models on human-centric scene understanding, covering eight progressive dimensions from fine-grained perception to high-level reasoning, with 19945 samples, an automated–manual annotation pipeline, and evaluations on 17 models.", "questions": "1. Could the authors clarify whether the poor performance of proprietary models such as GPT-4o on face-related or human-identity tasks is due to policy or privacy restrictions, or inherent model limitations?\n2. Would it be possible for the authors to include or discuss results from newer proprietary models to better reflect the current performance landscape?\n3. Could the authors provide a more detailed error analysis for the best-performing models, like highlighting representative failure cases across different task types to better understand current limitations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses the underexplored problem of human-centric multimodal understanding and provides a clear motivation for its importance.\n2. It provides a holistic eight-dimension evaluation that spans fine-grained perception to high-level reasoning, with 19,945 real-world image–question pairs covering 43 diverse scenarios across four human activity domains.\n3. The hybrid automated and expert-reviewed annotation pipeline combines scalability with high annotation quality and reduced bias.", "weaknesses": "1. Proprietary models such as GPT-4o perform poorly on face-related or human-identity tasks, which may stem from privacy or policy restrictions, but the paper does not discuss this possible factor.\n2. The evaluation could be improved by including results from newer proprietary models to better reflect the current state of the field.\n3. Lacks deeper error analysis, especially for the best-performing models, to identify detailed failure patterns across tasks and provide clearer insights for future model development.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vNeoG7scwz", "reviewer": "Reviewer_jnXs", "summary": "This paper presents Human-MME, a large-scale benchmark for evaluating human-centric multimodal large language models (MLLMs). It includes 19,945 image-question pairs across 43 scenarios, covering 8 core human understanding dimensions and 5 question types. Through both automated and manual annotation, the evaluation of 17 major MLLMs reveals clear weaknesses in tasks like left-right hand recognition, emotion analysis, and hallucination suppression.", "questions": "- Has a quantitative analysis been conducted to prevent data leakage (i.e., training/testing overlap)? If models have potentially seen duplicates such as HICO-DET, how might this affect the evaluation conclusions?\n\n- The remaining issues will be further explored after communicating with the other reviewers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Pioneers the construction of a systematic, multi-dimensional, end-to-end benchmark for human-centric scene evaluation, offering comprehensive content coverage.\n\n- The benchmark is data-rich and evaluates a wide range of capabilities: from low-level perception (face, body, interaction) to high-level reasoning (emotion, intent, causality, multi-image/multi-object), forming a progressive evaluation hierarchy with rich annotation dimensions (such as left/right hand, emotion, intent, causality).", "weaknesses": "- Although the benchmark is extensive in both scale and coverage, its methodology (automated annotation plus manual validation, Q&A-style evaluation) is largely similar to existing benchmarks (e.g., MMBench, MME), lacking methodological novelty.\n\n- The current benchmark focuses mainly on static images, without covering video-based continuous scenarios or multimodal temporal reasoning, which falls short of some human-centric video benchmarks in terms of scenario, action, and intent diversity.\n\n- The analysis of model differences and conclusions is not sufficiently in-depth. While the performance disparities among models are reported, the study lacks a thorough exploration of the underlying causes—for example, the influence of training data, visual encoder, or alignment strategies on the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsCyjRq21t", "reviewer": "Reviewer_rEtP", "summary": "This paper addresses the gap of insufficient human-centric evaluation benchmarks for Multimodal Large Language Models (MLLMs) by proposing **Human-MME**, a holistic benchmark for human-centric scene understanding. Key features of Human-MME include: (1) Diverse human scenes (4 primary domains, 15 secondary domains, 43 sub-fields) and 19,945 real-world image-question pairs; (2) Progressive evaluation across 8 dimensions (from fine-grained perception like face/body understanding to high-level reasoning like intention/causal discrimination); (3) High-quality annotations via an automated pipeline (YOLOv11, Qwen2.5-VL, Grounding DINO) plus manual refinement (Gradio interface). The paper conducts extensive experiments on 17 state-of-the-art MLLMs (open/closed-source), revealing critical findings (e.g., stronger scaling effects in Choice/Ranking tasks, training data’s greater impact on grounding than model size, left-right body part discrimination challenges). Contributions include the benchmark itself, a scalable annotation pipeline, and insights guiding MLLM development for human-centric understanding.", "questions": "* The results in Table 2 show that models like Gemini-2.5-Pro and Intern-S1 have very poor grounding (BBox) scores but strong high-level reasoning scores, while GLM-4.5V shows the opposite pattern. Does this suggest a fundamental architectural or training trade-off between fine-grained spatial localization and high-level abstract reasoning, or do you believe this is purely an artifact of their specific training data (as discussed in 3.3)?\n* Did you measure difficulty (e.g., inter-model accuracy variance, human annotation time) across dimensions/sub-fields? Are there imbalances (e.g., over-representation of easy Daily Life questions) that might skew model rankings?\n\nI look forward to an active discussion with the authors during the rebuttal phase and will revise my score accordingly.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Human-MME fills a key gap by covering 8 dimensions that progress from fine-grained perception to abstract reasoning (e.g., causal inference), with 21 question types. This design captures the full complexity of human-centric understanding, unlike existing benchmarks that focus on either coarse perception or single-task reasoning.\n\n- The hybrid automated-manual pipeline balances scalability and accuracy. Automated steps extract fine-grained features (13 bounding box types, 42 facial attributes) using state-of-the-art models, while manual review (cluster de-duplication, instance correction) and identity-neutral processing (removing age/gender from emotion/intention labels) reduce bias and errors—critical for reliable human-centric evaluation.\n\n- Evaluating 17 MLLMs reveals nuanced findings and general trends (scaling benefits Choice/Ranking tasks most). These insights directly guide future MLLM research (e.g., prioritizing grounding data over size for spatial tasks).", "weaknesses": "* For the most abstract dimensions (Intention, Causal, Emotion Discrimination), the \"ground truth\" options are narratives generated by Qwen3. Although these are reviewed by humans, it's unclear if these annotations represent genuine human ground truth or simply a \"plausible, model-generated\" narrative. Using model-generated text to benchmark the reasoning of other models in such subjective tasks is a potential methodological weakness that merits more discussion.\n* The authors need to provide more latest MLLMs to support experimental conclusions, such as Claude-4-Sonnet,  GPT-5, and o3.\n* The data and code not be publicly released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P5Mv7XEy2V", "reviewer": "Reviewer_ZsgC", "summary": "This paper proposes Human-MME, a comprehensive evaluation benchmark for the human-centric scene understanding capabilities of Multimodal Large Language Models (MLLMs). It aims to address the issue that existing benchmarks lack evaluations of fine-grained human perception and high-dimensional causal reasoning. The benchmark secures the quality of annotations by collecting and screening 16,765 high-quality images from free stock image libraries and the HICO-DET dataset, combined with an \"automated annotation (5-step process) + manual adjustment (cluster deduplication and instance correction)\" approach. It designs 19,945 image-question pairs covering 8 evaluation dimensions (such as face understanding and causal discrimination) and 5 types of question-answer paradigms. Experiments on 17 mainstream MLLMs show that GLM-4.5V achieves the best overall performance, while Qwen2.5-VL-72B excels in high-level reasoning. Meanwhile, the experiments reveal the limitations of the models in aspects like left-right discrimination of body parts and precision-recall trade-off in judgment tasks, providing support for research on human-centric scene understanding of MLLMs.", "questions": "It is suggested that the authors add a table to comparatively demonstrate the differences and advantages between Human-MME and previous relevant benchmarks.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To address the issue that existing benchmarks lack comprehensive evaluation of \"fine-grained perception + high-dimensional reasoning\" in human-centric scenarios, this study constructs, for the first time, an 8-dimensional progressive evaluation system ranging from face/body understanding (fine-grained) to intention/causality/emotion discrimination (high-level reasoning). This system covers the core capability dimensions of human scene understanding and can more comprehensively reveal the capability boundaries of Multimodal Large Language Models (MLLMs) in this field.\n- Five types of question-answer paradigms are designed, including multiple-choice questions, bounding box questions, short-answer questions, ranking questions, and judgment questions, to adapt to different evaluation needs. Moreover, each type of paradigm is matched with exclusive and precise metrics (e.g., Intersection over Union (IoU) for bounding box questions, Kendall’s τ for ranking questions, and F1 score for judgment questions). This avoids the flaw of a \"one-size-fits-all\" metric approach and enables a more objective assessment of model performance across different tasks.", "weaknesses": "- The annotation of high-level reasoning tasks such as intention, emotion, and causality relies on Qwen3 to generate \"identity-neutral descriptions\". Although manual verification is conducted, there are still certain subjective differences in judgments like \"reasonableness of intention\" and \"emotion matching degree\" (for example, different experts may have different opinions on whether \"a person's intention is 'preparing for an adventure' or 'organizing equipment'\"), which may affect the objectivity of the evaluation for such tasks.\n- The human-centric scenes in the dataset do not explicitly mention cross-cultural diversity (such as differences in clothing and behavioral habits across regions), nor are there specially designed evaluation scenarios for specific populations (such as people with disabilities and groups of different age groups). This makes it difficult for the benchmark to measure issues like \"cultural bias\" or \"insufficient population coverage\" of MLLMs in human-centric scene understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)