Image Quality Assessment for Embodied AI
Authors: Chunyi Li
Jiahao Xiao
Jianbo Zhang
Farong Wen
Zicheng Zhang
Yuan Tian
Xiangyang Zhu
Xiaohong Liu
Zhengxue Cheng
Weisi Lin
Guangtao Zhai
Venue: ICLR 2026 Poster (2026)
Keywords: Image Quality Assessment; Image Processing; Perceptual Quality; Embodied AI;
Primary area: datasets and benchmarks
TL;DR: Image quality assessment in Embodied scenario

Abstract: Embodied AI has developed rapidly in recent years, but it is still mainly deployed in laboratories, with various distortions in the Real-world limiting its application. Traditionally, Image Quality Assessment (IQA) methods are applied to predict human preferences for distorted images; however, there is no IQA method to assess the usability of an image in embodied tasks, namely, the perceptual quality for robots. To provide accurate and reliable quality indicators for future embodied scenarios, we first propose the topic: IQA for Embodied AI. Specifically, we (1) based on the Mertonian system and meta-cognitive theory, constructed a perception-cognition-decision-execution pipeline and defined a comprehensive subjective score collection process; (2) established the Embodied-IQA database, containing over 30k reference/distorted image pairs, with more than 5m fine-grained annotations provided by Vision Language Models/Vision Language Action-models/Real-world robots; (3) trained and validated the performance of mainstream IQA methods on Embodied-IQA, demonstrating the need to develop more accurate quality indicators for Embodied AI. We sincerely hope that through evaluation, we can promote the application of Embodied AI under complex distortions in the Real-world.

Published as a conference paper at ICLR 2026
IMAGE QUALITY ASSESSMENT FOR EMBODIED AI
Chunyi Li,.2,3, Jiahao Xiao Jianbo Zhang Farong Wen 2.2 Zicheng Zhang² Yuan Tian
Xiangyang Zhu? Xiaohong Liu' Zhengxue Chengl.t Weisi Lin Guangtao Zhai 1,2
Shanghai Jiao Tong University
Shanghai Al Lab 3 Nanyang Technological University
Project Page:
Human Visual System
Machine Visual System
Perception] is poor. Since
[Perception good Since
Cogaition] table objects
[Cognition]
affected by
mup remains unchanged
[Decision] and [Execution]
[Decision] and [Execution]
aligned with [Cogaition].
alggedd with [Coguition).
Robot Visual System
S [Perception]
Cogaltion
[Decision]
[Execution]
Eye Camera
It Cembrum
Cenébellum R: VLA
R: YLM
[Coguition]
[Decision]
Decision FS [Execution]
[Peeception]: Quality
Dumriion
Extemal
Reality
Embodied IQA
"escresse
Intemai
VLM right grab
Task: Pio the rode unio fohur bose!
The path piatned by VLA -dis- is
Reality
00.00
siglilyy off Just grüpper
Ssallate nie acgumve of diintition [Cagnitton], [Decidin)) (Esecution)
condioatt paroos nslis
ngo
Figure 1: The significant gap between human, machine, and robot visual systems. Humans and Machines are sensitive to different distortions, while Robots have Decision and Execution steps beyond Cognition, highlighting the importance of Perception quality index for Embodied AI.
ABSTRACT
Embodied AI has developed rapidly in recent years, but it still mainly deployed in laboratories, with various distortions in the Real- world limiting its application. Traditionally, Image Quality Assessment (IQA) methods are applied to predict human preferences for distorted images; however, there is no IQA method to assess the usability of an image embodied tasks, namely, the perceptual quality for robots. To provide accurate and reliable quality indicators for future embodied scenarios, we first propose the topic: IQA for Embodied AI. Specifically, we (1) based the Mertonian system and meta-cognitive theory, constructed perception- cognition-decision deroonnddccisiooeeetttioooo pipeline and defined a comprehensive subjective score collection process; (2) established the Embodied-IQA database, containing over 30k referencce/distorted image pairs, with more than 5m fine-grained annotations provided by Vision Language Models/Vision Language minionnmodds//eal-eeeee robots; (3) trained and validated the performance of mainstream IQA methods on Embodied-IQAA IQA, demonstrating the need develop more accurate quality indicators for Embodied AI. We sincerely hope that through evaluation, we can promote the application of Embodied Al under complex distortions the Real-world.
INTRODUCTION
To achieve Artificial General Intelligence (AGI), Embodied AI, as a bridge connecting external and internal realities, has developed rapidly in recent years. Relying its ability to interact with the physical environment, Embodied Al Duan et al. (2022): Savva et al. (2019): Liu et al. (2024b) has been applied to simple scenarios such as factories and warchouses, but it not yet capable of handling complex environments like autonomous driving and wilderness exploration. Unlike traditional roboties driven by fixed algorithms, Embodied Al collects signals from the Real-world

---

Published as a conference paper at ICLR 2026
and is therefore susceptible to distortions. For example, pick-and-place task may be successfully debugged in the laboratory, but it may fail in Real-world applications due to slight lens shaking, Therefore, the preferences of Embodied AI should be analyzed to filter out these low-quality images.
For human viewers, this problem can be solved through Image Quality Assessment (IQA) metrics. For example, in streaming media, collect human subjective preferences for distorted images. Since human resources are expensive, IQA will develop objective quality indicators to fit subjective scores. Similarly, for Embodied AI, it also necessary collect the success rate of downstream tasks using distorted images, quantifying the fidelity with the reference results, and developing IQA metrics.
Unfortunately, due the significant differences between Human/Machine/RRool Visual Systems (HVS/MVS/RVS) previous IQA methods cannot be directly transferred to Embodied AI scenarios, as shown in Figure | First, HVS and MVS are sensitive different types of distortions. Humans are sensitive to distortions such as noise and compression, which do not affect the downstream tasks of machines. Brightness and contrast are the opposite. Therefore, as a machine, Embodied AI cannot use the past human-oriente processing methods. Second, although MVS and RVS are sensitive to some common distortions, the perceptual quality of a general machine only depends on the performance of segmentation and detection tasks that belong to Cognition. Robots, however, have subsequent Decision and Execution steps. High fidelity in the previous step does not guarantee the success of the next. Therefore, unlike general machine, it is necessary to fully consider Cognition, Decision, and Execution to characterize the Perception of Embodied AI. Considering these issues, we first attempt to implement IQA metrics into Embodied AI Our contributions summarized as follows:
Theory:
We refer the Mertonian Law in robotic intelligence construct the Perception- Diniition-Deciiion--oe
pipeline. We define tasks related to Embodied Perception and specify the subjects for each step in Cognition, Decision, and Execution.
Data: We add corruption to images in Embodied tasks, collecting over 36k refer- ence/distorted image pairs. We perform inference using Vision Language Models (VLM) and Vision Language Action-model (VLA) for over 5 million annotations. This large-scale database can effectively drive the development of quality metrics for Embodied AI.
Experiment: We experiment with 15 advanced IQA methods on our database, proving that more sophisticated IQA metries are needed for Embodied AI. Additionally, we first conduct real-world experiments in the IQA field, executing 1.5k Embodied tasks in the Real-world, revealing the internal connections between Cognition, Decision, and Execution.
2
RELATED WORKS
2.1 MERTONIAN SYSTEM FOR ROBOTIC INTELLIGENCE
Intelligent models are divided into Newtonian and Mertonian Wang (2012) systems. Newtonian systems typically refer to those systems that can be precisely described and predicted by deterministic physical laws, such as classical mechanical systems. contrast, Mertonian systems involve systems that include 'free will', whose behavior is influenced by feedback between beliefs and actions. The characteristic of such systems is that even given the current state and control conditions, the next state of the system cannot be accurately obtained by solving, so its behavior is difficult to predict precisely.
HVS and MVS can both be simplified as Newtonian systems, since their Decision and Execution processes are robust. However, the Decision and Execution of RVS do not fully match Cognition. For example, a deviation of one character Cognition may greatly change the pose in Decision; a one-centimeter path offset Decision may also cause Execution hit obstacles. Since the impact of distortion on these steps is unpredictable, it is necessary to handle them separately for the IQA task.
2.2 IMAGE QUALITY ASSESSMENT FOR MACHINE
Since 1999, Perception has been recognized as the first step in the interaction between Al agents and external reality, whose mechanism Rickel & Johnson (1999): Cassimatis et al. (2004): Lepora & Pezzulo 2211);; (.15):.53 Balke & Gilbert (0014) has been revealed. However, no perceptual quality score has been assigned to each distorted image like current IQA et al. (2224b 2025b); Liu et al. (2024a) metrics, which is exactly Embodied AI needs in Real-world applications. In the past decades, IQA has been widely studied as shown in Table [ but none of them meet the above needs of Embodied AT.
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison of Embodied-IQA with other perceptual quality databases. As machine- oriented database, Embodied-1O has not only more image samples and a larger annotation scale, but also comprehensive labels on three downstream steps. [Keys: Cognition, Decision, Execution]
Image
Corrugtion
Annotation
Database
Reference
Distorted
Resolution
Types
Strength
Num
Dimeeisio
Subjccts
Cog.
Dic.
LiviMuy
(General)
TID201
Pomomarenko
2015
3%
512
24
514k
Human (General)
KADID-TU
TU Lin al
2019
81
10k
1k
25
5
304k
Human (Gienorl)
CLIC202
Ballé & et 2020
585
3%
10
3
48-kk
Human (General)
NTIRE202
2022
1.13m
Human (General)
AGIQA-3K (oA--3KLien Ei
X723
3%
125k
1+1
Human (Mullimodal)
NTTRE202 Li
al.
102
20R
Ik
420k
I+]
Human (Multimedal)
MiD
al
3025m
30k
30
2.25m
Machine (General)
Ziann
Machine (Genural)
Embodied- B.A
1.23k 36.9%
30
5.53m
3+3+1
Machine (Robot)
First, most databases are oriinte,,,nted,, with only two coarse-grained Zhang al. 2024c d). two
fine-grained Li et al. (2025)); Zhang et al.] (2024)) machines as subjects (VLM and reinforcement learning); second, as mentioned above, no database covers Cognition, Decision, and Execution altogether. Considering the characteristics of Embodied AI, it is necessary to establish a new dataset accordance with the requirements of the Mertonian system.
3 DATABASE CONSTRUCTION
3.1 REFERENCE & DISTORTED IMAGE COLLECTION
To comprehensively characterize the data in Embodied scenarios, we collect 1,230 high-quality samples reference images. (see Supplementary for data source) All data are pre-processed by Q-Align Wu et al. (2024) to avoid pre-distortion before adding distortion. We focused on two aspects, ensure coverage of both real and simulation, as well first-person and third-person perspectives. In addition, we divided the subjects and backgrounds into five categories to ensure versatility by involving each category in the database.
Sim2Real and Perspective,
each, as shown in Figure 2
For the distorted images, according to the corruption caused by the current communication protocols, 30 distortion types are considered and classified into 7 categories: Blur, various types of unclear image; L Luminance, global brightness changes; Chrominance, global color changes; Noise, random noise of different distributions: Compression. codec algorithm like JPEG; Spatial, local pixel-level changes; and others. For each distortion,
defined 5 intensity levels, ensuring the quality degradation perceived by the HVS is aligned at the same level. Thus, for each reference image, we randomly selected the intensity to add all the corruptions mentioned above, resulting in 36,900 distorted images. The referenne/ddstorteed image pairs will then be annotated by Embodied Al subjects.
3.2 PERCEPTION: TASK DEFINITION
Perception refers to receiving information about the external environment, where Embodied AI obtains information through sensors, similar to human sensory organs. Considering that more than 82% of human external input signals come from vision, simplify this step of Embodied AI to the camera. First of all, we need to clarify the factors that Embodied Al focuses on in Perception. When viewing an image, HVS pays attention to factors such as brightness/chromatiitt while MVS/RVS relies on specific downstream tasks. Therefore, based on information such as objects, layout, and environment in the image, we manually annotate 5 tasks for each reference sample in natural language, as in previous MVS et al. (2025a) works. The difficulty of the tasks here increases in sequence and limited to [Cover, Insert, Move, Pick, Place, Pour, Press, Pull, Push, Twi st] to avoid being too difficult. All subsequent steps are based on the task corresponding to each image, and the image quality depends on the similarity of the reference/distortedd image pair inference results.
3.3 COGNITION: VLM ANNOTATION
Cognition refers to the process of processing and understanding information after perception, includ- ing recognition, classification, memory, and reasoning. This function of Embodied AI is implemented
3

---

Published as a conference paper at ICLR 2026
Reference Data Collection
(1,230 instance)
Distorted Data Collection (36, instance)
Type tu Sencken heppattee o0 Sadignand
Otbers
Blar
Simuatton
Real
Simubation
Fins Think-pence
Embodied
Chrominance
IQA
Database
(6) Daily
Luminance
Compresion
3C
Noisc
(d) Home
(d) Iechorrll
(d) Sorfwore
Ldh
Oi-Chhee
Pereeption Question (6,150 fasks)
Cognition Annotation (2,767,5000 Inbels)
Docision Annotation (2,767,500 labels)
Difficall:: I: Pare back the board on aabie
15 mainstream VIM
mainstream VI.A
Difficalty Lower the oeller
olvicr to the lof.
<net- Mose doroane Mer greve houmd
Diffically:: Moss sho rnd blook rwo syuans bortioonally to Her noht beyynng she serical poetliss strksteged
sdielo Mose invart the Mark
(scoes: 4.38)
<dit> (0.000) 681] [0 15 [] (ecome: 3.85)
Willin There hoard on the sable
(earre 1.05)
<d62>: (8.20A 181../000111.....000 HND (0), (score. B.8-5)
Execution Vaildation (1,500 Inbets)
Reakwworld pio
dall: 13.2mm, 4.4cm)] awa frow nosh seaf: <dii2> Hu deskrop forco intorapption
Figure 2: Database construction of the Embodied-IQA, with 30k+ large-scale reference/distorrted image pairs, meticulously annotated with 2m+ fine-grained Cognition score from 15 mainstream VLMs, 2m+ Decision score from 15 VLAs, and 1.5k real-world experiments as Execution score.
through VLM, which corresponds to the human cerebrum. Specifically, we selected 15 commonly used VLMs for cognition, including:
VL Gao al. (2024).
Dong et al. 2024),
M-t.ll..occooo2222... //aan et al. (2024b)
VL.2 Chen et al. (202b)),
Intern Chen et al. (2025). Intern /hhu al. (2025), J. MPl Ye et al. (2024). Ovisl.5-
Gemma Team et al. (2025). ), 1--Llma Touvron et al. (2023). Ovis2 Lu et al. (2024). Phi3-Vision
Abdin al. (2024b). Phi3.5 3-5- Vision Abdin al. (2024a). Mullimmll Microsoft
Microso et al. (2025).
VL Yang et al. (2024), and Qwen2.5- VL Bai et al. (2025). To ensure usability in the Real-world, the parameter size we selected all below 8B to ensure real-time inference.
Considering the output modality is textual, VLMs will be required to solve the pre-defined task in about 10 words, and the difference between reference/distorteed output sentences will be measured. Specifically, the difference between the two output sentences includes three dimensions: accuracy, recall, and semantics, which are realized by the average of three classic indicators: GLEUUappinen eet al. (2002), ROUGE Vedantam et al. (2015). and CIDEr Vedantam et al. (2015).
3.4 DECISION: VLA ANNOTATION
Decision refers to selecting the best course of action based on goals, rules, and experience, according to Cognition results. This function of Embodied Al is implemented through VLA, which corresponds to the human cerebellum. Specifically, we selected 15 commonly used VLA for Decision, including: CogACTLi et al. (2024c), Mombodied-CoT/Zawalsse
et al. (2025), Octo Team et al. (2024). Open VLA Kim et al. (2024a), Open Kim et al. (2024c), OoanVLA-Goal Kim et al. (2024b), Kim et al. (2024b), Open Aloha-Pen Black et al. (2024b), 100AAloha-Towel Black al. (2024b). PlO-Allha-Tupp Tupper-re Black ct al. (2024b). PiO-Base Black et al. (2024a), PiO-Droid PiO-Droid[Pertsch Pertsch et al. (025), PiO-Fast Pertsch et al. and RT-X-1 Collaboration et al. (2025), whose parameter size is controlled at 8B.
Kim et al. (0024)). PiO-
(2025)
Noted that since Embodied-IQA first introduced VLA into the IQA task, we define the quality of VLA as three dimensions. First, we parse the 7-DoF Pose Poseooutput output field. According the mechanism of VLA, the first three represent position 2 (translation of the operator along the three-dimensional coordinate system, in mm), the middle three represent rotation (rotation of the operator along the dice-dimmnsiooal coordinate system, in rad), and the last one represents state (opening and closing of the operator, range [0- [0-1]). The position score based on the spatial distance of the coordinate points
'We will discard information heyond the above 7-DoF like depth, for alignment between VLAss. For two-arm VLAs, we only select the arm with the larger movement range.
4

---

Published as a conference paper at ICLR 2026
Somuntic
Qwan2.5-VL
Position
Phis Multimdai
Level
Rutation
Pa0-Baso
Ovis2
OpenVLA
MPlugOw3
Octo
Embodied-Cot
Level
Recall
IntemX2
Level
Co9ACT
Lovel
Level
Leval
Level
Level
Level4
Figure 3: Benchmarking & in 3 different score dimensions and 5 distortion levels. Their performance varies in 3 dimensions and decreases with the distortion. (Zoom in for detail)
Precision (0.31) Recall (0.31) Semantic (0.30) Position (0.25) Rotation (0.23)
State (0.28)
Figure 4: Correlation matrix &LMs& VLAs subjects, the a-o order follows Section 3.3 Darker colors denote higher SRCC, with the averaged SRCC attached to the bottom of the matrix.
obtained from the reference/distorted image, the rotation score is based on the cosine similarity of directional vectors, and the last one is the absolute difference. The three dimensions averaged after 0-1 normalization, and report the sum of the five task results as the final Decision score.
3.5 EXECUTION: REAL-WORLD WORLD VALIDATION
Execution refers to the process of transforming decisions into actual movements. This part of Embodied AI relies specific actuators, corresponding to the human motor system. Based on kinematic statistics, the upper limbs dominate among all muscles and complete over 50% of daily movements. Therefore, execute tasks based on the inference results of VLA, with three scenarios: (1) Success: Directly assign score 100 to the sample; (2) Failure: Measure the Euclidean distance between the reference and distorted results based on the final pose of the actuator, and deduct points in centimeters; (3) Emergency stop: the actuator hits the table or wall, directly assign score O. Considering the uncontrollable factors in real-machine experiments, we only execute the task with the lowest difficulty level among the tasks verify whether the results of VLM and VI.A align with the Real-world.
the robotic as the most representative actuator. Specifically,
DATABASE ANALYSIS
This section analyzes Embodied-IQA. database from four dimensions: On the model level, we (1) Benchmark the VLMs and VLAs when processing the distorted images; (2) Explore the internal correlation between VLMs and VLAs; On the instance level, we (3) Compare the score distributions under different categories and distortions; (4) Analyze the distortion sensitivity of VLMs/VLAs.
[Benchmark] We select representative VLM and VLA in Figure 3 As the distortion level increases, the total scores of both VLM and VLA gradually decrease. However, the differences among the three scoring dimensions of VLM are much greater than the level of distortion. After distortion, the Semantic score of the image decreases relatively little, followed by Recall, and then Precision. In VLM, the reference/distorteed output of MPlugOw13 is the most consistent, while advanced models like Qwen2.5-VI. are less robust. Therefore, distortion usually affects VLM at the character level rather than the semantic level, and it is more likely to output redundant text than to lose information. Meanwhile, there are also sigmificant differences among the three scoring dimensions of VLA. In VLA, Octo shows strong robustness distortion in Position and Rotation, while models like CogACT and Open VLA are more faithful in State. Among them, State changes little after distortion, Position changes more, and Rotation is the most easily affected by distortion. This indicates that distortion
5

---

Published as a conference paper at ICLR 2026
30
20
10
J Gataiia fibz
Law Hlu
Sosiio bax
Color dilaion
Color skift
Color quasliaaion
HSV
Lab anumttin
IP3K
IPGG comprisionn
WEIP comprrosaton
White
30
200
Color noise
Inpelse mone
Mitiplicntive notse
Caunsion danale
CNS detois
Maximmo brightn
1 Minirann dutker
Moan
Meai dulloo
Clock jitering
Block
Blook lost
26
va
Block inteepolatioo
Blocks
Reselution linit
Grggcell quattlattio
Stappenos change
Cottrat change
Figure 5: Decision score visualized in 30 distortion subsets. Different color denotes distortion Level L-Level 2-Le 3-Level 4-Level 5. Different distortions affecting VLAs vary significantly.
usually does not affect the end operator but has a significant impact on the robot arm. Therefore, VLA needs to be carefully selected to ensure high-quality output of the first 6-DoF.
[Correlation] We visualize the correlation between subject models in Figure 4 according to the order of models in Section 3.3 3.4 through Spearman Rank-order Correlation Coefficient (SRCC). According to past IQA Li et al. (202a)) work, the correlation of HVS is usually above 0.6, while that of MVS is often less than 0.5. In specific machine tasks, detection has better correlation, while Visual Question Answering (VQA) may even be less than 0.4. For Cognition, which uses VLM to solve embodied tasks (a degraded form of VQA), the correlation is only about 0.3. Moreover, we find that the correlation of VLA is even lower than that of VLM, at around 0.25. Therefore, when evaluating VLM and VLA, using only one model as the subject is far from sufficient, especially for VLA. is necessary to collect their general preference. This also reflects the necessity of constructing the Embodied-IQA database and the separation of Cognition and Decision the RVS.
[Distribution] Since Decision is more downstream than Cognition and has never been deeply investigated in IQA, we show the distribution of Decision and put Cognition in the supplementary. Figure 3 shows the Decision score distribution under 30 types of distortions and 5 intensity levels. Results show that RVS and the traditional HVS have significant differences. Taking brightness as an example, Embodied Al is highly sensitive to 'Maximum brighten/darken' and the quality significantly decreases with the distortion level; however, it is rarely affected by 'Mean brighten/darken' and there is no significant distribution change from level 1 to 5. These findings emphasize the differences between RVS and HVS. Figure 6 lists the relation of the three Decision dimensions and the score distributions corresponding to different image categories. Overall, Position, Rotation, and State the Sim2Real distribution, the real scores are relatively high,
show independent distributions. indicating that VLA is better Real-World data; the first-person results are far worse than the third-person results, indicating that in the training data of VLA, the sampling tools and actuators rarely integrated, which needs to be improved in the future. These findings jointly support the rationality of the division of source image data and annotation dimensions in Embodied-IQA.
[Sensitivity] The difference between MVS and HVS causes VLM to be highly sensitive to some distortion categories, but robust to others; the difference between RVS and HVS also causes VLA to have a similar phenomenon. We combined the Www.NNiicaal--Dfffeeeee
(JND) theory to analyze the similarities and differences in the distortion sensitivity of VLM and VLA, as shown in Figure 7 The sensitivity is divided into three levels, Mild, Medium, and Severe, each accounting for one-third of all image samples, according to the Cognition and Decision scores. Results shows although
VLM and VLA have certain commonalities in sensitivity, there are also distortions such as Dis02 'Lens blur' that mainly affect VLM, or 1is15 (Multiplicative noise) that mainly affect VLA. This VLM& VLA- babedd partition further explains the gap between Cognition and Decision, and can serve important reference in the IQA for the Embodied AI topic in the future.

---

Published as a conference paper at ICLR 2026
Al
6:44
6:44
0.79
1.0
State
Rotaion
Portiinn
10c
6:30
0.30
861
Podiion
Rousina
Staee
A0
Resl Serreation Sim2Real
First Thrd Meciazical Duil Beemanic
Tool
Other
Home
Indinnial Sollorr: Background
Lab
Otbexs
Perspective
Mai Object
Figure 6: Correlation between the general Decision score and the dimensions from VLAs, and the score distribution in Sim2Real, First/Third perspectives, Main object, and Background sub-categories.
Leves
Lovi-
112
1:20
1.23
127
Lavel
144
1.30
1.74
136
134
174
1/1
330
1.
500
1.6
1.74
140
16
179
1W
5M
121
12
344
531
Malkan
Lod
105
1.
56
10
144
121
531
130
122
1
A22
3.5
335
Kevi3
3:2
1.2h
1.2
10
30
AM
A2
JM
XX1
121
3.5
A.
VUNI Disveroo Sceeeiitty
155
3.2
1.t
1:27
1.20
1.24
1.5
12
321
32
XM2
X22
322
X31
3:2
3.9
331
122
1.2
1.22
L00.
535
1.5
125
1.5
12
121
319
33
321
521
121
1.
131
131
534
327
3.1
1.34
116
120
12
S1N
52
521
10
17
135
1.
Figure 7: Just--oricable--fffeeeee (JND) of VLMs and VLAs. Distortion order follows Figure
5 EXPERIMENT
5.1 EXPERIMENT SETUPS
We randomly partitioned the Embodied-IQA database into the train/val set, with 29,520 and 7,380 reference/distortee image pairs according to 8:2 ratio. Since VLA is more downstream than VLM in the Mertonian system of Embodied AI, and previous orrieee IQA works are all based on VLM, we use the Decision scores from VLA in the main experiment and put the Cognition scores from VLM in the supplementary. 15 objective IQA metrics are implemented to predict the Decision score, including: (1) 5 baseline metrics: PSNR, SSIM Wang (2004), Brisque Mittal et al. (2012), Q-Align Wu et al. (0224), and Q-Align+ Zhang et al. (2025). Q-Align and Q-Align+ are loaded in quality/aesthetic weights. As the most commonly used IQA metrics, they are performed in
Lao Zero-shot setting: (2) 5 Full-Reference (FR) metrics: AHIQ Lao et al. (0022). CKDN Zheng et al.
DISTS DDng et al. (2020), LPIPS Zhang et (2018a,, and TOPIQ-FR Chen et al. (2024a). Those learning- based metries are fine-tuned on our training set, which takes both reference/distortee images as inputs; (3) 5 No- Mo-Reference (NR) metrics: CLIPIQA Wang et al. (022), CNNIQA Kang al. 2014), DBCNN Zhang et al. (2018b), QualiClip Agnolucei et al. (2025), and TOPIQ-NR Chen ct al. 2024a). They are also fine-tuned on a training set, which takes only distorted images as inputs.
To benchmark the performance of quality metries, three global indicators were employed: SRCC, Kendall Rank-order Correlation Coefficient (KRCC), and Pearson Linear Correlation Coefficient (PLCC), to evaluate the consistency between the objective quality score and the subjective MOS. Among these, SRCC and KRCC represent the prediction monotonicity, while PLCC measures the accuracy. We train the FR/NR metrics on Embodied-IQA with the learning rate as 10-3 for 50 epochs, under the default settings in pyiqa toolbox, and evaluate the performance on (1) 3 scoring dimension; (2) 3 IND-based distortion sensitivity in Figure 7 (3) First/Third person perspective; (4) Sim2Real; (5) 5 Distortion level. The partitioning, training. and testing pipeline is repeated 10 times, and the mean value is reported as the experimental result. The perception module is based on the Intel RealSense D455 array, supporting both First-person (wrist) and Third-person (top, side) as input. Cognition and Decision annotations are collected on two servers with 16xNVIDIA A800 SXM4 80GB GPUs, and then conduct IQA vraining/validation on one GPU above. Execution is achieved through the UR5 robotic arm and Robotiq 2F- 140 gripper, with a working radius 85em.
5.2 RESULT AND DISCUSSION
Table presents the performance of advanced quality metrics on the Embodied-IQA database. For the three dimensions of the 7-DoF VLA output, Position is the easiest to predict, followed by State, with Rotation being the most difficult. The SRCC of FR IQA methods with subjective labels is less than 0.65, while NR is even less than 0.6. Note that these methods have correlation close to 0.9
7

---

Published as a conference paper at ICLR 2026
Table 2: Using 15 advanced IQA metries to evaluate the Decision score from VLAs, including
zero-shot, FR, and NR metrics. [Keys: Best/Second best in group; Baseline; Lower than baseline.]
Dimeraion
Poeiiun
Rotation
Sunc
Fini Prnpustie
Thinl Panpasiie:
Eroog
Metric
KRCUT
PLOC1
SRCCT
KRCUT
PLOC1
SRCUT
KROUT
PLOC
SRCCE
KRCUT
PICC
SRCUT
KROCT
PLCCT
0.2367
0.1873
0.4084
0.354
0.1255
0.1015
0.2384
0.1522
0.3271
0.2561
04451
0.3493
0.4126
SS0M Wing Ttwss
0:4802
0.3545
0.6438
0.4245
0.2891
0.3912
0,3047
0,3668
0,3216
0.5434
0.4101
0.5256
0,5478
0,3815
0.5132
Zem
2013
0.3073
0.270
0.2519
0.2255
0.2965
0.2210
0.4020
0.3636
0.Algh We
0.5325
0.466
0.5387
0.5329
0.253
0.3346
0.663
0,4715
0.5854
0.5578
0-Aliyer (inn di B: 20o5
0.3275
0.2157
0.2410
0.3596
0.3465
0.3818
0.1402
0.0072
0.1272
0.663
0.3167
0.428
0.3104
0.2021
0.2649
1 Lasmet Myn
1360
3.3.6
0.76
1.667
6,0.5
17613
0,000
3.500
0.507
1.
0.00
10:90
17700
0.569
CRDN Zirog
06748
0.4917
0.671
0.606
04275
0.0001
0.634
0,515
06610
0.7610
0.7641
0.5634
0.75%
FR
DASTSDDigge e 38
A303
0,5797
0.4010
0.5846
0.5365
0.3748
0.5390
0.453
0,3180
0.6611
0.458
0.4644
0.6249
0.6545
0.4654
0.6642
PiPS Zeee 2i
0.3622
0.7642
0.511
0.297
0.1994
0.397
0-4110
0.269
0.1612
0.3300
04466
0.3113
0.4805
1OP
0.7745
014475
0.4836
0.671
0.5277
0.022
(3.1904
Wane al
0:1784
0.1177
01116
0.030X
000191
0.0666
111148
010770
0.067
01348
0004
0.021
022155
01777
011415
CNNN0
03188
0.318
0,4618
0.3221
0.3203
0.4667
0.377
0.5618
06407
0.668
Phase d
0.045
0.4303
0.6054
0,5325
0.3687
0.534
0,5419
0.376
0,5441
0.453
0.6005
0.6565
0.463
0.6009
QualiChip
Armelass dial mips
0.663
0.4619
0.6648
0.3387
0.3768
0.5384
0,558
0.3912
0.5518
0. 0.3620
0.5400
0.7218
0.524
0.7175
TUP1 NX
M0.46
0.70%
0,7686
0.5981
0,6120
0,5100
0,0060
0.500
Dimeraion
MBd Distonars
Maduu Disteeties
Sexen: Distetion
Realowekk
Simulation
SROC7
KRCUT
SRCOT
KRCUY
PLCCT
SRCU
KRCET
PLCC""
SRCCT
KRCST
PLCC
SRCCT
KROCT
PLCCT
PSNB
0.3515
0.3995
0.4035
0.2299
0.ENS5
0.3175
1111000
0.0906
0,1445
0.3754
0.3575
0.3167
0.3611
0.1775
0.3901
WSiM/Wand
2014
0,3778
0,2620
0,2581
11.15M3
0.1617
0.1004
0,2754
0.405
0,3754
0.4545
01533
0.3724
0,4550
Beough
0.150
0.150
0.1168
0.0017
0,7196
0.1677
0.061
Q.Aligh We ol
0.3541
0.2013
0.3057
0.4099
0.2875
0,3899
0.3158
0.2216
0.3725
0.6179
0.402
0.563
0.6565
0..600
0.6333
Q-Aliyn Prlnnn
2025
0.169
0.1132
0.1375
0,2799
0.1005
0.2287
0,0618
0,0425
0.1506
0.1677
0,2753
0.3620
0.424
0.3285
0.4711
0260
8481
0.005
0.725
6355%
0.555
CKDN
Z3rog <01
0,573
0.400
0.545
00000
0.4841
0,6079
0.000
0,5044
0.087
0.7277
0.5303
0.7311
0.7676
0.576
0.7709
FR
DISTS
Ding
0.4564
0.3191
04407
03323
0.1604
0.2755
03177
0.2155
0.7731
0.413
0.451
0.699
86608
0.459
0.6560
0:304
0.3884
0.3220
0344
0.19
0.966
0.318
TOPK
bes
10
0.712
0.5357
0.7626
0.7255
0:337
6.7881
0.7555
0.5634
0.7614
03004
04319
0-335
0.7115
0
0
CLIPIQA
02848
0.0863
0.0588
0.413
06600
0.0799
02288
0.1509
0.110
0.556
0.000
0.020
0.128
0083
0.101
CNNION
03766
0.2007
0.3647
0,3367
03314
0.3M07
0
0.2045
0,345
0.511
0,3907
0.175
01535
0.117
0.5911
NR
DBCNS
Phose 2i
30.86
0,488
0.3056
0.430
0,423
0.2994
0,4699
0.4622
0.3195
0,4459
0.0728
0.4533
0.6866
8.6488
0.4613
0.6345
QualiCTip
0.494
0,3425
0.4794
0.439
0.3094
0.4607
0,3152
0.2631
0.3917
06712
0,4882
0.6982
06618
0.4468
0.692
0.605
9.5164
0.7363
0.7174
0.5813
0:7174
0,7337
0.5312
0,7310
0,4073
0,8148
0.7697
0.7777
Dinezaiio
Dielend-1
Di-load-2
De-ladde3
De-level44
Dia-exel-5
Groep
Metric
SRCTS
KRCC
PLEXT
KRCCH
PLEC1T
SRCW
KRCE1
PLC0-
SRCUE
KRCET
PICC3
SRCU1
KRO01
PSNR
0,2931
0.1987
0,3856
0.2522
0.1532
0.2837
0.2155
0,3194
02249
0.239
0.1591
0.2642
0.3575
0.2444
0.3571
SSEM Wazy 2004
0:4397
0.4036
0.4258
0.3500
0,4446
0.3383
0.4745
0.5558
0.5327
7cm
Brisgh
2003
0.7690
0.1786
0.342
0.7466
0.1618
0.3445
0. 0,3516
0.7545
0.003
022088
0.1951
0:7604
0.9612
0.7454
0.1369
Aligt We of
0,5429
0,3415
0.4024
0.5534
0.3665
0.3951
0.5600
0.367
0,5133
0.5753
0.4015
0,5124
0.5567
0,3118
0.5275
0.2009
0.2210
0,2537
0.2585
0.3074
0.2694
6363
1.5616
10550
03.97
1...6
1066
0,76
CRON
73rro of 1l A0
04253
0.4513
0.6406
046122
0.4785
0.6845
0.7006
0,5105
0.124
0.7030
0.5131
0.7105
0.7194
0.527
0.7240
DISTS
0.5699
0,5774
0.5753
0,3903
0,5636
05431
0.3766
0.5834
0.5967
0,2736
01811
0.813
0.3723
0:3553
0.4033
0.354
0.207
0.428
0.669
0.3156
0.456
0.434
0.3122
0.436
TOPKSTO Teo be di 10
0.761
0,501
0.0134
1.7766
0.5826
0
1.7741
0.501
0.7871
0.7429
0.5682
6.777
0.776
0,5766
0.787
CLIPIOA
0.1541
0.1054
0.1204
0:035
0.004
0.1130
0.0466
CNNIQW
0.671
0.3362
0.507
0.5215
0:1669
0.5522
0.544
0.363
0,5710
0.5433
0.3783
0.5570
0.6033
0.4244
0.607
NR
DRCNS Thing dall
10
0.5982
0.4225
8,623
0.6194
0.4411
0.6210
0.4%%
0.6146
0.610
0.4944
06161
8.6141
0.433
0A387
Qualiclty
2025
0.5815
0.787
0,6359
0.4372
0.4509
0.0030
0.6399
TOPIQ-MX
2004
0.7547
0.5626
0.7761
0.7868
0.5937
8.7989
0.7566
0.5601
0.7695
0.7450
0.5532
0.7658
0.7529
0.5558
0.7670
with HVS in traditional human-oriented IQA tasks, which is sufficiently excellent, but they cannot adapt to the database we proposed, indicating that IQA for Embodied Al needs further research. For distortion sensitivity, the Decision scores are high under mild distortions, with small internal gaps and difficulty in prediction; whereas when the distortion becomes severe, the Distortion scores fluctuate more, resulting in a higher SRCC. For Perspective and Sim2Real, most IQA methods perform better on third-person, real images. Therefore, in Embodied scenarios, more content captured by the robotic arm itself or from simulation software should be used. For the five absolute distortion levels, the performance of IQA methods does not change much. This further proves that dividing distortion levels based on HVS is insufficient, and the distortion levels of Embodied AI should be divided by the JND of RVS itself, as we have done in the Embodied-1Q2 database. Comparing various IQA methods longitudinally. FR is superior to NR in most cases, with TOPIQ maintaining the leading performance in most cases, with an SRCC of about 0.75, which still needs improvement compared to human-oriented IQA. It is worth mentioning that the main parameters of some methods have been frozen based on HVS, such as LPIPS, DISTS, and CLIPIQA. Thus, although after training, they are even worse than the sero baseline. This further reflects the gap between HVS and RVS, implying the significance of proposing the Embodied IQA task.
5.3 CROSS DATABASE VALIDATION
To further analyze the performance of RVS RVi-oriented IQA on HVS/MVS, we conducted cross- validation using Embodied-IQA VIA Decision score for training, VLM Cognition score, and two of
8

---

Published as a conference paper at ICLR 2026
Mechod
5BCC: PLCCT
LIVE
SR0C
110
FLOC
SBCCT
VLM
PLOCT
Real Woold Dark 305---mmm Score
0336
0.724
1198
6.074
63449
113713
0.009
18300
03316
0.5430
0.4327
0.3540
Ruoal wortd Dock
10010
0,7355
0.755
0.450
03317
0.7115
0.0788
Wary
00432
01833
B1029
30544
0.1977
DBUNS
013900
0.954
0.008
0.631
10m1--i
0.8353
11307
0.139
CAR26
0.007
VLM Infernnce (SRCC: 0.48)
VLAIniirmme (80CC-0.77)
(a) Cross Validation
(b) VLM & Real-world
(c) VLA & Real-world
Figure 8: Cross database validation Cognition and human-oriented score, and the correlation
between V and Real-world. denotes distortions with greater Real-world impact.
Task: T'iist top on the hoard.
Task: Place the weoden blck outsido
Tack: Pick up the red block.
Task: Pull out book fimmt the shelf.
Dic: Others 7
Resalt 0.0cm
Dis: Neise ?
Reslt: 0.5cm
Da: Compresioo
E Result: 3.2am
X
Dis Sputial B8
Result: 42.8cm
A
Figure 9: Positive/Negative and deduct the Euclidean distance for failed examples. Triggering interruption will be scored as O.
of real-world experiment. The score of successful examples is 100,
the most commonly human-oriented databases, LIVE2008 Moorthy & Bovik ( and TID2013 Ponomarenko et al. (2015) for validation, employing the same parameter settings Section 5.1 According to the results in Table g (a), IQA methods fine-tuned on Embodied Al data lose certain human-oriented evaluation capabilities, where the SRCC is even lower than 0.4 in the LIVE database. Fortunately, the IQA model trained with VIA annotations can also predict VLM scores, and the SRCC of ÁHIQ can reach 0.7, revealing the internal connection between Cognition and Decision.
5.4 REAL-WORLD EXPERIMENT
Since Embodied AI is ultimately applied in the Real-world, compare Execution with Cog- nition/Decision to link External and Internal Reality, thereby proving the reliability of the 5m+ annotations in the Embodied-IQA IQQ database. Specifically, output and executed the 10 tasks in Section 3.2 on 30 types of distorted images. Note that among the five difficulty levels in Perception, we only executed the simplest one ensure that the execution result of the reference image is correct. Thus, we ensured that the reason for execution failure came from the added distortion, not the image itself. Figure shows examples of successful execution. results deviating from the ground truth, and emergency stops triggered by collisions with the table. We calculate the average Execution score under 30 distortion types and compare it with the Cognition and Decision scores, as shown in Figure (b)(c), where findings are summarized as follows:
selected VLA that support multi-step
Cognition Execution:
The SRCC of VLM results with the real world less than 0.5. This corroborates the necessity of using VLA subjects in the Embodied IQA task beyond VLM.
Decision VS Execution: The SRCC of VLA results with the real world exceeds 0.6, indicating that Decision represent Execution to extent. However, this correlation is still not high enough, proving that certain real-world experiments are still indispensable for Embodied AI development. Perception VS Cognition& Decision: Existing quality metries have initially demonstrated the ability to handle Embodied IQA tasks, but there is still a gap compared to the traditional human-oriented paradigm. More advanced metrics should be developed in the upcoming Embodied AI era.
CONCLUSION
In this paper, we extend the application IQA from a traditional human-oriented paradigm to Em- bodied AI. To study which distortions have negative impact on Embodied AI, we built a Perception- Seenition-Decisiine Pntisiion-xxecttoonn pipeline based on Mertonian Law and established a database for Embodied subjective preferences. Experiments show that more advanced IQA methods needed to identify quality degradation for Embodied AI. We sincerely hope this Embodied IQA task can promote the application of Robotic Intelligence under complex distortions in the Real-world.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported in part by the National Natural Science Foundation of China under Grants 625B2118, 18, 62501387, 62572317, 62501337, 62225112, in part by the Singapore Ministry of Ed- ucation under Grant ZDSYS20020552111446000 Intelligence-Nationn Science and Technology Major Project (20252D0124104) collaboration with Shanghai Artificial Intelligence Laboratory.
and in part by the New Generation Artificial
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, and et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024a. URL
https::/aaxiiei org. abs 2404 210.224441111199
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and al. Phi-3 technical report: A highly capable language model locally on your phone, 2024b. URL
abs/2404. 14219111 9
Lorenzo Agnolucci, Leonardo Galteri, and Marco Bertini. Quality-aware image-text alignment for opinion-unaware image quality assessment, 2025. BBlL:httss///rrav..rr////////
11176
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, and et al. Qwen2.5-vl technical report, 2025. URL htt.h:///s///ttt or org/abs/ 2502. 13923
Tina Balke and Nigel Gilbert. How do agents make decisions? survey, Journal of Artificial Societies and Social Simulation, 17(4):13, 2014.
Johannes Ballé and et Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, (5(2):339-553, 2020.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. up. A vision-language- action flow model for general robot control, 2024a. URL RRLLhttps:// arxiv. org/ ab// / 24164v1
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. To A vision-language-- action flow model for general robot control, 2024b. URL httph::t/////. httpp/s/////i..../// org baalggg/bb//44/ 24v16v33
Nicholas L. Cassimatis, Gregory Trafton, Magdalena D Bugajska, and Alan C Schultz. Integrating cognition, perception and action through mental simulation in robots. Robotics and Autonomous Systems, 49(1-2):13-23, 2004.
Chaofeng Chen, Jiadi Mo, Jingwen Hou, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin. Topiq: top-down approach from semanties to distortions for image quality assessment. IEEE Transactions Image Processing, 33:2404-2418, 2024a. doi: 10.1 00.1109/7P.2022433786666
Zhe Chen, Jiannan WU, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo. Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- Souuul-inggustic tasks, 2024b. URL https: a
axxxxxxii rxiv..v...... wtwg/aas//// 22tp://..///22.1111111 14388
10

---

Published as a conference paper at ICLR 2026
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuchui Wang. Qingyun Li, Yimin Ren. Zixuan Chen, Jiapeng Luo, and et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL
2411 05271
Embodiment Collaboration, Abby O'Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acor Pooley, Agrim Gupta, Ajay Mandlckar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Herzog, Alex Irpan, and et al. Open x-embodiment: Robotic learning datasets and rt-x models, 2025. URL http:///w/ww.oo...////////////.. 08864
Amaury Depierre, Emmanuel Dellandréa, and Liming Chen. Jacquard: large scale dataset for robotic grasp detection, 2018. BRLIhttps://aaax http:////xxvvvv../.aaaee org.org////a... 1803. 11469
Keyan Ding, Kede Ma, Shiqi Wang, and Eero Simoncelli. Image quality assessment: Unifying structure and texture similarity. IEEE transactions pattern analysis and machine intelligence, 44(5):2567-2581, 2020.
Xiaoyi Dong, Pan Zhang, and et al. Internlm-xcompossr2:: Mastering free-form text-image tion and comprehension in vision-language large model, 2024. URL /tLp://o////// abs / 2201. 16420 16420
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-244, 2022.
Zhangwei Gao, Zhe Chen, and et al. Mini-internvl: A flexiblettransfor pocket multimodal model with Sparameters and 90URL https: http//axxvvvvro/..ea. wttpgba/////....../////// 16261
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills, 2023, htthts/
04659
Jinjin Gu, Haoming Cai, and et al. Ntire 2022 challenge on perceptual image quality assessment. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 951-967, 2022.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation, 2018. URL https
//arxiv.orq/aba
10293
Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1733-1740, 2014.
Justin Kerr, Huang Huang. Albert Wilcox, Ryan Hoque, Jeffrey Ichnowski, Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features, 2023. URL httpt:tt:://ettt http:::::///xxvv.xx//a/ abs 220992299...1 13042
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, and Others. Droid: A large-scale in-the-wild robot manip- ulation dataset. In Proceedings of Robotics: Science and Systems, 2024. URL https org/ Roboties: Science and Systems, R:SS Conference date:
15-07-2024 Through 19-07-2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel. Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source law---ann-aattiinnn action model, 2024a. URL https: arxiv. org/abs/ 2406. 69246v1]
11

---

Published as a conference paper at ICLR 2026
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-atttoo model, 2024b. URL 2406. 09246v2
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model, 2024c. URL.
2406. 09246v3
Shanshan Lao, Yuan Gong, Shuwei Shi, Sidi Yang, Tianhe Wu, Jiahao Wang, Weihao Xia, and Yujiu Yang. Attentions help cnns see better: Attention-based hybrid image quality assessment network, 2022.
Nathan F Lepora and Giovanni Pezzulo. Embodied choice: how action influences perceptual decision making. PLoS computational biology, 11(4):e1004110 2015.
Chunyi Li, Zicheng Zhang. Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. Agiqa-3k: An open database for ai-generated image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Chunyi Li, Tengchuan Kou, Yixuan Gao, Yuqin Cao, Wei Sun, Zicheng Zhang, Yingjie Zhou, Zhichao Zhang. Weixia Zhang, Haoning Wu, Xiaohong Liu, Xiongkuo Min, and Guangtao Zhai. Aigiqa-20k: A large database for ai-generated image quality assessment, 2024a.
Chunyi Li, Xiele Wu, Haoning Wu, Donghui Feng, Zicheng Zhang, Guo Lu, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. Cme-bench: Towards new paradigm of visual signal compression, 2024b.
Chunyi Li, Yuan Tian, Xiaoyue Ling, Zicheng Zhang, Haodong Duan, Haoning Wu, Ziheng Jia, Xiaohong Liu, Xiongkuo Min, Guo Lu, Weisi Lin, and Guangtao Zhai. Image quality assessment: From human to machine preference, 2025a. https::s::/ //axxvv org/ abtpa/as//2210000 10078
Chunyi Li, Jianbo Zhang, Zicheng Zhang, Haoning Wu, Yuan Tian, Wei Sun, Guo Lu, Xiongkuo Min, Xiaohong Liu, Weisi Lin, Xiao-Ping Zhang, and Guangtao Zhai. R-bench: Are your large multimodal model robust to real- world corruptions? IEEE Journal of Selected Topics in Signal Processing, pp. 1-16, 2025b. doi: 10.1109//S 100000/SSSP.222.55558888
Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu. Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, and Baining Guo. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation, 2024c. URL https://
arxiv. org. abs 2411. 19650
Hanhe Lin, Vlad Hosu, and Dietmar Saupe. Kadid-10k: A large-scale artificially distorted iqa database. In 2019 Tenth International Conference on Quality of Multimedia Experience (QoMEX). pp. 1-3. IEEF, 2019.
Xiaohong Liu, Xiongkuo Min, Guangtao Zhai, Chunyi Li, Tengchuan Kou, et al. Ntire 2024 quality assessment of ai ai-generated content challenge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 6337, June 2024a.
Yang Liu, Weixing Chen, Yongjie Bai, Xiaodan Liang, Guanbin Li, Wen Gao, and Liang Lin. Aligning cyber space with physical world: A comprehensive survey on embodied ai, 2024b.
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model, 2024. URL https
//arxiv. www./ba..a........... /20/22405.... 20797
Microsoft, :, AbdeIrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, and et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-oo -ffturreff---ras,,, 2025.
htRlhttss:// htattpss///axiiie www.vvorrggaaee wttpo//g///./..oo/////////// 01743
12

---

Published as a conference paper at ICLR 2026
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality as- sessment in the spatial domain. IEEE Transactions on image processing, 21(12))4695 4708, 2012.
Anush Krishna Moorthy and Alan Conrad Bovik. Blind image quality assessment: From natural scene statistics to perceptual quality. IEEE transactions on Image Processing, 20(12):335--330 2011.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision- saioo-language-action models, 2025.
http::::s///xxxi..r..oo// abt/ 2501. 09747
Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al. Image database tid2013: Peculiarities, results and perspectives. Signal processing: Image communication, 30:57-77, 2015.
Jeff Rickel and W Lewis Johnson. Animated agents for procedural training in virtual reality: Perception, cognition, and motor control. Applied artificial intelligence, 13(4-5):343-35) 382,, 1999.
Erick Rosete- Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. In Proceedings of the 6th Conference on Robot Learning (CoRL), 2022.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied research. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9338-9346, 2019, doi: 10. 10.1109/CCC..219900994
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière and Others. Gemma technical report, 2025. URL /tttt::tt//aatteete /ttpiii//arxvv.oo org/ wadss/2555 19786.
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy, 2024. URL.nt.pe:://cx...orr///////
12213
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: http///axvvv/aa////a//e.//a.avv/ org/ abs 2302..01 13971
Georgios Tziafas, XU Yucheng, Arushi Goel, Mohammadreza Kasaei, Zhibin Li, and Hamidreza Kasaei. Language-guide guided robot grasping: Clip- Clip-based referring grasp synthesis in clutter. In 7th Annual Conference Robot Learning, 2023.
Ramakrishna Vedantam, C Lawrence Zimick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566 4566-4575, 2015.
Fei-Yue Yue Wang. A big-data perspective on ai: Newton, merton, and analytics intelligence. IEEE Intelligent Systems, 27(5):2-4, 2012.
Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Exploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2555 2565- 2023.
Zhou Wang. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600-612, 2004.
13

---

Published as a conference paper at ICLR 2026
Haoning Wu, Zicheng Zhang, and et al. Q-align: Teaching LMMs for visual scoring via discrete text-defined levels. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 54015-54029, PMLR, 2024.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, and et al. Qwen2 technical report, 2024. URL
https://axxiv>
Qinghao Ye, Haiyang Xu, and et al. mplug-owl: Modularization empowers large language models with multimodality, 2024. URL SRLL:tttpes/aarxxx.orrr/a//////0
Michal Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning, 2025. URL https::/axxiv...tttom /bss/
Jianbo Zhang, Chunyi Li, Liang Yuan, Guoquan Zheng, Jie Hao, and Guangtao Zhai. Embodied image quality assessment for robotic intelligence, 2024a. URL
2412 18774
Pan Zhang, Xiaoyi Dong, and et al. internlm-xcomposerr A versatile large vision language model supporting long-contextual input and output, 2024b. URL httpps://axxxv..... wwg.://ii.orraoooooo..o wwwp/rggg/bbbaaaeeee 03320
Qi Zhang, Shanshe Wang, Xinfeng Zhang, Chuanmin Jía, Zhao Wang, Siwei Ma, and Wen Gao. Perceptual video coding for machines via satisfied machine ratio modeling, 2024c.
Qi Zhang, Shanshe Wang, Xinfeng Zhang, Siwei Ma, Jingshan Pan, and Wen Gao. Predicting satisfied user and machine ratio for compressed images: unified approach, 2024d.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 386-595,10 2018a.
Weixia Zhang, Kede Ma, Jia Yan, Dexiang Deng, and Zhou Wang. Blind image quality assessment using deep bilinear convolutional neural network. IEEE Transactions on Circuits and Systems for Video Technology, 30(1):36-4) 2018b.
Zicheng Zhang, Haoning Wu, Ziheng Jia, Weisi Lin, and Guangtao Zhai. Teaching Imms for image quality scoring and interpreting, 2025. URL httthhtts///aaatt ort. 2503. 09197
Heliang Zheng, Huan Yang. Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Learning conditional knowledge distillation for degraded-referrnce image quality assessment. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pp. 10242-10251, October 2021.
Jinguo Zhu, Weiyun Wang, and et al. Internv13; Exploring advanced training and test-time recipes for open-source multimodal models, 2025, (ttptt / arxiv. org/ abs 2504 10479
14

---

Published as a conference paper at ICLR 2026
A STATEMENT
[LLM Usage]: LLM is applied to aid and polish writing, including grammar, rhetoric, and proper citation format. No scientific conclusions, experimental designs, or technical contributions are generated by LLM.
[Ethics statement]: This article covers Embodied Al's four steps: perception, cognition, decision, and execution. Most experiments focus on machine intelligence and humans are involved only in the perception annotation phases. We obtained consent from each participant before data collection and ensured that the experimental procedures adhered to the Declaration of Hclsinki. For Embodied AI execution, we implemented an emergency stop mechanism for the robotic arm to ensure compliance with the Three AI Laws (do no harm to humans/the environmen//isell).
[Reproducibilityy] We will release all quality-related issue for non-commercial use, including the quality score from each VLMs/VLAs, the fidelity of positiion/rotatioon/statee, and the overall score.
B LIMITATION BROADER IMPACT
[Limitation 1]: As the first Embodied IQA work, we simplify the Perception to vision and Execution to robotic arms. Since the visual signals processed by humans account for 80% of the total signals, while upper limb movements account for 50% of all movements. Considering the consistency between humans and machines and the current limitations of Embodied AI, our simplification is reasonable. This will not affect the current main applications of Embodied AI, such as industrial assembly and home services. After the future vision-tactile fusion (Perception), quadruped robot dog (Execution), and other task scenarios are improved, we will further update the quality assessment data.
The scale of our Real-world data is relatively small compared to Cognition and Decision. In the VLM and VLA steps, we already have the largest amount of data (millions) compared to previous IQA work. This is because the high cost of real machine data requires lot of manpower in the site layout and verification stages. Although 1,500 annotation samples are not enough as quality labels, they are sufficient verify the Sim2Real consistency of Cognition-Execuut Exxcutiin and Decision- Execution. With the further development of Embodied AI, we believe an automated Real-world pipeline will be developed, from which we will expand the scale of Execution labels.
[Limitation 2]:
[Broader Impact] (Positive): IQA can expand the application scenarios of Embodied AI, extending from the in-lab environment to distortions in the Real-world. We collect the subjective preferences of Embodied AI, thus objectively judge the 'utility' of images before executing specific tasks. In this way, distorted images such as jitter and blur can be effectively filtered. Such quality indicators can be used for all visual applications for Embodied AI, such as video coding, super-resolution, defogging, denoising, etc. Considering that the amount visual signals consumed by machines has exceeded humans since 2023, visual quality indicators for Embodied AI can fill this research gap.
[Broader Impact] (Negative): The general use of visual quality indicators in Embodied Al may affect traditional human-oriented tasks. Considering that humans, VLM (Cognition), and VLA (Decision) have different preferences, only evaluating the preferences VLM and VLA will inevitably lead to scores that not relevant to humans. Therefore, in future international protocols, it is recommended to integrate the three IQA paradigms for humans, VLM, and VLA together, and select appropriate quality indicators based the user end.
C ROBOTICS SETTINGS
This section provides a detailed derivation of the forward and inverse kinematics for the Universal Robots (UR5), 6-DoF collaborative robot. The Dentvit-Hartenberg (D-H) convention is used to establish the kinematic model.
In the experiments part, the initial pose is obtained through forward kinematics by recording the initial rotational angles of the six joints and calculating the and-effector's pose relative to the base coordinate frame. The incremental pose output by the VLA is then multiplied with the initial pose to derive the by-seb poses. The robotic arm's 's actual motion is resolved via inverse kinematics, which computes the required rotational angles for each joint motor achieve the target configuration.
15

---

Published as a conference paper at ICLR 2026
Table 3: Parameter settings of Robotic arm UR5 D-H. The specific action depends 6 frames.
Joint Frame
0;-1
(rad)
0:-1
(m)
d; (m)
0, (rad)
0
d1
09
2
x/2
0
0
02
3
0
0
05
4
0
d4
0
5
x/2
0
ds
05
6
-2
0
dg
0g
The D-H parameters define the geometry of the robot manipulator by establishing coordinate frame {i} attached each link The transformation from frame {i 1} to frame {i}, denoted A is described by four parameters associated with link 1 and joint
0g Joint Angle the rotation about the 2;-1 axis, from 2;-1 x4 For a revolute joint, 0, is the joint variable.
dę: Link Offset the distance along the 2 axis from the origin of frame {i }} to the intersection of the - axis with the x; axis. For prismatic joint, de is the joint variable.
a;-1 Link Length the distance along the x; axis (which is the common normal between - and zi) from the intersection of zi-1 and x; axis to the origin of frame {i}.
0:-1: Link Twist the angle about the x; axis, from 2;-1 to 24.
The UR5 D-H parameters used this paper shown in Table Where 02 ag are physical link lengths associated with links 2 and 3 respectively (used as a:- parameters in the table for joints 3 and 4), and di, da, ds, de are link offsets. The 0f are the joint variables.
Typical UR5 parameter values (example, signs depend on coordinate frame choices): d1 0.0891 59 m, @2 = 0.42500 m (often negative some tables: 02.4200), a3 = 0.39225 m (often negative: 0..99225), d4 0.10915 d; 0.09465 de 0.0823 m.
D FORWARD KINEMATICS
The standard D-H transformation matrix 4!-1 from frame {i - 1} frame {i} defined product of four basic transformations:
A R.
(a4-1),
(I)
cos(O (.)
- sin(O,) cos(ar-1) cos(O,) cos(aa-1)
sin(6,) sin((a;11) cos(O:) sin(a;-1) 0: sin (0,)
0;-1 cos(O:)
sin (0:)
A
sin(a
cos 0
di
where Tr(-) and R(-) denote the trajectory and rotation matrix projected on certain axis. For simplicity, the following symbols will be defined in the subsequent sections: cos(6,). sin (0.). Gjj = cos(O, + 0,), 8ij sin(0; - 02). Using the D-H parameters from Table the individual transformation matrix A for robotic manipulators are:
(os(O1) (in
0))
0
cos(O
0
AQ
(3)
1
d1
0
cos( O2)
0
sin(O))
0
sin(A2) Oz)
0
cos 0z)
0
4
(4)
0
O
0
0
0
0
1
16

---

Published as a conference paper at ICLR 2026
Manipulation
Manipulation Annotation
MAGE:
TASKS (Progressive Difficultyy:
IMAGE:
TASKS (Progresxive Difficully::
pen
slck bracked
Teak: 4. Nowe tas rade the palicon board
GUIDELINES:
GUIDELINES:
Took 5 Place tlock insato mesia
independert.
Type here
indepaendent.
Type here to modily task:
D lifuuity should iomerae
strely
sivelyy
Deficuity thould incresse
Plaze pen the cas.
Specity styect narnes cleant.
Speatly odject names dlearly.
Report
Move:
Frevious
Next so
Phenexux
Noot
(a) Annotation
(b) Verification
Figure 10: Human annotation and verification Interface. Subjects will cyclically define five tasks with different contents and increasing difficulty, and submit them to robotic experts for verification.
cos(O2)) sin(Os)
sin sin(Os) (0s)
a2 cos(
A3
cos (0z)
0
(0n)
(5)
0
(4) sin (s) 0
sin (0.)
O 0g cos((1))
cos (0)
ag sin (04)
A3
(6)
1
d4
0
cos(Os) sin
0
sin (05)
0
cos (0s)
0
Ag
ds
cos (os)
O
sin
sin
(n)
0
cos
0a)
A
(8)
0
de
The total transformation matrix To from the base frame {0} to the end-effector frame {6} is:
fh
Sz
a
Px
Te - A
R2
p
nl
Sg
Py
0 x
1
12
2
02
Pz
0
1
where R. = [n, s, a| refers to the rotation matrix part of TÍ where n [Tzz, ny. nz]
[Sz, Sy, sz] and a = jaz, ag TT are column vectors representing the æ. y, axes of frame {6} expressed in frame {0), respectively, p [Px. Ps]] represents the translation vector part of T namely the position of the origin of frame (6} expressed in frame {0}.
E INVERSE KINEMATICS
The objective of Inverse Kinematics (IK) is to determine the set of joint angles (01...., that achieve desired end-effector pose TØ. The URS possesses a spherical wrist (axes joints 4, 5, and 6 intersect at common point, the wrist center), which allows for a decoupled analytical solution. First, the position of the wrist center determined, which allows solving for the first three joints. Then, the orientation of the end-effector is used solve for the remaining three wrist joints.
17

---

Published as a conference paper at ICLR 2026
A8
1.
2.95
6.14
100
Somanti:
Recall
124
100
0.9
Proditen
116
8/6
134
Prrchtent
Roall
Semeetlc
A0
Real Setrehation
Finst
Thrd Muchanical Duily Elecrvniic
Tool
Otber
Home
Indinnial Sotlwore
Lab
Otbens
Sim2Real
Perspective
Main Object
Bockground
Figure 11: Correlation between the general Cognition score and the 3 dimensions from VLMMs, and the score distribution in Sim2Real, First/Third perspectives, Main object, and Background sub sub-categories.
E.I CALCULATION THE WRIST CENTER POINT
The wrist center point (WCP), Pwe. is typically defined as the origin of frame (5). Its position can be found by translating from the end-effector origin, ph. backwards along the approach vector (the ze-axis expressed in frame {0)) by a distance de:
Pr.
az
Ps dg
6 Pwe
Po
R
0 d6
Pv
de
0gy
=
Py dgd
(10)
P=
Ozz
Pz=
dga
where Pwe (Iwer Dwes Z
T means position vector of the wrist center point frame {0}.
E.2 SOLVING FOR BASE JOINTS
The y-coordinate of Pwe when expressed in frame {1}, denoted pwe.y can be shown to be d4 + ds for this specific D-H parameterization (where Puc is the origin of frame {5)). We have pweyy -wue sin (01) + Uwe cos(01) d4 ds. This equation can be solved for 01:
atan2 rwee) atan2(dd dg, 01 s v (d4 dz)²),
(11)
where 01 = =1 denotes two possible solutions for 0j. Function atan2(...) transforms two Cartesian coordinates polar. If the term under the square root is negative, the target Puce is unreachable.
E.3 SOLVING FOR ELBOW JOINTS
With 0, known, transform Pwe into frame {1}. Let K2 and K₂ be coordinates of Pwe relevant for the planar geometry of links and 3: K₂ Twe cos(O;) + Ywe yee sin(01) K₂ Zwe dj. From the geometry of the first three links (considering making 21 horizontal in the new projected plane 02 O): K2 a2 cos(f2) + as cos(O, fs)) K2 -22 (f2) ag sin(f, + 0z). Squaring and adding these two equations yields: K? + K?
π a/2 which introduces rotation
a² + a² + 20203 cOs( This allows solving for 03:
cos(fs)= K3 + K a az
(12)
20z0s
where d3 +1 corresponds to up/down configurations. From sin(Os) 03 03v1 cos((g)) have:
0g = atan2(sin(8s)) cos((s)))
(13)
To solve for 02 rearrange the equations for K2 and K3: : Let kj = ag tag cos (f)) and h; = ag sin
Then K₂ kj cos(02) li s0n( and K; -i sin (0g) la cos(O2). Solving this system for sin (02) and cos(02): s0n( = (kiK2 + Kg / (k? k cos(O2) = (ki K₂ (2)))/ k3)
(Note: k2 + 13 K2 K
00 = atan2(-(A)) K3 + 1o Ka), ln K2 k2
(14)
Alternatively, more robust form is often 0₂ atan2( - Ke, Kw) atan2(k;,,s)) kT).
E.4 SOLVING FOR WRIST JOINTS
Once 01,02, 0 are known, the rotation matrix RB from the base to frame {3} can be computed:
RB (4 44 A3 The rotation matrix from frame (3) frame (6) is then Rg (R$) R2 Let
18

---

Published as a conference paper at ICLR 2026
Table 4: Using 15 advanced IQA metries to evaluate the Cognition score from VLMs, including
zero-shot, FR, and NR metrics. [Keys: Best/Second best in group; Baseline; Lower than baseline.]
Dimenniin
Prossion
Recall
Sameniic
Find Paregective
Thind Panpasivv:
Meerrc
SRCUT
KRCCT
PLOC1
SRCUT
KRCEY
PLOCE
SBCCT
KRCO
PLACT
SROCTT
KRRD
PEACY
SROCT
KROCT
PLCCT
0.9452
0.3399
0.1661
0.3186
0.3168
0.539
0.3287
0.2215
0.3530
041441
0.2811
0.4193
0.963
0.2477
0:4140
S51M
Wing T03351
11550
0.4155
0.5561
0,5554
0.3671
0.524
0,579
0.4157
0.5521
0.6344
0.4355
0.5305
0.5649
01.2244
0,5438
Zem
2012
0,3527
0.3342
0,2412
0.3596
0.2422
0.3375
03323
0.3751
0.3554
0-Alig
0.2865
0.5067
0.6721
0,4798
0.6278
0.7040
0.5058
0.6687
0.6622
0.4767
0.6214
0.7321
0.5349
0,6970
12 U.A
M015
0.007
0.318
0.001
014544
0,3063
013675
0,473
0.3000
11 1950
0.4647
0.3188
0.4263
0.5052
113421
014153
AHHO
2014
0.591
0.330
0.7501
0.796
0.5076
0.515
0.3035
0.0050
0.3015
006342
0.8222
CKDN
Clmmriii ci
0.2461
0.5460
0.7444
0.7387
0.580
0.332
0.7516
0.5936
0.7536
0.5387
0.7547
0.7816
0.5806
0.7810
FR
DISTS
Xin 21
0.2017
0,3126
0.7052
116887
0,5010
0.865
6.7000
0.5155
01 1130
0.7907
0.5424
0.7714
0.7515
0.515
0.7590
LPIPS
047885
0.6165
0.4010
0.975
0.6714
0.412
0.797
0.6895
06445
0.8209
0.6241
0.8194
0.810
0.6170
08126
0.835
0.6363
0.8289
0.7997
0.9967
0.7932
0.8821
0.8422
S..ng Wang of
0:3111
0.3101
0.1185
11:111
0.3011
01:141
6.1667
0.2156
015557
0.174K
11111
0.132
0.984
0.2630
0:1521
0.464
04818
633812
0.480
0.3358
04719
0.3247
0.533
0.5431
NR
DBCNNN
0.987
0.3921
0.9555
0.138
0.366
0.5185
0,9896
0.3815
01 0.5453
0.5541
0.3578
0.514
0.6613
(0,8699
016888
QuelCCi
Aggnnlei re MD5
0.7416
0.5425
0.7389
0.7399
0,5399
0.7385
0.7534
0.5514
0.749
0,6960
0,5032
0.6913
0.7864
0.5832
0.7711
TOP1O-NE
0.7897
0.8512
0.5449
Dinsenin
MiM Pistortie
Distottion
Rosldword
Simulation
Gro4p
Merrcc
SRCCT
KRCCT
FLCC1
SBCCT
KBCC:
PLCC
SROCT
KRCC-
PLCCT
SR0CCT
KRCC-
FLCC1
SROCT
KROCC
PLCCT
PSNR
0.2350
0,1595
0.3725
0.1122
0,0751
0.1527
0
0.00006
0.0763
0.439
0.3056
0.4575
0.3615
0.2466
0,3945
Wayz 2094
0.3138
6,002
0,1080
0.2102
0.1375
0.5319
0.4507
0.3916
0,6645
0.6449
Zar
Bese
0.1525
0.1205
0.1017
0,1503
G:110
0.9857
0.3005
0.1502
0.1334
04144
0,2907
0.321
0.0626
0.0423
Q-Aligh
0,5092
0,3572
0.4793
0.449
0,367
0.4800
0,4166
0.2775
0U5947
0.7764
0,5739
0.7475
0.6642
0.4645
0,6319
0.295
0.701
0.226
01290
0,148
0.6111
0.399
0.0894
0.4709
AHIO
kilbs
0.5911
0.191
0.5942
0,5538
0:3855
0.664
0.558
0.3668
0.006
8.5355
0.033
0:314
6.8136
0.6117
0.804
CK3N/ Ztenz e1 101
0.547
0.3855
0.5072
1.559
0.3473
03537
6.5167
0.3535
0.5387
0,7480
0.539
0.7467
0.7706
115051
0,7127
FR
DISTS
0.3376
0.300
000000000000
0399
0.3258
0.2215
0.7006
0,5655
0.556
6.075
0.267
LPIPS 2ires al
0.6635
0:329
0.4255
0.3526
63361
03300
0:366
0.3148
0.823
0:1128
05100
0.6691
0.699
0.4325
0.600
107
0.6755
0.4875
0.6661
111533
0,4385
9.5798
0.017
106135
0,5392
03313
0.6422
CLIPIOA
2043
01330
30000
01010
0,0974
0,0645
0.1245
03334
0.355
04178
0.5342
CNNTQA King
0.2677
0.1797
0.326
0.189
011001
0.3003
0.332
0:5347
03732
0.4419
0,930
0.365
0.3380
NR
DBCNN Chase ct a
2016
0.227
0.200
0.3455
0.421
6.139
02333
0.2215
0.1480
0.2456
0.5085
0.4743
0.55
0.6255
0.4306
0.5990
QualiCl
2005
0.3258
8-7647
0.5049
0.7330
8.7691
0.7428
TOP10-NR WWWOO-RRUhree Chrr mal
0.433
0,4588
6,6295
0,4248
0.599
9.578X
6,0114
0.0151
0.6001
0.6454
0,032
6.8122
RAND
0:6311
Dimenin Verric
Do-levd-1
Dislovd-3
Dio-look33
Dis-lock4
Dis-losd.5
Grrup
SRCPT
KRCO
P1,001
SRCT
SR000
KRCO
P1,001
SR001
KRD00
SROCT
KROC1
0,164
0.1512
0.2065
0,4258
0.4473
S51M Wanz
0.5065
0.3520
0.4753
0,488
03389
0.547
0.560]
0.5878
0.5309
0:5646
0.3522
0.5450
0.9077
0.4997
0.6846
7cm
Brisgi
Minii <Tw
2012
0.1471
0.971
0.1276
02155)
0,7665
(33150
6.1106
0.3005
012760
0.1113
0
0.305
010000
0.4550
0.5100
0.4101
166788
0,487
0.657
0,044
0.639
0.517V
0.7316
0.5390
09600
8.7155
04775
0-Aliu
Zinni
0.4598
0.3166
0.4055
0,5730
64005
0,003
0.5487
0.3758
0,6988
0.5992
0,3634
0.5110
04769
0,3238
04140
1,704
0.526
1090
6.932
0.54
CKIN
06433
0.5044
017418
0.527
07186
0.7104
0.5085
0,0440
0.5050
0,7045
DISTY
08559
0.4199
0.6411
06885
06448
0.673
0.4856
0.0680
0:6875
0.4945
0.2040
0.754
0.550
0.7521
LPIPS LPIPS Zhare 4t
0.674
0.400
046255
116743
0.085
00405
6.005
0.4774
115895
0:5185
0.455
0.567
0.644E
0.2544
04348
TOPIC
0.0010
0.7944
0.9141
8.5161
0.9651
0.8054
0.116
CLIMOA
Mane
0,259
6:1727
0:3215
0.3158
63201
0.3206
0:2608
0.1758
0.2473
63331
0.2252
0.348
8.4283
0.2021
0.122
UNNIQA Karg al
0.384
0.355
0.3572
0.398
0:2675
0.3737
6.5143
0,3497
1 18952
0.5114
0.5412
0.5002
3.6176
013311
DRCNN
A018
06571
0.645
QudlChi
Nos
0.2072
0.5115
6.7188
0.7357
0.5366
0.7187
8.7077
0.508
0.0866
0.7693
0,5537
0.7493
8.7352
0.5438
0.7379
TOPIQ
NR. Lach
302444
0.7788
0.5815
0,5044
0.7974
0,6014
0.7993
0.796
0
0.7911
0.8071
0.6035
0.7967
8.7965
0.5952
0.8827
Rg = [r'g]. The matrix Rg can also be expressed as the product of rotations for joints 4, 4,5,6 using
their D-H parameters: R = (, (a3) (R.)). (8s R.
(a.) (B.) For the URS D-H
parameters in Table 3 R = R. (0, R. (0s R. (T/ ( R. (g) R (- The symbolic product is:
/C4C5CB - S455C6
C455 + s.0s 8485
C1 - $45558
RG- =
8405C6 + cas5Cs
cars
840588 C48588
(15)
86
-66
By comparing elements of the numerically computed Rg- = [r]]] with this symbolic form:
1. r's2 must be O. If the computed ((R)) Rg is significantly non-zero, it indicates no solution for this wrist structure or a modeling error.
2. From T'1 se and r's = -o:
fg - atana - )s))
(16)
This provides unique solution for 0g in (-T, x). Another solution is 6g + (if sa, Co are flipped), but usually we seek solutions within joint limits.
3. From r/2 = C4S5 + S10s = $145 and r'22 = 8185 C4Cs = C445
0₂ + 0g- = atan2(ri -1(2)-
(17)
Consider the movement of these two elements whole, we have thas 04 + Oz.
19

---

Published as a conference paper at ICLR 2026
Carpatonn fitver
Lone Maxr
Moxion Mir
Color
Color grartionton
I8SV satuntoo
Lab srussiioo
JPIK compression
JPEG conppessiion
WGBP congrresico
White moiee
Coloe moise
Impuke arive
Maloplictive milse
Caumdzz dunvinc
CNN dunvinc
Maximum brighin
Mirimars
Meaa brialies
Clock
Bluck ouchasge
Block
Intepetation
Block ergent
Grysaae (Madizaaio
Shappzzes change
change
Figure 12: Low-level feature distribution of MPD, normalized and visualized in 30 corruption subsets. Different colors denote Luminance, Contrast, Chrominance, Blur, and Spatial Information.
4. find f and 04 separately: 85 carí2 + S = Sass + sss +s((-cos)) Coas) sin - 04)
sin 05. s4r12 C4r22 = is ca(=Cons) ) = cos 04s 0,) = cos 05. A common method solve for 0g (wrist roll) for many spherical wrists involves:
r r22
0g o5 arccos
(18)
co(c4
54 24
According to 03 we have 04 = 045 0s. Thus all rotation angles can be retrieved.
Typically. URS has unique inverse kinematics solutions (o; =1,03 =1, o5 +1 for the choice of s5). Singularities (e.g., 0) lead to infinite solutions where 04 and 0s are coupled.
E.5 HANDLING SINGULARRIIES,
Shoulder Singularity: Occurs if x + v.. (d4 + ds)² 0. The wrist center lies on the axis (for d4 d3 = 0) or cylinder around 2p. 01 is not uniquely defined.
Elbow Singularity: Occurs if K2 + K2 a² až = +222a3, meaning cos(Og) = +1 (arm fully extended folded). sin (0)) number of 02 might exist if Pwe is on 21.
so solution becomes simpler but an infinite
Wrist Singularity: Occurs if 55 = (i.e., 0g = 0 or π). Axes 24 and z5 align. this case, only the sum difference (0g + 0g) can be determined according to Section D.4. One angle can be chosen arbitrarily, and the other is then fixed.
F SUBJECTIVE PERCEPTION TASK DEFINITION
Before VLM and VLA inference, organized five Ph. D. candidates a panel to define five downstream tasks for each image as shown in Figure 10 To avoid bias from a single subject, each
sample is sent to five subjects in random order to design specific tasks based the image. The 1,230
samples to be annotated come from seven Robotic database Khazatsky et al. (2024): Kalashnikov et al.
(2018) Kerr al. (2023). Depierre al. (018): Tziafas al. (203) Gu al. (2023) Roseee-Bees Beas
20

---

Published as a conference paper at ICLR 2026
Distortion:
Type: Block interpolotaion
Type: Block interpolotaion
Strength level: 2
Strength level: 5
VLA: CogACT
VLA: OpenVLA
Decision score: 4,93
Decision score: 4.52
Task: Pick up the green cup from the table.
Task: Place the red block on top of the gray block.
Referencc result:
Distorted result:
Reference results
Distorted results
Position: [-0,02, 0.00,-0.03]
Position: [-0.02, 0.00,-0044
Position: [-0.02.00011-0..8
[-0.02, 0.00,
Rotation: 0.00,-0.11, 0.01]
Rotation: 0.10,-0.01, 0.01]
Rotation: 0.22, 0.05,-0002
Rotation:
0.11, 0.00, 0.02
State: 1.0
State:
1.0
State:
Stale:
Distortion:
Distortion:
Type: Block lost Strength
Type: Block exchange
Strength level:
VLA: PiO
VLA: Octo
Decision score:
Decision score: 2.47
up the black object from the
Task: pen the right side of
result:
Reference result:
Distarted result:
Position: 0.06,-0.09, 0.11]
Position: [-0.08, 0.15
Position: [ 0.01. 0.01, 0.05]
Position: [-0.04, 0.01,-0.05]
Rotation: 0.51, 0,35, 0.15]
Rotation: 19,0000.10, 0.07]
Rotation: [ 0.24, 0.06, 0.11]
Rotation: [-0.07.-0211.0.00]
State:
State:
State: 1.0
State: 0.0
Positive and negative cases. Slight distortion may significantly affect the inference result of Embodied AI, while severe distortion may not. Emphasizing the significance of Embodied IQA.
Figure 13:
al. 2022). and are filtered according to the settings in the main text, to retain only high-quality imgges. Subjects can see the previous tasks, and the new tasks they design need to be more difficult than them and test different abilities (for example, if an object has been pushed, try not to grab again). Images with display errors, NSFW, or offensive content will be removed. After each image has five task labels, professional robotics engineer will adjust the specific samples. Based on operational experience, the difficulty of the five tasks will be re-ranked, and unreasonable tasks will be modified.
G COGNITION IQA EXPERIMENT
Due to space limitations in the main text, we mainly discuss the Decision step (specific to Embodied AD), and the Cognition step (common to general machines) is listed in this section. First, the Cognition score given by VLM is shown in Figure IT Compared with the three dimensions of Decision in Figure the correlation between the Cognition dimensions is higher, and the distribution difference between different categories of data is smaller. This fully demonstrates the difference between the reasoning mechanisms of VLM and VLA. and proves the rationality of separating these two steps.
Therefore, in addition to Decision, we also conducted IQA experiments on Cognition, following the training/testing settings in the main text. Table presents the performance of advanced quality metrics on Cogmition, compared with Decision in Table current IQA metrics has better prediction results on Cognition. Since the current IQA method more related to VLM than VLA, the quality indicators that general machines already have are initially available. but Embodied AI cannot be effectively evaluated. is worth mentioning that the zero-shot baseline method on Cognition can occasionally even achieve an SRCC of more than 0.7, surpassing a number of fine-tuned methods; while the bascline on Decision is significantly weaker. This is exactly why we separated the Robot Visual System from the Machine Visual System and used the Mortonian system to model the Intelligent System in four steps. In short, we hope that the Embodied IQA database can promote more complete quality indicators, whether applied for VLM or VLA subjects in Embodied tasks.
H LOW-LEVEL. ATTRIBUTE DISTRIBUTION
Figure 12 shows the distribution of low-level features of all instances of Embodied IQA. After overall regularization, 30 types corruption are grouped and displayed. The features considered include Luminance, Contrast, Chrominance, Blur, and Spatial Information. There significant differences
21

---

Published as a conference paper at ICLR 2026
Gaussian lilter
Lens hlur
Motion blur
Color diffusion
Color shift
Color quanitzztion
HSV saturation
Lah saturation
JP2K compression
JPBG compression
WEBP compression
White moise
Color noise
Impulse noise
Multiplicttive noise
Gaussian denoise
CNN denoise
Maximum brighten
Minimum darken
Mean brighten
Mean darken
Clock littering
Block exchange
Block Jost
Block interpolation
Block repent
Resolution limit
Grayseale quantization
Sharpeness change
Contrast change
Figure 14: Visualization of distortion types. Strength from left (Level 1) to right (Level 5).
in these low-level attributes for different corruptions. For example, in the first three blurry cases, the blur curve is left-biased and becomes right-biased after sharpening. In general, similar corruption categories will lead similar results (such five noise-related and four block-related). Two of the denoises have the sharpest distributions; Color quantization. Grayscale quantization, Sharpness change, and Contrast change are the most irregular. These findings deserve further exploration.
22

---

Published as a conference paper at ICLR 2026
I CASES STUDY
shows four typical examples from the Embodied IQA database (center-cropped for visu- alization), including the VLA inference results for reference/distorted image pairs under different distortions. The results in the upper left and lower right corners are as expected, the more severe the distortion, the lower the score. However, for the distortion level in the upper righ. since does not affect any objects on the desktop, the 'gray block' as the target of the task is not affected, so the subjective score as high as 4.52; on the contrary, although corner is only 1, the 'lost macro block' happens to be the target object, so the VLA Position and Rotation are greatly changed with a score only 2.98. Figure T4 shows distortion types different strength levels from I to 5. In the previous human-orientes scenario, the visual quality of different corruptions similar at the same strength. However, from the example above, the preference of Embodied AI depends the task, which significantly differs from traditional IQA paradigm. We hope that our database can further inspire better quality metrics for Embodied AI.
Figure 13
distortion level the lower left
J DISCLAIMER
The main purpose of this study is to apply IQA to Embodied Al to promote its Real-world application, rather than praise criticize any VLM, VLA, IQA model. We evaluate image samples rather than models. Lower scores do not mean that the performance downstream VLMM/LLA is poor, but distortion has greater impact it; similarly, lower correlation coefficients do not mean defects in the IQA method, but rather indicate the huge difference between Embodied and traditional IQA. Considering the scale of the database, will open it in several stages for con-commercial use, and sincerely hope that future nobotic-oriented IQA metrics can drive the development of Embodied AL.
23


Reviews:
[{"review_id": "s7yNZPFPgR", "reviewer": "Reviewer_1xDt", "summary": "In this paper, the author proposes Image Quality Assessment for Embodied AI, where the quality of perception is not judged by human preferences, but a robot's ability to perform the task after perceiving distorted images. It introduces a database with 36900 reference distorted images.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of IQA for machines vs humans is established in prior works. This paper reframes the problem for robotics, the stage of how degradation of images affects robot task execution, not just visual recognition.\n- The paper was interesting to read, with extensive experiments and detailed analysis.\n- The Embodied-IQA database is very large, containing 36,900 image pairs and over 5 million fine-grained annotations, having good scale. It is also annotated along three unique axes, reflecting different visual systems and their real-world performance.", "weaknesses": "- The pipeline assumes vision is the dominant modality, neglecting that in true Embodied AI, perception often must fuse audio, tactile, and temperature cues. Is it possible that temperature might play a role on how bright the image might be?\n- The writing is often verbose and repeats technical claims across sections, particularly regarding pipeline design and dataset composition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gEnrRpxyaN", "reviewer": "Reviewer_USAJ", "summary": "This paper proposes IQA for Embodied AI, arguing that existing IQA methods of HVS and MVS are insufficient to judge image usability for robots, because robot performance depends not only on perception/cognition (VLM) but also on decision-making/planning (VLA) and real execution.To this end, the paper proposes: 1. A robot intelligence perception-cognition-decision-execution workflow based on Merton's systems perspective; 2. Embodied-IQA, a novel and diverse dataset. Furthermore, experiments demonstrate that embodied artificial intelligence requires more sophisticated IQA metrics.", "questions": "1. How often do BLEU/ROUGE/CIDEr disagree with human judgments of task equivalence for VLM outputs? Have you validated a small subset with human raters using task-specific correctness criteria?\n\n2. How were the 15 VLAs harmonized regarding action parameterization (units, reference frames, delta vs absolute commands)? Could inconsistencies reduce inter-model SRCC?\n\n3. Are there results from more powerful VLA models (exceeding 8B parameters) or closed-source baselines to understand the impact of parameter scale on distortion robustness?\n\n4. How are the difficulty levels for the five tasks defined? Are they objective?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The paper presents a clear motivation and significant innovation. It defines the image quality assessment (IQA) problem in embodied intelligence as “image usability for robots,” transcending traditional frameworks based on human or machine vision systems (HVS/MVS). It innovatively models the robot's “decision-making” and “execution” phases explicitly.\n2. The research exhibits high quality. First, its constructed dataset is exemplary in scale (36k+ images, 5m+ labels), breadth (30 distortion types, multiple viewpoints and scenarios), and depth (incorporating 1,500 valuable real-world robotic execution trials). Second, the experimental design is rigorous, employing a comprehensive multi-dimensional scoring system and robust benchmarking protocols. This includes 10 data-split replicates and multiple correlation metrics, ensuring reliable results.\n3. The study maintains rigorous structure, with appendices candidly discussing its limitations.\n4. This work holds significant value. It not only provides a novel benchmark dataset but also successfully establishes critical empirical links between image quality and robots' actual decision-making and execution capabilities.", "weaknesses": "This study exhibits several critical weaknesses.\n1. Methods that evaluate differences based on metrics may inherit and amplify inherent biases and errors within the model and its assessment indicators. Specifically, using metrics like BLEU/ROUGE to measure cognitive comprehension is highly sensitive to phrasing and redundancy, potentially failing to accurately reflect task equivalence. Adopting structured patterns (e.g., action-parameter tuples) or task success classifiers may be more robust alternatives.\n2. The scoring design for the decision phase is also inadequate. The study simply averages standardized position, rotation, and gripper state metrics, failing to account for the weighting of critical failure modes across different tasks—for example, a minor rotation error may cause collision in some tasks, while gripper state importance varies by task. The absence of task-specific weighting may undermine label authenticity.\n3. The real-world robot execution experiments are limited in scale and diversity. The 1,500 trials are confined to the simplest tasks per image, failing to comprehensively cover all distortion types, intensities, objects, and viewpoints. Thus, the generalizability of these findings to more complex multi-step tasks remains unclear.\n4. The low correlation (SRCC) among different VLA models may stem not only from “genuine preference differences” but also from their heterogeneous action spaces or variations in pre- and post-processing workflows.\n5. The study leaves a notable gap: while successfully demonstrating the inadequacy of existing IQA baseline models for new tasks, it fails to propose an “embodied IQA model” or architecture specifically designed for robotic usability (e.g., a model capable of predicting multidimensional scores based on task/text conditions).\n6. Regarding clarity, the paper exhibits a few ambiguous sentences. The figures are information-dense, and critical conclusions are occasionally buried in supplementary materials, hindering the communication of core findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rsnzmWizQl", "reviewer": "Reviewer_YAEp", "summary": "This paper proposes a completely new research task: Image Quality Assessment for Embodied AI . The authors' core argument is that traditional human-centric or general machine-centric IQA metrics are not applicable to Embodied AI, because RVS involves a more complex Perception-Cognition-Decision-Execution pipeline. To address this issue, the paper makes three main contributions:\n\n1. Based on the \"Mertonian system\" theory, it constructs the aforementioned four-step pipeline, theoretically demonstrating the fundamental difference between RVS and MVS/HVS.\n\n2. It constructs a large-scale benchmark dataset named Embodied-IQA. It contains over 30k reference/distorted image pairs and 5 million fine-grained annotations. These annotations are collected separately from VLMs, VLAs, and real-world robots (representing Execution). \n\n3. The paper benchmarks 15 mainstream IQA methods on the Embodied-IQA dataset. The results show that existing methods perform poorly on this task (e.g., SRCC is much lower than on HVS tasks), which strongly demonstrates the necessity of developing new Embodied-IQA metrics.", "questions": "1. You define the \"Decision\" score as a simple average of position, rotation, and state errors. How did you account for the unequal importance of these three dimensions in different tasks? For example, in a \"turn faucet\" task, shouldn't rotation error be weighted much more heavily than position error?\n2. You found that the correlation between different VLA models is extremely low, indicating they have vastly different \"preferences\" for distortion. Your final \"Decision\" score appears to be an average of these 15 VLAs. Does this imply that an IQA metric performing well on the \"average VLA\" might perform poorly for a specific VLA?\n3. Regarding the correlation between VLA and Execution: You found the SRCC between VLA (Decision) and real-world (Execution) to be ~0.67. In your opinion, is this correlation high enough for the community to confidently use VLA annotations as a proxy for real execution in the future? Or does this gap suggest that VLA annotations themselves still have limitations?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper identifies and clearly defines a completely new, critical, and timely research problem: assessing image quality for Embodied AI. Its theoretical framework based on the \"Mertonian system\" to differentiate RVS, MVS, and HVS is highly novel and persuasive, laying a solid theoretical foundation for this new field.\n\n2. The paper's main contribution—the Embodied-IQA dataset—is an extremely valuable resource. Its scale and granularity are unprecedented in the IQA field. This dataset will likely drive much research in the coming years.\n\n3. The paper not only provides the dataset but also conducts a comprehensive benchmark of 15 existing IQA methods. The analysis convincingly demonstrates the failure of existing methods on this new task, strongly validating the paper's motivation and the necessity of its contribution.", "weaknesses": "1. The paper defines the VLA \"Decision\" score as a simple average of errors in three dimensions: Position, Rotation, and State. This metric seems overly simplistic. In real robotics tasks, the importance of these three dimensions can be highly imbalanced (e.g., a minor rotation error could cause catastrophic failure, while a larger position error might still be acceptable).\n2. As a benchmark paper, its primary duty is to define the problem and provide data, which it does exceptionally well. However, after proving that 15 existing methods fail, the paper does not attempt to train even a simple new baseline model (e.g., a simple CNN trained from scratch on Embodied-IQA) to set an initial performance bar for future researchers to challenge.\n3. While 1,500 real-world robot experiments (for execution validation) is already impressive and costly, it is still a small scale compared to the 5 million VLM/VLA annotations. Although the paper acknowledges this, it leaves in question whether VLA annotations can serve as a high-fidelity proxy for real execution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fFXNJAnVoU", "reviewer": "Reviewer_BC9i", "summary": "The paper proposes that image quality assessment methods focus on the ability to predict human ratings for distorted images, which lacks meaning when applied to perceptual quality for robots. The authors propose the topic IQA for Embodied AI, a topic focusing on a robots ability to perform downstream tasks on distorted images. The Embodied-IQA database, a set of reference-distorted image pairs, with corresponding decisions or task executions, where labels are provided by a set of vision language models, with some evaluation against existing datasets and traditional IQMs.", "questions": "- How similar is the topic of Embodied AI to image distortions effect downstream tasks in topics such as image classification, object recognition, ect? I imagine some of the distortions are quite correlated with issues in these topics as discussed in papers such as [1] \n- Do authors expect the above to be captured in their dataset?\n\n[1] Zhou, Yiren, Sibo Song, and Ngai-Man Cheung. \"On classification of distorted images with deep convolutional neural networks.\" 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The topic is very relevant for ICLR and the contributions are significant.\n- The dataset is impressive, in the amount of distortions applied to images, the cognition/decision/execution labels, and the breadth of situations covered.\n- The dataset analysis is thorough, providing correlations between different aspects of the datasets and evaluations using traditional full-reference and non-reference IQAs.\n- I am not aware of any datasets covering the end-to-end pipeline of using distorted images in the field of robots", "weaknesses": "- A discussion on how likely the distortions are to show up in the robotic systems is missing. For example, are changes in chroma often observed in the sensors used in robotic systems? I imagine this is similar to the ISP pipeline where out of focus (blur), sensor noise (noise) and compression are more likely than others.\n- It would be interesting to note where certain distortions effect the pipeline - for example spatial distortions are more likely for the robot to miss and object, chroma distortions are more likely to misinterpret which object to act upon ect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)