STORM: Benchmarking Visual Rating of MLLMs with a Comprehensive Ordinal Regression Dataset

## Abstract

Abstract Visual rating is an essential capability of artificial intelligence (AI) for multi-dimensional quantification of visual content, primarily applied in ordinal regression (OR) tasks such as image quality assessment, facial age estimation, and medical image grading. However, current multi-modal large language models (MLLMs) under-perform in such visual rating ability while also suffering the lack of relevant datasets and benchmarks. In this work, we collect and present STORM, a data collection and benchmark for S timulating T rustworthy O rdinal R egression Ability of M LLMs for universal visual rating. STORM encompasses 14 ordinal regression datasets across five common visual rating domains, comprising 655K image-level pairs and the corresponding carefully curated VQAs. Importantly, we also propose a coarse-to-fine processing pipeline that dynamically considers label candidates and provides interpretable thoughts, providing MLLMs with a general and trustworthy ordinal thinking paradigm. This benchmark aims to evaluate the all-in-one and zero-shot performance of MLLMs in scenarios requiring understanding of the essential common ordinal relationships of rating labels. Extensive experiments demonstrate the effectiveness of our framework and shed light on better fine-tuning strategies. The STORM dataset, benchmark, and pre-trained models are available on the following webpage to support further research in this area. Datasets and codes are released on the project page: https://storm-bench.github.io/ .

## 1 Introduction

With the success of large language models (LLMs) like GPT-4 achiam2023gpt and Gemini team2023gemini, researchers have been enhancing these models by incorporating visual understanding capabilities. This enthusiasm has led to the emergence of multi-modal large language models (MLLMs), such as LLaVA liu2023improvedllava; liu2023llava, GPT-4o openai2023gpt4, and Qwen-VL bai2023qwen; bai2023qwen-vl, which show demonstrated viability in various VQA scenarios.

However, the potential of MLLMs in visual rating capabilities has not yet been fully explored despite their critical importance in various visual analysis applications, such as image quality/aesthetic assessment, face age estimation, medical image grading, etc.
The hindrance in the development of stronger MLLMs for visual rating is attributed to the following three challenges.
(1) The complexity of task labels, that is, inconsistent numbers and levels of labels of different visual rating tasks. Existing methods only train MLLMs with the same number and definition of level labels wu2023q, which could yield unsatisfied performance when users propose a different rating protocol. (2) The hallucination phenomenon of MLLMs for numeric labels. MLLMs typically use contrastive learning for pre-training and may pay more attention to high-level semantics than to precise numerical features wu2023qinstruct. Furthermore, the subjective inconsistency of human annotation can also lead the model to learn noise.
(3) Poor zero-shot performance. Existing MLLMs can only be trained on specific tasks, which can incur severe limitations when the model is tested on out-of-domain datasets and may lack general rating practicality. Unfortunately, there is still a lack of relevant datasets and benchmarks to train and evaluate trustworthy MLLMs with strong and general visual rating capabilities.

Figure: Figure 1: An overview of our STORM benchmark. STORM consists of four key components: 1) Broad domain data (14 datasets across 5 domains); 2) diverse level annotations; 3) coarse-to-fine CoT; 4) all-in-one visual rating framework.
Refer to caption: x1.png

To address the above challenges, we look into the inherent logic of common visual rating tasks and observe a shared nature of these tasks: They are all ordinal regression (OR) problems whose labels are ordinal. Therefore, we introduce STORM, a data collection and benchmark for Stimulating Trustworthy Ordinal Regression Ability of MLLMs for universal visual rating. First, STORM includes a comprehensive OR data collection comprising 655K question-answer pairs across 5 popular visual rating tasks. Through joint training based on this comprehensive OR dataset, an MLLM is initially endowed with a fundamental ability to tackle most visual rating tasks. Furthermore, we develop a lite version dataset of about 250K samples for faster model training. Second, for all question-answer pairs, the answer not only adopts a mixed description of text and numbers to significantly mitigate the model’s numeric hallucination but also includes an extra intermediate prediction step, which is designed to instruct the MLLM with a logical, coarse-to-fine Chain-of-Thought (CoT) process to understand a general way of thinking about OR problems, enabling MLLMs to attain a better zero-shot performance on out-of-domain visual rating tasks. Third, we provide the corresponding visual rating benchmark and pre-trained models for reproducibility, aiming to foster further research in visual rating for MLLMs.

In summary, the key highlights of our STORM benchmark include:

- •
Broad Domain Data: STORM contains high-quality data including 14 popular ordinal regression datasets comprising 655k data items across five distinct domains.
- •
Diverse Level Annotations. STORM includes basic numeric labels, suitable for fundamental settings of all visual rating questions. It also incorporates diverse text labels to strengthen the specific semantic understanding for different visual rating tasks and the capabilities of MLLMs in explainable rating predictions.
- •
Coarse-to-fine CoT: We introduce a coarse-to-fine Chain-of-Thought (CoT) pipeline for MLLMs, enabling them to learn a universal paradigm of ordinal regression and providing intermediate interpretable thoughts.
- •
All-in-one Evaluation Framework: We propose a comprehensive evaluation framework to benchmark the all-in-one visual rating capability of MLLMs on both in-domain and out-of-domain datasets. To the best of our knowledge, STORM is the first benchmarking and dataset building effort to test the universal visual rating abilities of MLLMs.

## 2 Related Works

Multi-modal LLMs.
The success of large language models (LLMs) in various language applications has paved the way for the development of multi-modal large language models (MLLMs), which integrate vision and language modalities. Initially, MLLMs were treated as dispatch schedulers to connect vision expert models, such as VisualChatGPT wu2023visual, HuggingGPT shen2024hugginggpt, and MM-REACT yang2023mm, in order to extend language models to other tasks and modalities. More recently, MLLMs have focused on aligning these modalities through extensive training on image-caption pairs or image-question conversations. Notable methods like LLaVA liu2023llava train a projector that maps image tokens to aligned representations of pre-trained LLMs. Other approaches, such as BLIP-2 li2022blip; li2023blip, adopt a query Transformer (Q-Former) to learn image embeddings using learnable queries after obtaining image features. MoVA zong2024mova designs an adaptive router to fuse task-specific vision experts with a coarse-to-fine mechanism. In terms of training strategy, recent works liu2023llava; bai2023qwen; wang2023cogvlm; zhu2023minigpt; chen2022pali; luo2024cheap commonly employed a 2-stage framework; the first stage involves pre-training on image-caption pairs, while the second stage focuses on alignment by using question-answering triplets. MLLMs have also been extended to various applications, including fine-grained localization wang2024visionllm; lai2023lisa such as object detection zhang2023gpt4roi, video understanding zhang2023video; li2023llama; chen2023longlora, and image generation koh2024generating; qian2023strait.

LMMs for Visual Rating.
Some recent studies have discussed the possibilities of adopting Large Multi-modality Models (LMMs) for visual rating/scoring. For example, Q-Bench wu2023qbench proposed a binary softmax strategy, enabling LMMs to predict quantifiable quality scores by extracting softmax pooling results on logits of two frequent tokens (good/poor). Based on this strategy, Q-Instruct wu2023qinstruct noticed that fine-tuning with question-answering text on related low-level queries can also improve visual rating abilities of LMMs. Another work, Q-Align wu2023q,
systematically emulated human rating and post-processing in visual rating. However, these methods are still limited in that they focus only on certain types of tasks, such as image/video quality assessment and image aesthetic assessment. In comparison, our STORM framework introduces a comprehensive ordinal regression data collection that contains many other tasks across different domains for visual rating in addition to image/video quality assessment and image aesthetic assessment, such as facial age estimation, medical image grading, and image historical estimation.

**Table 1: A summary of the ordinal regression datasets in STORM for visual rating. STORM spans 5 domains and includes various source datasets, offering a broad representation of visual data styles.**
| Domain | Source Dataset | Full Version Size | Lite Version Size | Category |
| --- | --- | --- | --- | --- |
| Image Quality Assessment (IQA) | SPAQ fang2020perceptual | 11,125 | 11,125 | 5 levels |
| ChallengeDB ghadiyaram2015massive | 1,169 | 1,169 | 5 levels |  |
| KonIQ hosu2020koniq | 10,073 | 10,073 | 5 levels |  |
| Image Aesthetics Assessment (IAA) | Aesthetics Dataset dosovitskiy2020image | 13,706 | 13,706 | 5 levels |
| TAD66K he2022rethinking | 66,327 | 27,132 | 5 levels |  |
| AVA gu2018ava | 255,508 | 51,104 | 5 levels |  |
| Facial Age Estimation (FAE) | Adience levi2015age | 17,321 | 17,321 | 8 groups |
| CACD chen2015face | 163,446 | 32,690 | 14-62 years |  |
| Morph ricanek2006morph | 50,015 | 20,006 | 16-77 years |  |
| UTK zhang2017age | 24,106 | 24,106 | 1-116 years |  |
| Medical Disease Grading (MDG) | Eyepacs dugas7diabetic | 35,127 | 35,127 | 5 grades |
| DeepDR liu2022deepdrid | 2,000 | 2,000 | 5 grades |  |
| APTOS karthik2019aptos | 3,662 | 3,662 | 5 grades |  |
| Historical Date Estimation (HDE) | HCI palermo2012dating | 1,325 | 1,325 | 5 decades |

Ordinal Regression.
Given an input image, ordinal regression (OR) in computer vision aims to map the image to a rank or a continuous value. Many popular methods rothe2018deep; geng2013facial; frank2001simple; li2006ordinal; chen2017using adopted a classification framework. Some recent studies lim2019order; liu2019probabilistic; lee2020deep; li2021learning proposed ordinal distribution constraints to exploit the ordinal nature of regression. Adding prior order knowledge to loss calculation, several methods fu2018deep; diaz2019soft created soft labels artificially by changing the distances between categories. A few advanced methods liu2017deep; liu2018constrained; li2021learning; shin2022moving sorted tuples that are formed by two or three instances with ordinal categories to learn the rank information. Ord2Seq wang2023ord2seq proposed to transform OR tasks to sequence prediction and solve ordinal regression using autoregressive models. Recent works like OrdinalCLIP li2022ordinalclip, L2RCLIP wang2023learning, and NumCLIP du2024teach used CLIP CLIP for OR tasks, focusing on designing a text encoder to map numeric labels to a continuous space for improved image-text alignment. Although these deep learning (DL) methods are general and effective, they need to train separate models for different OR tasks. In comparison, our proposed STORM is a general framework built on MLLMs and aims to construct an all-in-one visual rating model.

## 3 Ordinal Regression Data Collection for Visual Rating

### 3.1 Overview

Currently, a general visual rating framework is still lacking. Existing domain-specific models are predominantly optimized for fixed-format labeling schemes, thus exhibiting poor generalization capability when encountering diverse label configurations or cross-domain scenarios. To address this gap, we curate a comprehensive OR data collection that spans five distinct domains and includes 14 various source datasets, as shown in Tab. [1](https://arxiv.org/html/2506.01738v1#S2.T1). For more details on distribution, see Appendix C.

To ensure a robust foundation for different visual rating tasks, our STORM data collection deliberately integrates a diverse selection of data including image quality assessment (IQA), image aesthetic assessment (IAA), facial age estimation (FAE), medical disease grading (MDG), and image historical date estimation (HDE). These data domains are intentionally chosen to cultivate a comprehensive skill set across varied visual rating tasks. 1) IQA and IAA are the most widely demanded scenarios, which enhance MLLMs’ capability in subjective qualitative judgment of quality or superiority gradation. 2) Facial age estimation aids in cognitive capabilities of objective estimation tasks with continuous and wide-ranging labels, particularly in scenarios requiring precise numerical regression like depth estimation. 3) Medical disease grading fosters the ability of severity assessment in complex scenarios, which are essential for medical and anomaly detection applications. 4) Historical date estimation develops temporal awareness of MLLMs, which is vital for time-related estimation tasks.

Figure: Figure 2: A data example with the original VQA compared with our coarse-to-fine CoT VQA.
Refer to caption: x2.png

### 3.2 Data Generation Details

To gather and build a comprehensive and diverse visual rating data collection, we select 14 source ordinal regression datasets across five distinct domains. As these datasets provide only images and digital labels, they are designed with a standardized VQA paradigm by reusing their images and modifying the annotations into a textual form to enable MLLMs to undergo joint training for heterogeneous tasks of diverse domains. Specifically,
each data sample originally consists of a simple question and a corresponding numeric answer. However, this paradigm can lead to numerical hallucination. Hence, we add extra domain-driven prompts and coarse-to-fine CoT to mitigate this issue. An example with the original VQA and our proposed coarse-to-fine CoT process is shown in Fig. [2](https://arxiv.org/html/2506.01738v1#S3.F2). Meanwhile, we adopt the form of text + numbers for the labels to enhance semantic understanding. In the following sections, we elaborate on the VQA details employed for each domain-specific visual rating dataset.

Image Quality Assessment (IQA). We choose three IQA datasets to create data in this domain: SPAQ fang2020perceptual, KonIQ hosu2020koniq, and ChallengeDB ghadiyaram2015massive.
The three datasets focus on the impact of distortions and
other quality issues in images on human perception. The fact that these datasets provide only mean opinion score (MOS) values makes it difficult to teach LMMs to predict scores aligned with human. Thus, we simulate the process of training human annotators. We convert the MOS values to five text-defined rating levels itu: {‘bad’ (0), ‘poor’ (1), ‘fair’ (2), ‘good’
(3), ‘excellent’ (4)}. For coarse intermediate thoughts, the candidates are: {‘below fair’ (0-1), ‘fair’ (2), ‘above fair’ (3-4)}.

Image Aesthetics Assessment (IAA). For this domain, we use Aesthetics Dataset dosovitskiy2020image, TAD66K he2022rethinking, and AVA gu2018ava, which are widely-used datasets for image aesthetics assessment. The IAA datasets provide images and the corresponding multi-rater scores. Similarly to IQA, we compute the MOS values of all raters and convert the MOS values to five text-defined rating levels: {‘unacceptable’ (0), ‘flawed’ (1), ‘average’ (2), ‘professional’
(3), ‘excellent’ (4)}. For coarse intermediate thoughts, the candidates are: {‘below average’ (0-1), ‘average’ (2), ‘above average’ (3-4)}.

Facial Age Estimation (FAE). We use Adience levi2015age, CACD chen2015face, Morph ricanek2006morph, and UTK zhang2017age as datasets for facial age estimation tasks. In the Adience dataset, each image encompasses a category label that is annotated
in 8 groups: 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and over 60 years old. Thus, we assign the most suitable text for each group according to the age range: {‘infants’ (group0, 0-2 years old), ‘preschoolers’ (group1, 4-6 years old), ‘preteens’ (group2, 8-13 years old), ‘teens’ (group3, 15-20 years old), ‘adult’ (group4, 25-32 years old), ‘midlifers’ (group5, 38-43 years old), ‘matures’ (group6, 48-53 years old), ‘seniors’ (group7, over 60 years old)}. In the other three datasets, each label is a specific age number. Hence, we set the answer as an age with a corresponding text, such as ‘adult’ (30 years old). The coarse intermediate thoughts for all these datasets are the same and the candidates are: {‘baby’ (group0-1, 0-7 years old), ‘teenagers’ (group2-3, 8-24 years old), ‘adult’ (group4-5, 25-47 years old), ‘elder’ (group6-7, over 48 years old)}.

Medical Disease Grading (MDG).
We select a series of Diabetic Retinopathy (DR) grading datasets, including Eyepacs dugas7diabetic, DeepDR liu2022deepdrid, and APTOS karthik2019aptos, as datasets for medical disease grading. In these datasets, images are annotated in five levels of diabetic retinopathy from grade 1 to 5. We also add text-defined rating labels for all the levels: {‘normal’ (1), ‘mild’ (2), ‘moderate’ (3), ‘severe’
(4), ‘extreme’ (5)}. For coarse intermediate thoughts, the candidates are: {‘normal’ (1), ‘early’ (2-3), ‘late’ (4-5)}.

Historical Date Estimation (HDE). We select the HCI dataset palermo2012dating as the dataset for historical date estimation. This dataset aims to estimate the decades of historical color photos. There are five decades, from 1930s to 1970s, annotated as 1 to 5. We also add text-defined rating labels for each phase: {‘early’ (phase1, 1930s), ‘early-mid’ (phase2, 1940s), ‘middle’ (phase3, 1950s), ‘mid-late’
(phase4, 1960s), ‘late’ (phase5, 1970s)}. For coarse intermediate thoughts, the candidates are: {‘before middle’ (phase1-2, 1930s-1940s), ‘middle’ (phase3, 1950s), ‘after middle’ (phase4-5, 1960s-1970s)}.

Figure: Figure 3: The model pipeline of STORM. It first extracts visual tokens from an input image and determines the task objective. Then, pre-generated coarse and fine candidate categories, including numeric and text labels (generated by GPT and manually adjusted, stored in the dataset), are used to formulate instructional prompts that guide the model to perform coarse-to-fine CoT, thus predicting the corresponding labels for the image progressively.
Refer to caption: x3.png

**Table 2: Accuracy performance of the visual rating benchmark (higher is better). “Tra.” indicates the datasets used for fine-tuning. “Zero” denotes the model without fine-tuning. “Lite” denotes that the model is fine-tuned on the lite vision of datasets. “Full” denotes that the model is fine-tuned on the full datasets. Datasets highlighted in gray indicate that their training splits are not used in our model’s fine-tuning phase.**
|  | IQA | FAE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLLM | Tra. | SPAQ | ChallengeDB | KonIQ | Adience | CACD | Morph | UTK |
| LLaVA-1.5-7B liu2023improvedllava | Zero | 0.243 | 0.296 | 0.396 | 0.452 | - | - | - |
| Lite | 0.259 | 0.249 | 0.263 | 0.333 | - | - | - |  |
| Qwen2.5-VL-3B bai2023qwen-vl | Zero | 0.512 | 0.472 | 0.493 | 0.444 | - | - | - |
| Lite | 0.600 | 0.446 | 0.561 | 0.480 | - | - | - |  |
| STORM-3B | Lite | 0.583 | 0.468 | 0.582 | 0.534 | - | - | - |
| Full | 0.585 | 0.466 | 0.568 | 0.551 | - | - | - |  |

## 4 Enhancing MLLMs with All-in-one Visual Rating Capabilities

Model Pipeline. Fig. [3](https://arxiv.org/html/2506.01738v1#S3.F3) presents an overview of the pipeline for our model. The pipeline mainly consists of three parts: Vision Encoder, Text Candidate Generation, and Coarse-to-fine CoT. The Vision Encoder processes visual input and encodes it into a series of visual tokens. Text Candidate Generation provides both coarse and fine text definitions for each numeric label, which will act as prompts and form a new question to instruct the LLM to provide an intermediate coarse thought for the coarse-to-fine CoT. Coarse-to-fine CoT generates a finer final answer based on the coarse thought. Our STORM chooses Qwen2.5-VL-3B bai2023qwen-vl as the LLM backbone. For more details, see Appendix B.

Text Candidate Generation. For different domain tasks, we first use GPT to generate a text definition for each numeric label. Then, manual adjustments are applied to make the text definition more realistic and compatible with human rating practices. After this, both the intermediate coarse thought and final answer have a text label and a numeric label. This offers several advantages: 1) Reducing digital hallucination. Since MLLMs are pre-trained using CLIP to align images and text rather than numbers, they are prone to numerical hallucination. By supplementing numeric labels with text definitions, MLLMs can learn more ordinal semantic relationships and reduce digital hallucination. 2) Differentiating task specificity. Since different tasks may share identical label ranges (e.g., 1-5 ratings) while having distinct task natures, the models could confuse label distributions across tasks. Leveraging textual definitions allows the models to capture task-specific specificity, while numeric labels can preserve the ordinal commonality essential for diverse rating tasks.

Coarse-to-fine CoT. To train an MLLM with our newly generated data, we add a CoT prompt (“Please provide the coarse category that can help you answer the question better. The candidates is: ”), followed by text along with numeric category candidates for the question. The MLLM is instructed to perform the following three steps:

- 1.
Make a coarse rating thought with the candidates (e.g. Child (0-10 years old), Teenager (11-20 years old), Youth (21-30 years old), Adult (31-44 years old), Middle (45-60 years old), Elder (60+ years old)).
- 2.
Based on the coarse rating thought, proceed to make a final answer.
- 3.
Check that the coarse rating thought and answer are consistent. (This strategy is designed to alleviate the problem of inconsistency between coarse intermediate thought and final answer, that is, to prevent the coarse intermediate thought from not including the final answer.)

This methodology aims to serve three key objectives. 1) First and foremost, through a coarse-to-fine progressive analysis process, it allows to learn universal solutions for ordinal regression to endow the models with the all-in-one visual rating capability. This hierarchical approach is universally applicable to ordinal regression problems, as only their ordered categorical nature permits merging of adjacent categories for candidate reduction, enabling recursive hierarchical decomposition of the problem. 2) It transforms a multi-class rating problem into several smaller rating tasks with fewer candidate categories, therefore reducing the classification complexity through progressive candidate pruning. 3) Coarse labels are equivalent to merged neighboring categories, partially helping alleviate the class imbalance issues through category aggregation.
For more VQA illustrations on other datasets, see Appendix E.

## 5 Experiments

**Table 3: MAE performance of the visual rating benchmark (lower is better). The other settings are the same as in Tab. [2](https://arxiv.org/html/2506.01738v1#S3.T2).**
|  | IQA | FAE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLLM | Tra. | SPAQ | ChallengeDB | KonIQ | Adience | CACD | Morph | UTK |
| LLaVA-1.5-7B liu2023improvedllava | Zero | 1.294 | 1.155 | 0.852 | 0.859 | 11.439 | 9.251 | 11.763 |
| Lite | 0.983 | 1.017 | 0.919 | 0.990 | 8.776 | 6.691 | 9.934 |  |
| Qwen2.5-VL-3B bai2023qwen-vl | Zero | 0.534 | 0.592 | 0.547 | 0.734 | 9.746 | 5.470 | 6.534 |
| Lite | 0.423 | 0.605 | 0.469 | 0.715 | 7.541 | 7.589 | 6.433 |  |
| STORM-3B | Lite | 0.442 | 0.597 | 0.431 | 0.636 | 8.202 | 5.975 | 5.879 |
| Full | 0.441 | 0.602 | 0.460 | 0.602 | 8.014 | 5.886 | 5.689 |  |

### 5.1 Visual Rating Benchmark

Our visual rating benchmark primarily focuses on scenarios where the MLLMs need to concentrate on ordinal understanding based on the visual input. Our experiments utilize 14 source datasets, and when an official training/evaluation split exists, we adopt it. In the cases where such a split does not exist, we randomly divide the dataset. Additionally, we incorporate the test splits of HCI, CACD, UTK, Aesthetic, KonIQ, and APTOS to evaluate the model’s zero-shot visual rating capabilities.

### 5.2 Performance Evaluation

We comprehensively evaluate STORM across various visual rating tasks to thoroughly assess our model’s ordinal understanding ability. Tab. [2](https://arxiv.org/html/2506.01738v1#S3.T2) and Tab. [3](https://arxiv.org/html/2506.01738v1#S5.T3) report the accuracy and MAE performances of LLaVA-1.5, Qwen2.5-VL, and our STORM benchmark. We test LLaVA-1.5 and Qwen2.5-VL only on the lite version of our datasets, and test our STORM on both the lite and full versions. By comparing the results of different models without fine-tuning and with fine-tuning on the lite version, we observe that after fine-tuning, the model significantly improves performances across all the datasets. This demonstrates the effectiveness of our data collection and benchmark.
Notably, our STORM shows remarkable improvement in zero-shot performance when the training splits for the corresponding datasets are not utilized for model training. For instance, on the Aes. dosovitskiy2020image datasets, our model achieves nearly 2.5$\times$ performance compared to the Qwen2.5-VL pipeline without a coarse-to-fine CoT process. Furthermore, the STORM pipeline trained on the lite versions yields superior results on HCI palermo2012dating which is a zero-shot domain not appearing during the training process, showing the efficacy of our benchmark in enhancing the model’s universal visual rating abilities. The STORM pipeline trained on the full versions achieves the best performances on both in-domain and out-of-domain tasks, which validate the effectiveness and potential of our data collection.

### 5.3 Ablation Studies

**Table 4: Ablation study on different instruct prompt strategies. “w/o CoT” denotes a standard, non-CoT-based inference process. “Only Num.” and “Only Text” use only numeric and only text instruct prompts, respectively. “Num. + Text” uses both numeric and text instruct prompts.**
|  | IQA | FAE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Instruct Prompt Strategy | Metric | SPAQ | ChallengeDB | KonIQ | Adience | CACD | Morph | UTK |
| w/o CoT | ACC | 0.600 | 0.446 | 0.561 | 0.480 | - | - | - |
| MAE | 0.423 | 0.605 | 0.469 | 0.715 | 7.541 | 7.589 | 6.433 |  |
| Only Num. | ACC | 0.573 | 0.399 | 0.547 | 0.531 | - | - | - |
| MAE | 0.461 | 0.751 | 0.487 | 0.674 | 9.856 | 9.620 | 9.464 |  |
| Only Text | ACC | 0.542 | 0.391 | 0.537 | 0.532 | - | - | - |
| MAE | 0.495 | 0.717 | 0.503 | 0.665 | 9.412 | 9.326 | 8.298 |  |
| Num. + Text | ACC | 0.583 | 0.468 | 0.582 | 0.534 | - | - | - |
| MAE | 0.442 | 0.597 | 0.431 | 0.636 | 8.202 | 5.975 | 5.879 |  |

**Table 5: Ablation study on different training strategies.**
| Training Datasets | IQA | FAE | IAA | MDG | HDE | Average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE | ACC | MAE |  |
| Single | 0.523 | 0.521 | 0.532 | 6.976 | 0.444 | 0.770 | 0.557 | 0.972 | 0.318 | 0.985 | 0.492 | 2.548 |
| Full | 0.544 | 0.490 | 0.534 | 5.173 | 0.562 | 0.483 | 0.559 | 0.963 | 0.341 | 0.924 | 0.533 | 1.958 |

In the ablation studies below, by default, we ablate STORM-3B that is trained and evaluated on the lite version of our datasets with the proposed coarse-to-fine CoT benchmark.

Different Instruct Prompt Strategies. Tab. [4](https://arxiv.org/html/2506.01738v1#S5.T4) shows the performances of our model on the lite version of the visual rating benchmark using different strategies for instruct prompts. As anticipated, the model not employing coarse-to-fine CoT yields lower performance, which indicates inherent challenges in directly predicting ratings. In contrast, our baseline with coarse-to-fine CoT performs better, especially on zero-shot datasets, illustrating the effectiveness of the coarse-to-fine CoT in enhancing robust and general thinking ability for visual rating by learning the ordinal regression nature. In addition, compared to using only numeric labels or text definitions, the MLLM with both numeric labels and text definitions achieves the best performance, showing the effect of both digital and semantic instructions. Notably, text proves to be more effective than numbers, which validates our previous hypothesis that LLMs pre-trained with CLIP are more sensitive to text prompts.

Figure: Figure 4: Visualization results of coarse-to-fine CoT on different datasets.
Refer to caption: x4.png

Different Training Strategies. We conduct ablation experiments on two aspects of the training strategies. (1) The first one is ablation experiments on different selections of training data. For each domain task, we compare the model’s performances after being trained on single-domain datasets versus being trained on all domain datasets. The results are shown in the top part of Table [5](https://arxiv.org/html/2506.01738v1#S5.T5), which indicate that the model performs better after training on all the domains compared to training only on a single domain. This demonstrates that the model can learn generalized and useful ordinal regression properties from different domain tasks, therefore improving the overall performance across various visual rating tasks. It also highlights the advantages and effectiveness of our benchmark and datasets. (2) The second aspect is to explore the effect of different parameter fine-tuning methods for LLMs. We compare the commonly-used Low-Rank Adaptation (LoRA) hu2022lora and Full Fine-Tuning (FTT) methods, and report the results in the lower part of Tab. [5](https://arxiv.org/html/2506.01738v1#S5.T5). One can observe that FTT performs better and is more robust. Hence, we adopt FTT for all the fine-tuning experiments.

### 5.4 Visualization

We visually display STORM’s performance qualitatively in Fig. [4](https://arxiv.org/html/2506.01738v1#S5.F4), highlighting its visual rating ability to conduct a coarse-to-fine CoT process and provide trustworthy predictions. Despite variations in label definitions and ranges across different tasks, the inherent commonality in ordinal nature of labels enables a unified thinking paradigm through progressive refinement of label granularity, achieving coarse-to-fine estimation across these visual rating tasks.

## 6 Conclusions

In this paper, we introduced STORM, a pioneering approach that enhances multi-modal large language models with the all-in-one visual rating capability. This methodology addresses critical gaps in MLLMs, especially in interpretability and processing of dynamic visual input. Our STORM data collection offers 655K annotated question-answer pairs from diverse ordinal regression tasks for comprehensive visual rating learning. Our novel coarse-to-fine processing pipeline allows MLLMs to learn a universal paradigm of ordinal regression and provide intermediate interpretable thoughts. STORM offers a general and trustworthy paradigm for tackling diverse visual rating tasks, and our visual rating benchmark advances the evaluation of MLLMs on both in-domain and out-of-domain tasks. Extensive experiments validated the framework’s effectiveness and robustness, putting forward a promising basis for further exploration in visual rating.

## References

- [1]
Recommendation 500-10: Methodology for the subjective assessment of the quality of television pictures.
ITU-R Rec. BT.500, 2000.
- [2]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
GPT-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- [3]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
GPT-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- [4]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al.
Qwen technical report.
arXiv preprint arXiv:2309.16609, 2023.
- [5]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A frontier large vision-language model with versatile abilities.
arXiv:2308.12966, 2023.
- [6]
Bor-Chun Chen, Chu-Song Chen, and Winston H Hsu.
Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset.
IEEE Transactions on Multimedia, 17(6):804–815, 2015.
- [7]
Shixing Chen, Caojin Zhang, Ming Dong, Jialiang Le, and Mike Rao.
Using ranking-CNN for age estimation.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5183–5192, 2017.
- [8]
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.
PaLI: A jointly-scaled multilingual language-image model.
arXiv preprint arXiv:2209.06794, 2022.
- [9]
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia.
LongLoRA: Efficient fine-tuning of long-context large language models.
arXiv preprint arXiv:2309.12307, 2023.
- [10]
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.
Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.
See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
- [11]
Raul Diaz and Amit Marathe.
Soft labels for ordinal regression.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4738–4747, 2019.
- [12]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv preprint arXiv:2010.11929, 2020.
- [13]
Yao Du, Qiang Zhai, Weihang Dai, and Xiaomeng Li.
Teach CLIP to develop a number sense for ordinal regression.
In European Conference on Computer Vision, pages 1–17. Springer, 2024.
- [14]
Emma Dugas, Jorge Jared, and Will Cukierski.
Diabetic retinopathy detection (2015).
URL https://kaggle. com/competitions/diabetic-retinopathy-detection, 7.
- [15]
Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang.
Perceptual quality assessment of smartphone photography.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3677–3686, 2020.
- [16]
Eibe Frank and Mark Hall.
A simple approach to ordinal classification.
In European Conference on Machine Learning, pages 145–156. Springer, 2001.
- [17]
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao.
Deep ordinal regression network for monocular depth estimation.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002–2011, 2018.
- [18]
Xin Geng, Chao Yin, and Zhi-Hua Zhou.
Facial age estimation by learning from label distributions.
TPAMI, 2013.
- [19]
Deepti Ghadiyaram and Alan C Bovik.
Massive online crowdsourced study of subjective and objective picture quality.
IEEE Transactions on Image Processing, 25(1):372–387, 2015.
- [20]
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al.
AVA: A video dataset of spatio-temporally localized atomic visual actions.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6047–6056, 2018.
- [21]
Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming.
Rethinking image aesthetics assessment: Models, datasets and benchmarks.
In IJCAI, pages 942–948, 2022.
- [22]
Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe.
KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment.
IEEE Transactions on Image Processing, 29:4041–4056, 2020.
- [23]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
LoRA: Low-rank adaptation of large language models.
ICLR, 1(2):3, 2022.
- [24]
Karl Ricanek Jr. and Tamirat Tesafaye.
MORPH: A longitudinal image database of normal adult age-progression.
In FG, 2006.
- [25]
Maggie Karthik and Sohier Dane.
APTOS 2019 blindness detection.
Kaggle https://kaggle. com/competitions/aptos2019-blindness-detection Go to reference in, 5, 2019.
- [26]
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov.
Generating images with multimodal language models.
Advances in Neural Information Processing Systems, 36, 2024.
- [27]
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia.
LISA: Reasoning segmentation via large language model.
arXiv preprint arXiv:2308.00692, 2023.
- [28]
Seon-Ho Lee and Chang-Su Kim.
Deep repulsive clustering of ordered data based on order-identity decomposition.
In International Conference on Learning Representations, 2020.
- [29]
Gil Levi and Tal Hassner.
Age and gender classification using convolutional neural networks.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 34–42, 2015.
- [30]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
arXiv preprint arXiv:2301.12597, 2023.
- [31]
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
- [32]
Ling Li and Hsuan-Tien Lin.
Ordinal regression by extended binary classification.
Advances in Neural Information Processing Systems, 19, 2006.
- [33]
Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou.
Learning probabilistic ordinal embeddings for uncertainty-aware regression.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13896–13905, 2021.
- [34]
Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu.
OrdinalCLIP: Learning rank prompts for language-guided ordinal regression.
Advances in Neural Information Processing Systems, 35:35313–35325, 2022.
- [35]
Yanwei Li, Chengyao Wang, and Jiaya Jia.
LLaMA-VID: An image is worth 2 tokens in large language models.
arXiv preprint arXiv:2311.17043, 2023.
- [36]
Kyungsun Lim, Nyeong-Ho Shin, Young-Yoon Lee, and Chang-Su Kim.
Order learning and its application to age estimation.
In International Conference on Learning Representations, 2019.
- [37]
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning, 2023.
- [38]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In NeurIPS, 2023.
- [39]
Ruhan Liu, Xiangning Wang, Qiang Wu, Ling Dai, Xi Fang, Tao Yan, Jaemin Son, Shiqi Tang, Jiang Li, Zijian Gao, et al.
DeepDRiD: Diabetic retinopathy—grading and image quality estimation challenge.
Patterns, 3(6), 2022.
- [40]
Yanzhu Liu, Adams Wai-Kin Kong, and Chi Keong Goh.
Deep ordinal regression based on data relationship for small datasets.
In IJCAI, pages 2372–2378, 2017.
- [41]
Yanzhu Liu, Adams Wai Kin Kong, and Chi Keong Goh.
A constrained deep neural network for ordinal regression.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 831–839, 2018.
- [42]
Yanzhu Liu, Fan Wang, and Adams Wai Kin Kong.
Probabilistic deep ordinal regression based on Gaussian processes.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5301–5309, 2019.
- [43]
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji.
Cheap and quick: Efficient vision-language instruction tuning for large language models.
Advances in Neural Information Processing Systems, 36, 2024.
- [44]
Frank Palermo, James Hays, and Alexei A Efros.
Dating historical color images.
In European Conference on Computer Vision, pages 499–512. Springer, 2012.
- [45]
Shengju Qian, Huiwen Chang, Yuanzhen Li, Zizhao Zhang, Jiaya Jia, and Han Zhang.
StraIT: Non-autoregressive generation with stratified image Transformer.
arXiv preprint arXiv:2303.00750, 2023.
- [46]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In ICML, 2021.
- [47]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
- [48]
Rasmus Rothe, Radu Timofte, and Luc Van Gool.
Deep expectation of real and apparent age from a single image without facial landmarks.
IJCV, 2018.
- [49]
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
HuggingGPT: Solving ai tasks with ChatGPT and its friends in hugging face.
Advances in Neural Information Processing Systems, 36, 2024.
- [50]
Nyeong-Ho Shin, Seon-Ho Lee, and Chang-Su Kim.
Moving window regression: A novel approach to ordinal regression.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18760–18769, 2022.
- [51]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: A family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 2023.
- [52]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
LLaMA: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.
- [53]
Jinhong Wang, Yi Cheng, Jintai Chen, TingTing Chen, Danny Chen, and Jian Wu.
Ord2Seq: Regarding ordinal regression as label sequence prediction.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5875, 2023.
- [54]
Rui Wang, Peipei Li, Huaibo Huang, Chunshui Cao, Ran He, and Zhaofeng He.
Learning-to-rank meets language: Boosting language-driven ordering alignment for ordinal classification.
Advances in Neural Information Processing Systems, 36, 2023.
- [55]
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.
CogVLM: Visual expert for pretrained language models.
arXiv preprint arXiv:2311.03079, 2023.
- [56]
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al.
VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks.
Advances in Neural Information Processing Systems, 36, 2024.
- [57]
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.
Visual ChatGPT: Talking, drawing and editing with visual foundation models.
arXiv preprint arXiv:2303.04671, 2023.
- [58]
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin.
Q-Bench: A benchmark for general-purpose foundation models on low-level vision.
2023.
- [59]
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, et al.
Q-Instruct: Improving low-level visual abilities for multi-modality foundation models.
arXiv preprint arXiv:2311.06783, 2023.
- [60]
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al.
Q-Align: Teaching LMMs for visual scoring via discrete text-defined levels.
arXiv preprint arXiv:2312.17090, 2023.
- [61]
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.
MM-REACT: Prompting ChatGPT for multimodal reasoning and action.
arXiv preprint arXiv:2303.11381, 2023.
- [62]
Hang Zhang, Xin Li, and Lidong Bing.
Video-LLaMA: An instruction-tuned audio-visual language model for video understanding.
arXiv preprint arXiv:2306.02858, 2023.
- [63]
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo.
GPT4RoI: Instruction tuning large language model on region-of-interest.
arXiv preprint arXiv:2307.03601, 2023.
- [64]
Zhifei Zhang, Yang Song, and Hairong Qi.
Age progression/regression by conditional adversarial autoencoder.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5810–5818, 2017.
- [65]
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
MiniGPT-4: Enhancing vision-language understanding with advanced large language models.
arXiv preprint arXiv:2304.10592, 2023.
- [66]
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu.
MoVA: Adapting mixture of vision experts to multimodal context.
arXiv preprint arXiv:2404.13046, 2024.

## NeurIPS Paper Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] See Appendix F.
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix G.
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[N/A]
(b)
Did you include complete proofs of all theoretical results?
[N/A]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We have provided the related details in Appendix. The code, training data, benchmark, and checkpoints can be found in this page: storm-bench.github.io
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See ‘Training Details’ in Section Experiments. We also provide reproducible scripts that contain all hyperparameters in this GitHub repo: https://github.com/aTongs1/STORM
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B.
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b)
Did you mention the license of the assets?
[Yes]
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes] We provide our code, training data, and checkpoints at this page: storm-bench.github.io
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See Appendix H.
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See Appendix H.
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]

## Appendix A Overview

Our supplementary includes the following sections:

- •
Section [B](https://arxiv.org/html/2506.01738v1#A2): Framework details. Details for model design, implementation and training data.
- •
Section [C](https://arxiv.org/html/2506.01738v1#A3): More Dataset Details and Visualization. More Details and Visualization of our dataset and demos.
- •
Section [D](https://arxiv.org/html/2506.01738v1#A4): More experiment results. Additional performance evaluation and performance analysis.
- •
Section [E](https://arxiv.org/html/2506.01738v1#A5): Prompt design. Prompt for generating the coarse-to-fine CoT dataset and evaluating the performance.
- •
Section [F](https://arxiv.org/html/2506.01738v1#A6): Limitations. Discussion of limitations of our work.
- •
Section [G](https://arxiv.org/html/2506.01738v1#A7): Potential negative societal impacts. Discussion of potential negative societal impacts of our work.
- •
Section [H](https://arxiv.org/html/2506.01738v1#A8): Disclaimer. Disclaimer for the visual rating dataset and the related model.

Following NeurIPS Dataset and Benchmark track guidelines, we have shared the following artifacts:

| Artifcat | Link | License |
| --- | --- | --- |
| Code Repository | [https://github.com/aTongs1/STORM](https://github.com/aTongs1/STORM) | Apache-2.0 license |
| Data | [https://huggingface.co/datasets/ttlyy/ORD](https://huggingface.co/datasets/ttlyy/ORD) | CC BY 4.0 |
| Model Weights | [https://huggingface.co/datasets/ttlyy/ORD](https://huggingface.co/datasets/ttlyy/ORD) | Apache-2.0 license |

The authors are committed to ensuring its regular upkeep and updates.

## Appendix B Framework details

### B.1 Model details

For LLaVA-1.5-7B, we choose the pre-trained ViT-L/14 of CLIP [47] as the vision encoder and Vicuna-7B [10] as our LLM, which has better instruction following capabilities in language tasks compared to LLaMA [52]. For Qwen2.5-VL-3B, the vision encoder the native dynamic resolution ViT. The overview of Qwen-2.5-VL [5] are shown in Fig. [5](https://arxiv.org/html/2506.01738v1#A2.F5). Considering an input original image, we take the vision encoder to obtain the visual feature. Our STORM-3B employes Qwen-2.5-VL-3B as the backbone.

Figure: Figure 5: Overview of Qwen-2.5-VL pipeline.
Refer to caption: extracted/6504787/figs/qwen2.5vl_arc.jpeg

### B.2 Implementation details

Our model undergoes a two-stage training process. In the first stage, we pre-train the model for 1 epoch using a learning rate of 2e-3 and a batch size of 128. For the second stage, we fine-tune the model for 1 epoch on our visual rating dataset, employing a learning rate of 2e-5 and a batch size of 128. The Adam optimizer with zero weight decay and a cosine learning rate scheduler are utilized. To conserve GPU memory during fine-tuning, we employ FSDP (Full Shard Data Parallel) with ZeRO3-style. All models are trained using 32 $\times$ A100s. In the case of training the setting with a 7B LLM and a resolution of 224, the first/second pre-training stage completes within 1/16 hours.

## Appendix C More Dataset Details and Visualization

### C.1 Datasets Training and Testing Split.

In this section, we provide the sample numbers of training and test split of all datasets, as shown in Tab. [6](https://arxiv.org/html/2506.01738v1#A3.T6) and Tab. [7](https://arxiv.org/html/2506.01738v1#A3.T7).

**Table 6: Training and testing split of IQA and IAA domain datasets. Training split includes full version and lite version.**
| Dataset | SPAQ | CDB | KonIQ | AVA | TAD66K | Aesthetic |
| --- | --- | --- | --- | --- | --- | --- |
| Training Full | 8900 | 936 | - | 229958 | 52224 | - |
| Training Lite | 8900 | 936 | - | 25551 | 13056 | - |
| Testing | 2225 | 233 | 2014 | 25550 | 14076 | 1370 |

**Table 7: Training and testing split of FAE, MDG and HDE domain datasets. Training split includes full version and lite version.**
| Dataset | Adience | CACD | Morph | UTK | Eyepacs | DeepDR | APTOS | HCI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training Full | 15589 | 147102 | 40012 | - | 31599 | 1200 | - | - |
| Training Lite | 15589 | 16345 | 10003 | - | 31599 | 1200 | - | - |
| Testing | 1732 | 16344 | 10003 | 2410 | 3527 | 400 | 366 | 132 |

### C.2 Datasets Distribution visualization.

In this section, we provide a visualization of the data statistics. We partition the category distribution of each dataset in Fig. [6](https://arxiv.org/html/2506.01738v1#A3.F6), Fig. [7](https://arxiv.org/html/2506.01738v1#A3.F7), Fig. [8](https://arxiv.org/html/2506.01738v1#A3.F8), Fig. [9](https://arxiv.org/html/2506.01738v1#A3.F9).

Figure: Figure 6: Statistics of the IQA domain datasets.
Refer to caption: x5.png

Figure: Figure 7: Statistics of the IAA domain datasets.
Refer to caption: x6.png

Figure: Figure 8: Statistics of the MDG domain datasets.
Refer to caption: x7.png

Figure: Figure 9: Statistics of the FAE and HDE domain datasets.
Refer to caption: x8.png

## Appendix D More experiment results

### D.1 Larger STORM Model

Tab. [8](https://arxiv.org/html/2506.01738v1#A4.T8) and Tab. [8](https://arxiv.org/html/2506.01738v1#A4.T8) show the performance of STORM-7B using Qwen2.5-VL-7B as the backbone. However, the performance is not much different from the 3B version. Therefore, we choose STORM-3B as the final model.

**Table 8: ACC performance of the STORM-7B.**
|  | IQA | FAE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLLM | Tra. | SPAQ | ChallengeDB | KonIQ | Adience | CACD | Morph | UTK |
| STORM-3B | Lite | 0.583 | 0.468 | 0.582 | 0.534 | - | - | - |
| STORM-7B | Lite | 0.514 | 0.438 | 0.543 | 0.503 | - | - | - |

**Table 9: MAE performance of the STORM-7B.**
|  | IQA | FAE |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLLM | Tra. | SPAQ | ChallengeDB | KonIQ | Adience | CACD | Morph | UTK |
| STORM-3B | Lite | 0.442 | 0.597 | 0.431 | 0.636 | 8.202 | 5.975 | 5.879 |
| STORM-7B | Lite | 0.562 | 0.652 | 0.496 | 0.641 | 7.776 | 5.405 | 5.508 |

### D.2 Confusion Matrixes analysis

We provide more visualization results of confusion matrixes of our STORM on zero-shot datasets in Fig. [10](https://arxiv.org/html/2506.01738v1#A4.F10), Fig. [11](https://arxiv.org/html/2506.01738v1#A4.F11), Fig. [12](https://arxiv.org/html/2506.01738v1#A4.F12), Fig. [13](https://arxiv.org/html/2506.01738v1#A4.F13) and Fig. [14](https://arxiv.org/html/2506.01738v1#A4.F14).

Figure: Figure 10: Confusion matrixes visualization results of the STORM on the KonIQ dataset.
Refer to caption: extracted/6504787/figs/KonIQ.png

Figure: Figure 11: Confusion matrixes visualization results of the STORM on the Aesthetic dataset.
Refer to caption: extracted/6504787/figs/Aesthetic.png

Figure: Figure 12: Confusion matrixes visualization results of the STORM on the Adience dataset.
Refer to caption: extracted/6504787/figs/adience.png

Figure: Figure 13: Confusion matrixes visualization results of the STORM on the APTOS dataset.
Refer to caption: extracted/6504787/figs/APTOs.png

Figure: Figure 14: Confusion matrixes visualization results of the STORM on the HCI dataset.
Refer to caption: extracted/6504787/figs/HCI.png

## Appendix E Prompt design

### E.1 Generating the dataset for IQA

### E.2 Generating the dataset for IAA

| <image> You are now an advanced Aesthetic Evaluation Evaluator, and your task is to assess the aesthetic quality of the provided image. Please evaluate the image’s aesthetic quality based on a 5-level scale: level0(Unacceptable), level1(Flawed), level2(Average), level3(Professional), level4(Excellent). Please first coarsely categorise the image: level0-1(Below Average), level2(Average), level3-4(Above Average). Based on the coarse classification, proceed to make a final level prediction. The specific steps are as follows:<br>1.<br>Make the coarse prediction with the candidates:level0-1(Below Average), level2(Average), level3-4(Above Average).<br>2.<br>Based on the coarse classification, proceed to make a final age prediction with the candidates: level0(Unacceptable), level1(Flawed), level2(Average), level3(Professional), level4(Excellent).<br>3.<br>Please note that the coarse thoughts and the final answer should be consistent. |  |
| --- | --- |
| Answer: [Coarse answer], [Final answer] |  |

### E.3 Generating the dataset for FAE

| <image> You are an experienced facial analysis expert, and you need to estimate the age group of the person in the provided facial image based on their facial features. The known age range of the image is from 16 to 77 years old. Please first coarsely categorise the image: Teenager(16-24 years old), Adult(25-47 years old), Elder(48+ years old). Based on the coarse classification, proceed to make a final age prediction.The final output should be in the format: Coarse Answer: [result], Predicted Age: [result]. The specific steps are as follows:<br>1.<br>Make the coarse prediction with the candidates: Teenager(16-24 years old), Adult(25-47 years old), Elder(48+ years old).<br>2.<br>Based on the coarse classification, proceed to make a final age prediction with the candidates: from 16 to 77 years old.<br>3.<br>Please note that the coarse thoughts and the final answer should be consistent. |
| --- |
| Answer: Coarse answer], [Predicted Age] |

### E.4 Generating the dataset for MDG

| <image> You are an experienced ophthalmologist, and you need to perform disease grading on the provided fundus image. These are all the candidate stages: stage0(no retinopathy), stage1(mild NPDR), stage2(moderate NPDR), stage3(severe NPDR) and stage4(PDR). Please first coarsely categorise the fundus: Normal(stage0), Early(stage1-2), Late(stage3-4). Based on the coarse classification, proceed to make a final stage prediction. The specific steps are as follows:<br>1.<br>Make the coarse prediction with the candidates:Normal(stage0), Early(stage1-2), Late(stage3-4).<br>2.<br>Based on the coarse classification, proceed to make a final age prediction with the candidates: stage0(no retinopathy), stage1(mild NPDR), stage2(moderate NPDR), stage3(severe NPDR) and stage4(PDR).<br>3.<br>Please note that the coarse thoughts and the final answer should be consistent. |  |
| --- | --- |
| Answer: [Coarse answer], [Predicted grade] |  |

### E.5 Generating the dataset for HDE

| <image> You are now an advanced history researcher, and you need to grade the provided images by decade. These are all candidate categories: phase0(1930s), phase1(1940s), phase2(1950s), phase3(1960s), and phase4(1970s). Please first coarsely categorise the image: Early(phase0-phase1), Mid(phase2), Late(phase3-phase4). Based on the coarse classification, proceed to make a final phase prediction.The final output should be in the format: Coarse Classification: [result], Predicted Phase: [result]. The specific steps are as follows:<br>1.<br>Make the coarse prediction with the candidates: Early(phase0-phase1), Mid(phase2), Late(phase3-phase4).<br>2.<br>Based on the coarse classification, proceed to make a final age prediction with the candidates: phase0(1930s), phase1(1940s), phase2(1950s), phase3(1960s), and phase4(1970s).<br>3.<br>Please note that the coarse thoughts and the final answer should be consistent. |  |
| --- | --- |
| Answer: [Coarse answer], [Predicted Phase] |  |

## Appendix F Limitations

The definitions of labels for different domain tasks are quite diverse.

In scenarios where the definitions of labels for different domain tasks are quite diverse, STORM may struggle to possess fluctuation in performance according to different text definitions generated of labels. This places a relatively high demand on the user’s ability to accurately define corresponding text prompts of rating categories.

Our data pipeline inherits the limitations of utilizing GPT-4 API to generate text definition. (1) Accuracy and Misinformation: Generated content may not always be accurate, which could lead to the spread of misinformation. To mitigate this, we have designed a manual adjustment script as a post-process to improve text prompt quality. (2) Bias and Fairness: Since we do not have access to the training data of GPT-4, the generated instructional data might reflect inherent biases, potentially reinforcing social or cultural inequalities present in the base model training. In terms of data usage, we explicitly state that OpenAI’s terms must be adhered to, and the data can only be used for research purposes.

## Appendix G Potential negative societal impacts

The potential negative societal impacts of our work are similar to other MLLMs and LLMs. The development of CoT and MLLMs, while advancing AI, poses societal risks like increased privacy invasion, the perpetuation of biases, the potential for misinformation, job displacement, and ethical concerns regarding accountability and consent.

## Appendix H Disclaimer

This dataset was collected and released solely for research purposes, with the goal of making the MLLMs dynamically focus on visual inputs and provide intermediate interpretable thoughts. The authors are strongly against any potential harmful use of the data or technology to any party.

Intended Use.
The data, code, and model checkpoints are intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision making purposes.

Primary Intended Use.
The primary intended use is to support AI researchers reproducing and building on top of this work. STORM and its associated models should be helpful for exploring various vision question answering (VQA) research questions.

Out-of-Scope Use.
Any deployed use case of the model — commercial or otherwise — is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. The conflict detection between the "intermediate thinking" and the final answer in coarse-fine CoT relies solely on a single prompt; a more comprehensive explanation is needed.

2. Unifying multiple tasks is indeed one of the contributions of this paper. However, is it reliable to divide all tasks into four/five tiers? The spacing between each tier is not equivalent. The authors should provide a detailed analysis of the alignment of each tier within the five task categories, such as 'bad quality' and '4-6 years old'. This seems illogical from a common-sense perspective.

3. More advanced MLLMs such as InternVL3 need to be considered.
soundness: 3
strengths: 1. This work gives the first comprehensive benchmark and dataset suite specifically designed to benchmark and boost the universal visual rating capability of MLLMs across 14 ordinal-regression datasets spanning five domains.

2. A coarse-to-fine Chain-of-Thought prompting pipeline that equips MLLMs with an interpretable, hierarchical ordinal-reasoning paradigm, substantially mitigating numerical hallucination and improving zero-shot generalization.

3. Experiment session is convincing, demonstrated that STORM establishes new state-of-the-art all-in-one and zero-shot performance on both in-domain and out-of-domain visual rating tasks, providing a solid foundation for future research on trustworthy MLLM-based scoring systems.
summary: This paper introduces STORM, an ordinal regression dataset and benchmark designed to enhance the general visual scoring capabilities of MLLMs. STORM integrates 600k+ image-question-answer pairs from 14 ordinal regression datasets across 5 major visual scoring domains, and proposes a coarse-fine CoT prompting paradigm. This paradigm allows the model to first perform coarse-grained candidate filtering during inference before providing fine-grained interpretable scores, thereby mitigating digital illusion and improving cross-domain zero-shot performance. Experiments show that the 3B model fine-tuned based on STORM significantly outperforms baselines such as LLaVA-1.5-7B and Qwen2.5-VL-3B on both in-domain and out-of-domain tasks, validating the effectiveness of the framework and data.
weaknesses: 1. All textual labels rely on GPT-4 generation supplemented by manual fine-tuning, a process that is neither reproducible nor transparent. I understand the high cost of manual data, but this weakens the data contribution of this paper, making it resemble a "combination of existing datasets."

2. The zero-shot claim is mismatched with the evaluation settings. **The paper emphasizes "zero-shot" capability**, yet in most experiments, it uses the training set of the corresponding dataset for fine-tuning (even providing both lite and full scales). Only six test splits are retained for true zero-shot scenarios, resulting in insufficient robustness of the zero-shot performance claim.

3. Different domain tasks (such as IQA, IAA, FAE, MDG) are uniformly evaluated using accuracy and MAE, a mechanism that may not be reliable. Quality and aesthetics are qualitative factors (dividing into five levels is merely for model scoring convenience), while age is clearly quantitative. I am unsure whether dividing age into levels and using a multiple-choice-like evaluation method is reliable.

4. The compared models are all old, such as LLaVA-1.5 and InternVL, lacking new baselines.

5. The article seems to have been written in a hurry, with similar terms (such as MLLM and LMM) used interchangeably. It is recommended to check it.

---

confidence: 3
contribution: 1
initial_rating: 4
presentation: 3
questions: 1. Why does the dataset of IQA exclusively select authentic distortion datasets, omitting synthetic distortion datasets?
2. In the Appendix, there is no detailed explanation for the reason why STORM-7B performs worse than STORM-3B, which is a particularly notable anomaly.
soundness: 3
strengths: The work constructs a dataset comprising 655,000 image-question pairs, encompassing the domains of image quality, aesthetics, age estimation, medical grading, and historical period estimation. Mainstream datasets in these areas often lack textual labels; thus, the authors employed GPT to generate labels, followed by manual adjustments to ensure that the textual definitions align more closely with human scoring habits.
summary: The work presents a systematic benchmark framework for evaluating and enhancing the ordinal regression capabilities of MLLMs in visual scoring tasks. By introducing a coarse-to-fine reasoning mechanism, it aims to improve the subpar performance of MLLMs in these tasks.
weaknesses: 1. Lacks of innovation: although the Coarse-to-Fine Chain-of-Thought approach is practical, this concept is common in IQA.
2. The experiments lack comparisons with SOTA models in the field, instead only comparing with other MLLMs, making it difficult to demonstrate its performance advantages in visual scoring tasks.
3. The work does not provide detailed information on the specific ratios and standards used for the manual adjustments in generating textual labels.
4. The work utilizes Qwen2.5-VL-3B as the backbone, but the results presented in the experiments do not show a significant improvement in performance.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: My rating is mainly based on the limited dataset contribution, relatively poor performance, weakness of novelty, and weak support of the so-called "universal".
soundness: 2
strengths: 1. The submission proposes a coarse-to-fine method to improve existing MLLMs, and it shows performance gains.
2. The submission is clearly written, well-organized, and easy to follow.
summary: The manuscript introduces STORM, a large-scale data collection and benchmark designed to evaluate the Stimulating Trustworthy Ordinal Regression ability of MLLMs for general-purpose visual quality/rating tasks. STORM aggregates 14 ordinal regression datasets spanning five representative visual rating domains.
weaknesses: 1. STORM is essentially a merger of existing datasets without new annotations, which limits its contribution. The text annotations are not truly new, but just reformatted versions of existing numeric scores, so they do not add meaningful value. Overall, for a dataset-focused paper, the work mainly consists of combining prior datasets, which does not meet the expected level of novelty, effort, or complexity.
2. Despite fine-tuning, performance remains weak. Prior work (Q-Align, DeQA-Score) trained on discrete levels reports SRCC/PLCC above 0.9 on KonIQ, SPAQ, and ChallengeDB. In contrast, the proposed method only achieves SRCC or PLCC in the 0.6+ / 0.7+ range on IQA datasets (as shown in the Appendix), which is notably lower than many non-MLLM baselines such as MUSIQ, ManIQA, NIMA, HyperIQA, and DBCNN. This calls the effectiveness of the proposed method into question.
3. Converting numeric labels into fixed textual categories has been well-explored in Q-Align and DeQA, which is not novel. Simply increasing the number of quality levels does not fundamentally distinguish STORM from prior schemes, since those methods could also adjust the level number. A large level number may even harm generalization, so the key issue is choosing an appropriate trade-off rather than just adding more levels.
4. To convincingly claim a universal numeric regression scheme, the method should include a fully training-free version (or at least report test results without any task-specific training). Relying on SFT over a limited set of tasks still leaves generalization concerns, even if some tasks are evaluated in an OOD setting.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Regarding the paper's methods and conclusions, I have the following questions: 
1. What is the specific mechanism of "Coarse-to-fine CoT" during the inference phase? Does the model first generate the coarse thought and then generate the fine-grained answer conditioned on it, or does it jointly output both results? If it is the latter, why is it termed a "Chain-of-Thought" rather than "hierarchical supervision"?
2. In the ablation study in Table 4, how is the "w/o CoT" baseline implemented? Was it also trained using the "Num. + Text" labels for direct prediction? Can you provide a fair comparison to disentangle the respective gains from the CoT structure versus the rich label format?
3. The paper emphasizes "universal" rating capability, yet the "zero-shot" testing is still confined to the five major domains used during training. Can you provide test results on ordinal regression tasks in entirely new (out-of-domain) domains?
4. In Table 2 and Table 3, the performance of STORM-3B (Lite) on several tasks (e.g., IQA-KonIQ, FAE-Morph, FAE-UTK) is significantly worse than the un-fine-tuned Qwen2.5-VL-3B (Zero) baseline. Does this indicate that fine-tuning on the STORM Lite dataset caused negative transfer or catastrophic forgetting? How do the authors explain this phenomenon?
soundness: 2
strengths: - The primary contribution lies in providing the first large-scale, multi-domain benchmark for assessing MLLMs’ visual ordinal regression capabilities. The dataset construction is rigorous, covering 655K samples across five distinct domains (Table 1), filling a gap in this line of research.
- The experimental design is relatively comprehensive, comparing against multiple baseline models (Tables 2-3), and the detailed ablations (Table 4) substantiate the effectiveness of key design choices, especially the advantage of “Num. + Text” labels over using numeric-only or text-only labels.
- Cross-domain training experiments (Table 5) preliminarily demonstrate that joint training on multiple domains yields performance gains over single-domain training, providing empirical support for building universal visual rating models.
summary: This paper introduces the STORM benchmark, designed to evaluate the visual ordinal regression capabilities of multimodal large language models (MLLMs). The authors construct a comprehensive dataset comprising 14 datasets across 5 domains (image quality assessment, facial age estimation, medical disease grading, etc.), totaling 655K samples (Table 1). The core method is a coarse-to-fine Chain-of-Thought (CoT) pipeline (Figures 2-3), which first prompts the model to predict a coarse-grained category (e.g., “Youth”), and then, based on this, predicts a fine-grained result (e.g., “27 years old”). A mixed “text + number” label format is adopted to mitigate numerical hallucinations. Experimental results (Tables 2-3) show that the STORM-3B model trained on Qwen2.5-VL-3B outperforms baselines such as LLaVA-1.5 and MiniGPT-v2 in terms of average accuracy and MAE, and ablation studies (Table 4) validate the effectiveness of both the CoT and the mixed-label strategies.
weaknesses: - The paper exhibits two fundamental issues. First, the so-called “coarse-to-fine CoT” is essentially closer to hierarchical supervision rather than genuine chain-of-thought reasoning. As indicated by Figure 2 and the description on page 5, lines 269-277, the model is trained to output both coarse and fine labels, but the paper does not clearly specify whether, at inference time, there exists a true dependency from the coarse prediction to the fine prediction. The setup of the “w/o CoT” baseline in Table 4 is also unclear, as the paper does not state whether this baseline uses the same “Num. + Text” labels, making it impossible to disentangle whether performance gains derive from the CoT structure itself or merely from the richer label format.
- Second, the argument for “universality” and “zero-shot” capability appears to involve circular reasoning. On page 7, lines 377-381, the paper claims “zero-shot” performance on datasets such as KonIQ, CACD, and APTOS, but these datasets actually belong to domains already used during training (IQA, FAE, MDG; see Table 1). They are merely different datasets within the same domains, which demonstrates cross-dataset generalization within a domain rather than true cross-domain zero-shot capability.