RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
Authors: Xuming He
Zhiyuan You
Junchao Gong
Couhua Liu
Xiaoyu Yue
Peiqin Zhuang
Wenlong Zhang
LEI BAI
Venue: NeurIPS 2025 poster ()
Keywords: Quality Assessment
Multimodal model
AI4Science
Primary area: machine_learning_for_sciences
TL;DR: We introduce an MLLM-based weather forecast quality analysis method, RadarQA, integrating key physical attributes with detailed assessment reports.

Abstract: Quality analysis of weather forecasts is an essential topic in meteorology. Although traditional score-based evaluation metrics can quantify certain forecast errors, they are still far from meteorological experts in terms of descriptive capability, interpretability, and understanding of dynamic evolution. With the rapid development of Multi-modal Large Language Models (MLLMs), these models become potential tools to overcome the above challenges. In this work, we introduce an MLLM-based weather forecast analysis method, RadarQA, integrating key physical attributes with detailed assessment reports. We introduce a novel and comprehensive task paradigm for multi-modal quality analysis, encompassing both single frame and sequence, under both rating and assessment scenarios. To support training and benchmarking, we design a hybrid annotation pipeline that combines human expert labeling with automated heuristics. With such an annotation method, we construct RQA-70K, a large-scale dataset with varying difficulty levels for radar forecast quality evaluation. We further design a multi-stage training strategy that iteratively improves model performance at each stage. Extensive experiments show that RadarQA outperforms existing general MLLMs across all evaluation settings, highlighting its potential for advancing quality analysis in weather prediction.

RadarQA: Multi-modal Quality A Analysis of Weather Radar Forecasts
Xuming Hel.2 Zhiyuan Junchao Gong', Couhua Liu', Xiaoyu Yue',
Peiqin Zhuang'. Wenlong Zhang' Lei Bai" Shanghai Artificial Intelligence Laboratory
2 ZZeJiing University 3 The Chinese University of Hong Kong
Center for Earth System Modeling and Prediction of China Meteorological Administration zhangwenl ongepjlat wla...g......................... org.cn...... bailei@pjlab. org. cn
Abstract
Quality analysis of weather forecasts is an essential topic in meteorology. Although traditional score-based evaluation metrics can quantify certain forecast errors, they are still far from meteorological experts in terms of descriptive capability, interpretability. and understanding of dynamic evolution. With the rapid development of Multi-modal Large Language Models (MLLMs), these models become potential tools to overcome the above challenges. In this work, we introduce an MILM- -based weather forecast analysis method, RadarQA, integrating key physical attributes with detailed assessment reports. We introduce a novel and comprehensive task paradigm for multi-modal quality analysis, encompassing both single frame and sequence, under both rating and assessment scenarios. To support training and benchmarking, we design a hybrid annotation pipeline that combines human expert labeling with automated heuristics. With such annotation method, we construct RQA-70K, a large-scale dataset with varying difficulty levels for radar forecast quality evaluation. We further design a multi-stage training strategy that iteratively improves model performance at each stage. Extensive experiments show that RadarQA outperforms existing general MLLMs across all evaluation settings, highlighting its potential for advancing quality analysis in weather prediction. The code and dataset publicly available at http:///thub.oomm/emmee//////ae
Introduction
Quality analysis of weather forecasts is an essential topic in the field of meteorology [21, 81, 87, 88), playing a critical role downstream applications such as disaster prevention, risk mitigation, and early warning systems [5, 8, 16]. This analysis evaluates the consistency between predicted and actual weather states, both in single frames and over temporal sequences, aiming to align with the assessment of meteorological experts. Previous methods usually adopt score-based metrics for quality evaluation, which still far from matching expert-level judgments. First, some descriptive properties (e.g., shape like "scattered and block-like" and movement direction like "moves to the northeast" in Fig. 1) are vital for weather forecasting, but cannot be captured by simple score. Second, existing methods fail to provide detailed interpretations of the evaluation results, making them less explainable and less convincing. For instance, Fig. I, human experts can first observe that "discrepancies arise in shape changes", and then conclude that the forecast's 's reliability is limited. However, previous score-based metrics lack such interpretive capabilities. Third, human experts can assess the dynamic evolution of weather systems (e.g.. "newly formed convective cells are smaller" in Fig. 1), while score-based metrics are primarily limited to pixel-level evaluations of single frames [11, 12, 48), lacking both temporal awareness and global understanding of large-scale weather systems.
'Equal Contribution. "oorrssponiigg Author.
3 'This work was done during his internship at Shanghai Artificial Intelligence Laboratory.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Query
Expert Knowledge
Our Assessment Report
the obuarisiis sequrie, the connective axteen
Understanding
Compurison
namber connective that intersfy time; expe seatttaned and Mack-iBa andergeing dilation
rheatt
Meving nettrantiend Bowlikk scho dilate over time
Lase nowe been odlle
Sinilar dissppatnng cells Saal dift. sourthwest
Accitionclly the rerge contrage incresse
legending quiliy oseetimett cedanted Consinescy
Boouse wore scatleed organized etructune
Spoliy mitn in
Inteority Inmmanee
Interpretable Dynomic
Descriptive
Santler noving disection
Scnilae speod
Remilar shape
particilorly is the carthest dhe sccle coneecive calls andler compancd deseretten while calle mutenis cinler codes artificct are pessent aradeced ssuence.
differences Noely
the speed nouseant olgg with insorwctione
Sunihe dstribution far dynamic omeaaturccy
dicriittedd
Score-based Metrica
Candlate Pree pitction perfor performance poor with High Valae Ratain parfommonce for care highualla reg fore rolaa are under-pnedctee with the west
northem centrol
CSI
p00
FAR
Prodiction dround Truth
Please sxacua quality predlicted
182
-3
-263
-34
-350
-30
paoviske cumpedaactavoe quality
0.49
0.23
0.28
0.07
0.42
ran
2n sunmary/eebbb deficeecie acale repeosenation relubiiity forccamt expecilly critical auch the aatth
Comparison of our RadarQA and previous score-based metrics. Although score-based metrics reveal some forecast deficiencies, such as false alarms, they lack interpretability and sensitivity to dynamics. Our assessment report combines expert knowledge with these metrics, providing more robust evaluation of the predicted sequence.
Figure 1:
To achieve a better weather forecast analysis aligned with human experts, we introduce RadarQA, a multi-modal model for quality analysis of weather radar forecasts. Inspired by the rapid development of MLLMs [3, 10, 32, 40] and MLLM-based image quality assessment methods [34, 73], we believe that descriptive language to achieve a more flexible analysis of weather forecast. As shown in Fig. 2d, given a reference sequence and model-gennrated prediction, RadarQA produces detailed analysis report from multiple perspectives. First, RadarQA characterizes dynamic properties (e.g.. "moves eastward ... block- like structures"). Then, it evaluates the forecast from various angles. For instance, in terms of the High Value Retain, the performance is just fair because the high value regions in the north under-predicted over time, which is a common over-smoothing problem in weather forecast models [17, 18, 61]. Finally, based on the above considerations, RadarQA judges the predicted sequence as poor quality, noting that it "struggles to accurately replicate key features such as scale changes, precipitation distribution, and high-value retention". This evaluation process aligns closely with human experts and offers better interpretability than traditional score-based metrics.
effectively incorporate expert knowledge and traditional mete-metries
To achieve himan-like weather forecast analysis, we propose a set of new and comprehensive tasks. Human experts typically begin by assessing a temporal weather sequence, where single-frat frame evaluation provides the foundation for sequence assessment. During this process, experts focus on several key factors(e.g.. g., false alarms and misses in a single frame, as well dynamic consistency and retention of high values in a sequence), integrating them into a detailed assessment report through an interpretation process. To imitate this analysis process, shown in Fig. 2, we propose progressive task paradigm consisting of four tasks: (1) Frame Rating, (2) Frame Assessment, (3) Sequence Rating, (4) Sequence Assessment. These tasks meet most common usage scenarios.
To train the expected MILM, we introduce a comprehensive multi-modal dataset, named RQA-70K. Based the SEVIR dataset [58], first implement seven weather nowcasting models to generate preeicepe data. We then carefully design an annotation questionnaire for human experts to annotate 17 key attributes. Besides, also scripts to obtain 20 easily computed metric- based attributes. Finally, all these attributes are input into a powerful large language model (i.e., GPT- 4o [28]) to generate fluent descriptive languages. To this end, we successfully construct large-scale, comprehensive dataset, RQA-70K, laying the foundation for model training.
Based on the collected RQA-70K dataset, we further propose multi-stage training pipeline to train our RadarQA. First, the supervised fine-tuning (SFT) is performed to equip the model with basic task-solving and interpretation capabilities. Second, we design two reward functions and employ reinforcement learning on two rating tasks. This step enhances the model's 's self-reasoning abilities based on the interpretation abilities acquired from the SFT stage. Third, post fine-tuning is applied with a small subset of samples to further refine performance. Our ablation studies show that this multi-stage training pipeline effectively improves performance on both rating and assessment tasks.
Extensive experiments are conducted to evaluate the effectiveness of RadarQA. First, with the support of RQA-70K, RadarQA outperforms open-source MLLMs by large margin (e.g., 66. 17% v.s.

---

(8)
Roting
10)
Tosk3: Sequence Rating
Plett an the pedieed segt band or the for divrtase wish
oo mou soarchs Ti Asseccc The for deseativ decris cestmAao
DDurd ferlo berfomercr Wey furferment youal Alam (orforerco))
FOwnr Purfercenn your. Jyne Owe sravc for Aurtomened yood Camulete harformate
Ierformnce
(b)
Frame Assessment
(d]
Taski: Sequence Assessment
what
dieend becun #foewedd couf
Givk spei The ahulted res aeuiiii Fort deur
Nggribluaand day Mhaee pruv de corll regort
P4
Do datourtoning
harke
conee desss foreeg cligt wif aorarad clate pettinn The
conatu nathon tiad-iita di
unyyy ds dii't per afis inreeaie
frum
daggte organ
The qer mureee wi oft nay moanh soord mopcss tera
congane id "Te di
Tommoro Whate gaeel seceanel the
Predetion
parts colanl dhott nosiis that for sodects
unde-ngeettc very or
fils dore
eborms coppeing byg with are legen or were diod in
she
sa port camu tany wil be dicensiig bok cam dele and and coe luly
let lighodk reriint Houtrer caes poudle peddined
Ovn the tnd and mage semera rones per formnce yotyle cermmnr vot by Theyy colec coloet deerteey inpartt tyr hapinn
ruperce deruadt rcled any per formanc dyner peocpe
mater
Refermee
high water in reteroos meto ahe pelormet of more improvmennt Earecoorig
Task paradigm and qualitative results. RadarQA focuses on four tasks, including frame rating, frame assessment, sequence rating, and sequence assessment, thereby covering both spatial and temporal modalities, and supporting both quantitative and descriptive evaluations.
Figure 2:
36.70% in overall sequence rating). Second, our RadarQA can generate a detailed and comprehensive assessment report, as shown in Fig. 2, even surpassing the powerful OpenAI ol [29] (6.58 v.s. 5.49 in GPT-4 Score for sequence assessment). These results demonstrate the superiority of RadarQA and highlight the research potential of multi-modal weather forecast analysis tasks. Finally, experiments on the out-of-distribution radar data synthesis task further verify the effectiveness of RadarQA.
2 Related Works
Quality assessment of weather forecast leverages verification metries evaluate the accuracy and reliability of weather predictions [13, 14, 31, 43-45, 54, 63, 64]. For example, the Critical Success Index (CSI) [13]. a traditional categorical metric, measures the ratio of correctly predicted events to the total forecasted and observed events, penalizing both false alarms and missed detections. In contrast, the Structural Similarity Index Measure (SSIM) [63], originally developed for general image quality assessment, has been adapted to evaluate the consistency of spatial patterns in weather forecasts. However, as stated Sec. 1, these score-based metrics do not fully align with human experts, particularly in terms of descriptive properties, interpretation process, and the perception of dynamic evolution, making them far from being satisfactory in real-world applications.
Multi-modal Large Language Models (MLLMs) extend Large Language Models (LLMs) [6, 20, 57, 70] by integrating other modalities, particularly vision, to enable unified understanding across different input types. Recent advances in MLLMs [3, 10, 32, 33, 35, 40, 60, 68, 69, 72, 79] have led superior performance on a wide range of tasks, including image captioning [1. 9, 37, 56, 76]. visual question answering [2, 23, 39, 42, 51, 59, 89, 86), and multi-step reasoning [52, 62). However, the weather forecast analysis ability of these MI.LMs is still limited, as shown in Sec. 5.
MLLM-based quality assessment utilizes the power of MILMS to conduct visual quality assessment
across diverse modalities, including images [15, 34, 65-67, 73-75, 85], videos [19, 30, 83] and 3D point clouds [84]. For instance, Q-Insight [34] employs Group Relative Policy Optimization (GRPO) [53] to guide models in reasoning across different tasks. Q-Bench- Video [83] incorporates a diverse
set of videos to assess the video quality through various AlusstionAnsser (QA) formats. LLM-PCQA [84] designs a novel prompt structure that enables MLLMs to perceive the point cloud visual quality.
However, the potential of MLLMs in weather forecast quality analysis is still explored
3 Task Paradigm and Dataset Construction
3.1 Task Paradigm
Meteorological experts typically construct a comprehensive reasoning chain based on both quantitative metrics and expert visual perception of convective structures to evaluate weather forecasting results.
3

---

By examining discrepancies between the ground truth and predictions. experts incorporate prior knowledge, such as domain expertise, provide a quality analysis of the predictions. To align with this expert evaluation process, as highlighted in Sec. I, we aim to establish multi-functional multi-modal, and multi-dimensional task paradigm for quality analysis of weather radar forecast scenarios. Specifically, our RadarQA should possess the following abilities:
Ability-I. RadarQA is required to evaluate differences in dynamic properties across the entire sequence over time, as Fig. 2c, d. Considering that single-frame analysis is the basis of sequence analysis, RadarQA also needs to analyze the quality of individual frames (e.g., tasks in Fig. 2a, b).
Ability-2. RadarQA is required to rate different general attributes, and integrate these ratings into an overall quality rating. This reflects that metcorological experts assist their evaluations by considering a combination of diverse general attributes (e.g., rating tasks in Fig. 2a, c).
Ability-3. RadarQA should be capable of generating high-quality evaluation reports for predictions. This mirrors the real-world workflow where metcorologists compose comprehensive reports for the forecasting department after forming brief judgment (e.g., assessment tasks in Fig. 2b, d).
To reflect the above abilities, we establish a task paradigm with the following four tasks to progressively guide MILMs toward expert-like analysis:
Task-l: Frame Rating. As shown in Fig. 2a, given a model-predicted image and its corresponding ground truth image, the model should assign discrete rating levels for four static general attributes: Miss, False Alarm, Sharpness, and High Value Match, each reflccing a specific aspect of the prediction quality. These are then followed by an Overall performance that summarizes the general quality.
Task-2; Frame Assessment. In addition to provide discrete ratings, the model should generate qualitative descriptions outlining both correctly predicted features and notable deficiencies with respect to some key attributes (e.g.. "significant misses occur for intense and extreme precipitation levels" in Fig. 2b, detailed below). and explain how these attributes affect the overall prediction.
Task-3: Sequence Rating. As illustrated in Fig. 2c, given forecasted sequence, the model is expected to assign quality ratings for three dynamic general attributes: Dynamic Consistency, Cumulative Precipitation, High Value Retain, followed by Overall quality rating.
Task-4: Sequence Assessment. Building upon the sequence rating levels and additional key sequence attributes (detailed below), the model should first provide a comprehensive description of the performance for each dynamic general attributes (e.g. "Newly formed convective cells smaller than observed" in Fig. 2d), then summarize how these dimensions affect the overall performance.
3.2 Scientific Attribute Library
As stated in Sec. 3.1, several key attributes are needed in our task paradigm. Existing evaluation attributes, such as Critical Success Index (CSI) [13] and Probability of Detection (POD) [45], assess prediction quality from various perspectives at the pixel level. Although these metrics capture certain characteristics of weather forecast scenarios, they fall short in identifying discrepancies at the structural level, especially from the perspective of physically grounded convective weather systems. Moreover, existing approaches often overlook the temporal dynamics inherent forecast sequences, which are crucial for analyzing the evolution of physical patterns. To address these limitations, we aim to develop comprehensive scientific attribute library that integrates physics-informet attributes into the quality analysis framework.
Juae Dene
Lign bere
Crange
FAl dverree
Decak
FAN naniel
Roadial
Trojerey
Pertie
al
tetfornca
Mis
was, kuudel
Land
Figure 3:
Overview of our scientific at- tribute library with 5 super-categories and 10 sub-categories in total.
Attribute library. As illustrated in Fig. A5, our at- tribute library is organized into five super-categories, encompassing fundamental physical attributes such as morphology and intensity, atmospheric physics properties like rainfall conservation and convective cycle, as well as temporal characteristies precipitation dynamic distribution. Each super-category comprises multiple sub-categorics, from which we identify key attributes that cover both frame-level

---

and sequence-level features. These attributes are then used to guide the dataset construction. In total, we define 15 frame attributes and 22 sequence attributes. See details in the Appendix.
General attributes used in rating tasks. Under the guidance of domain experts, we identify seven general evaluation attributes. These general attributes used in rating tasks, while all attributes used in assessment tasks as stated in Sec. 3.3. The general attributes are derived by refining existing score-based metrics and integrating perception-basee attributes. The definitions of these attributes are detailed below. (a) Miss. The proportion of convective regions in the ground truth that are not captured by the prediction. (b) False Alarm. The proportion of predicted convective regions that do not correspond to any actual event in the ground truth. (c) Sharpness. The degree to which the predicted convective structures maintain clear, well-defined boundaries. (d) High Value Match. The extent to which the core regions of convective systems, i.e., high-intensity areas, in the prediction align with the high-intensity regions the ground truth. (e) Dynamic Consistency. The ability of the model to accurately capture the evolution of convective systems over time, including factors such as the movement speed, the genesis of convection, and the dissipation of convective cells. (f) Cumulative Precipitation. The ability of the model to reproduce the temporally integrated precipitation amounts associated with convective systems. (g) High Value Retain. The ability of the model to preserve high-intensity regions throughout temporal evolution.
3.3 Dataset Construction
High-quality and large-scale datasets are crucial for training MLLMs to conduct reliable quality analysis. Although post-training techniques such GRPO [53] have shown promising capabilities in enhancing model performance with limited data, it remains essential to first empower the model with intensive and diverse data to ensure bascline competency for the target task. In this section, elaborate on the construction of our dataset, covering forecast data collection, query collection, and response generation. An overview of the dataset construction pipeline is shown in Fig. 4.
Forecast data collection. As shown in Fig. 4, we construct the RawRQA-200 -0K dataset based on the widely used SEVIR dataset [58], which encompasses a wide range of events, including various types of storm events and random phenomena. For our task, we focus exclusively on storm events to build the dataset, covering thunderstorm wind, flood, flash flood, funnel cloud, hail, heavy rain, and tornado. The strong convective nature of these events poses greater forecasting challenges and thus provides higher value for analysis. We focus on the Vertically Integrated Liquid (VIL) modality and split each storm event into three input-targe pairs, where each input consists of 10 consecutive frames and each target consists of the following 12 frames, thus forming a specialized SEVIR subset.
For sequence prediction, we adopt a variety of weather prediction models to generate diverse predicted sequences. These models include EarthFormer [17], PredRNN [61]. Cascast [22], DGMR [47], Diffcast [77], Simvp [18], and Nowcastnet [82], covering a wide range of model architectures, including generative adversarial networks, recurrent neural networks, and diffusion models.
With these model-predicted sequences, we apply VII. discretization and colorization to render the radar data into RGB space. Following [22, 49, 77, 82]. we categorize the VIL values into six precipitation levels reflecting different intensities of convective activity. We then apply the colormap provided by SEVIR to the generated prediction sequences for visualization, resulting in our raw prediction dataset, RawRQA-20K, Additionally, to conduct quality analysis on single frames, we randomly select one frame from each prediction sequence in RawRQA- corresponding ground truth frame. Together, these two data modalities enable a comprehensive evaluation of both static and dynamic properties within individual frames and sequences, respectively.
and pair it with the
Query collection. Following [73, 74], we leverage GPT-4o [28] generate 50 candidate questions for both the brief and detailed tasks. Based on syntactic structure, lexical diversity, and overall clarity, we manually select a set of 10 questions that are both clear and varied. During training and evaluation, these questions are randomly sampled to construct data tuples for model input.
Response collection. As shown in Fig. 2, we employ two types of responses. The first comprises concise, structured outputs for rating tasks, while the second consists of detailed quality reports for assessment tasks. For detailed responses, existing methods primarily rely on either human annotation [67] or generation by MLLMs [66, 74]. However, human annotations often vary in quality [74], and MLLMs remain unreliable for meteorological tasks [7, 41], as evidenced by the results in Sec. 5.
We propose an Attribute-Informee Generation method to enable effective annotation for detailed responses. We observe that key attributes can often be decoupled within evaluation responses
5

---

Paired Data
Annotation
Text Generator
Image
Sequence
Scriets Amsotation
Rating Generator
Fanction
Scierirt Leaking
Varbhry
Human Annotation Questionaire
RawRQA-22 20K Generator
Caption
Comperison
Report Generster
Aur l leaewvng dinal
wiic deanler change anstaeern
RawRQA- 28K
SEVIR
Svert
What
Sucden
Prediction
Preprocess
sydiem chang??
RQA-700
Construction of RQA-70K dataset. First, RawRQA-20K Generator produces frame and sequence samples based on the SEVIR dataset. Next, we annotate the data using script functions and the Human Annotation Questionnaire. Then, Text Generator produces corresponding responses from the annotated attributes, which are paired with question templates to construct RQA-70K.
Figure 4:
constructed by human experts. Inspired by this insight, given a set of annotated key attributes, we leverage them to produce highly informative quality assessment reports, as shown in the Text Generator Module part of Fig. 4. For rating tasks, we automate the generation of JSON-formatted responses based on the general attributes outlined in Sec. 3.1. For assessment tasks, all frame or sequence attributes from the key attribute database are provided to GPT-4o to generate detailed assessment reports. To ensure the reliability the generated response, we also provide GPT-4o with all relevant visual information and explicitly instruct it to correct potential inconsistencies.
Under the Attribute-Infon Informed Generation framework, the focus of dataset construction shifts to attribute annotation. All attributes are categorized into two types: 17 perception- based and 20 metric-basee attributes, whose annotation processes are detailed below.
Perception- attributes involve the understanding of visual content and convective structures, which requires expert knowledge for reliable annotation. Therefore, we employ human annotation to ensure high-quality labeling, as shown in the Human Annotation Questionnaire in Fig. 4. The questionnaire consists of two types of questions: one focuses understanding the observation (i.e., the caption part), and the other evaluates the quality of predictions (i.e., the comparison part). First, experts define labeling guidelines, construct golden standards, and provide reference samples. Second, using these samples, annotators are guided to align with domain experts through pilot testing and iterative refinement to ensure annotation quality. Third, once annotators meet alignment criteria, they proceed to large-scale labeling, during which experts conduct random checks to ensure consistency. If a batch passes validation, it is included in the key attribute database; otherwise, it is returned for re-annotation until the quality standards are met. More details are provided in the Appendix.
Metric-based attributes require precise numerical values. We use the script function to annotate and involve experts in setting key parameters. See Appendix for details.
Dataset statistics. The statistics of our dataset are summarized in Tab. Our dataset consists of 40,000 brief templated samples (training set of rating tasks), along with 29,000 detailed, high-quality samples (training set of assessment tasks). To ensure the reliability of these samples, all annotations undergo expert validation, and automated annotations are routinely verified through expert spot-checking on sampled batches to ensure accuracy.
Table 1: Statistics of our RQA-70K dataset.
Task-1
Task-2 Frame Assessment
Task-3
Task-4
Seguence Rating
Seguence
Rating
Assessment
Train Validation
20,000
14,500 410
20000 801
14500
860
179
4 Model Training
Inspired by [24], we adopt multi-stage training strategy to progressively adapt the model to the domain-specific tasks. In Stage 1, we perform supervised fine-tuning on large-scale multimodal data to equip the model with basic task-solving capabilities. In Stage 2, we use reinforcement learning [50, 53, 78] and carefully design two reward functions for the rating tasks. We encourage the model to reason based on the interpretation abilities acquired from Stage I. In Stage 3, we apply

---

Supervised Finetuning
Reinforcement Learning
Post Training
Encoder
Image
*
Fusion Enaurt Tukene into Tonral Tskoo
Decoder
Reference Model
O
DB
Frozen Weights
Tokena
Tckana
LoRA Weights
KL-Loss
O₂
LoRA Weights
Policy Model
O2
F2
Frozen
Text
Weights
Reward
RadarQA
Enceder
Tecthal Takens
GRPO
Figure 5: Training pipeline of RadarQA. First, we apply supervised fine-tuning with LoRA on RQA-70K to equip the model with basic capabilities. Then, GRPO is used to enhance performance on rating tasks by leveraging its learned assessment ability. Finally, post- pest-traaiing is applied to standardize output formats and further improve overall performance.
post-training with small set of samples further refine performance. An overview of training pipeline is shown in Fig. 5. We validate the effectiveness of our multi-stage training strategy in Sec. 5.3, which demonstrates consistent performance improvements at each stage.
Stage Supervised fine-tuning. We employ RQA-70K for supervised fine-tuning this stage. Since full LLM fine-tuning is highly computationally demanding and requires large-scale datasets, we adopt LoRA [27]. a parameter-efficeee into certain layers while keeping most original parameters frozen, address the issue of limited data.
ane-ttunng method that injects trainable low-rank matrices
Stage 2: Reinforcement learning. Inspired by [34], adopt GRPO [53] in the second stage to optimize the model's 's performance on the rating tasks. In this phase, the fine-tuned model from Stage serves as the policy model be further refined. Since GRPO requires well-defined reward functions to guide policy updates, we introduce two task-specifie rewards. (a) Format Reward. The model is required generate responses in a well-structured JSON format, where each key corresponds to general attribute of the brief task. Denoting the format reward as ffmat. If the response can be successfully parsed into valid ISON object and required keys are present, we set T'mmat Otherwise, the reward is O. (b) Accuracy Reward. If the response generated by the policy model can be correctly parsed into a valid ISON format, we compare the predicted performance levels for each general attribute with the corresponding ground truth labels. Let Naw be the total number of general attributes and Nba the number of correctly predicted general attributes. The accuracy reward defined as face Nha/Nau if mat 1; otherwise, it is set to 0.
Stage 3: Post-training. To further refine model performance, we conduct post- post-training in this stage by using a small subset of RQA-70K, applying low-rank LoRA updates for effective adaptation.
5 Experiments
5.1 Details and Metrics
Implementation details. We adopt Qwen-2.5-VL-7B [3] the base model. In Stage 1, we employ A as the optimizer, with an initial learning rate of 10-4, We integrate LoRA with a rank of 8, The model is trained with total batch size of 128 for 5 epochs on RQA-70K. In Stage 2, we set the generation number of GRPO brief task samples with total batch size of 32. In Stage 3, we set the LoRA rank to 4 and fine-tune the model for epoch using 2,500 samples from each sub-task. The entire training process takes approximately 50 hours using NVIDIA A800 GPUs.
and train the model for I epoch on 10,000 randomly selected
Metrics. For the rating tasks, we adopt accuracy as the evaluation metric. Specifically, we prompt MLLMs to generate responses in a structured JSON format with predefined keys. Accuracy is then computed separately for each general attributes. For the assessment tasks. employ standard metrics, including BERTScore [80]. BLEU [46], ROUGE [36], and METEOR [4]. Following [38, 73], we also incorporate the GPT-4 score, where the model's response is rated from 10 based on relevance, accuracy, and level of detail with respect to the ground truth.

---

Table 2: Results on general attributes for the frame rating and frame assessment tasks. Accuracy is used as the metric for the frame rating task. RadarQA surpasses all baselines by a large margin.
Methods
Frame Rating
Frame Assessment
[Overall False Alarm
Miss
High Value Sharpmess
BLEU
ROUGE METEOR 4PT4Score
Qwen2.5. VB
20.10
36.40
30.00
16.51
35.93
0.122
0.750
0.389
0.332
3.81
Open Source
InternV1.2.5-888
30.89
21.86
8.95
1.04
36.51
0.114
0.745
0.426
0.335
3.50
Qwan225-VLL-728
23.76
27.72
40.59
6.93
39.60
0.132
0,749
0,396
0.324
4.32
GPT4o
48.84
31.40
23.85
11.04
52.91
0.116
0,760
0.408
0.345
5.27
API- hased
Clande3.7 somnel
39.77
32.79
27.21
21.74
43.14
0.083
0.754
0.377
0.350
5.89
Gemini2.5 pro
21.40
29.65
31.16
30
40.58
0.080
0.741
0.348
0.326
5.77
ol
52.67
28.86
23.83
28.15
50.58
0.091
0,739
0,330
0.288
5.63
Ours
RadarQA
61.51
65.35
67.67
69.19
78.60
0.213
0.809
0.512
0.420
6.87
Table 3: Results on general attributes for the sequence rating and sequence assessment tasks.
Accuracy is used as the metric for the sequence rating task. RadarQA achieves the best performance.
Sequence Rating
Sequence Asseasment
Methods
Overall
Dynamic Consistency
Cumulate Precipitation
High Value Retain
BLEU
BERTScore ROUGE_L METEOR GPT4Score
Open
Qwen2.55VL-77
7.99
16.10
17.49
23.22
0.090
0.745
0.281
0.342
3.92
In. 8B
36,70
40.20
31.46
21.10
0,010
0.636
0.241
0.251
2.61
Source
Qwen2.5.
19.80
46.53
23.76
7.92
0.132
0,740
0.329
0.335
4.72
GPT4e
45.00
22.60
26.59
4.99
0.11
0.757
0,323
0.369
4.39
API- based
Claude3.7 sonnet
19.48
26.22
21.10
14:48
0.052
0.737
0.266
0,337
5.56
Gemini2.5 pro
27.59
28.34
26.72
22.47
0.055
0.739
0,254
0.341
5.63
29.70
33.66
29.70
19.80
0.091
0.733
0.254
0.304
5.49
Ours
RadarQA
66.17
53.31
48.94
80.52
0.212
0.815
0,436
0.461
6.58
5.2 Experimental Results
Quantitative results of frame rating task are shown in Tab. 2. First, the performance of open-source MILMMs remains limited. In particular, for the High Value Match attribute, all three open-source baselines achieve accuracies below 20%, indicating that they still struggle to associate different rainfall intensities with the corresponding color mappings. Second, among the API-based methods, ol outperforms other models under the same evaluation setting. Finally, RadarQA significantly surpasses all baseline methods, demonstrating the superior effectiveness of our approach.
Quantitative results of frame assessment task are illustrated in Tab. 2. First, open-source models exhibit clear limitations on the more challenging frame assessment task; their relatively low GPT-4 scores indicate lack of domain-specific understanding. Second, among the API-based models, Gemini 2.5 Pro achieves the best overall performance. Finally, RadarQA outperforms all baselines across all metrics, demonstrating its superior ability to capture and interpret convective features.
Quantitative results of sequence rating task are demonstrated in Tab. 3. First, among open-source models, In-e---V--2.5-BB [10] achieves the best performance, even surpassing the larger Qwen-VL. WL- 2.5-72B [3]. Second, API-based models consistently exhibit limited capability on sequence rating, with average accuracies ranging between 20% and 30% Finally, RadarQA outperforms all baseline methods, particularly achieving over 80% accuracy on the High Value Retain attribute.
Quantitative results of sequence assessment task are shown in Tab. 3. First, compared to frame assessment, both open-source and API-based models perform worse on sequence assessment, indicating that understanding and assessing sequences is more challenging. This is primarily due to two factors. (a) The inherent complexity of video modality, which requires analyzing temporal correlations across frames. (b) The construction of ground truth responses based on a large number of expert-annon annotated attributes, which involve various metcorological concepts such as "Convection Genesis" in Fig. A5. Second, RadarQA still achieves excellent performance, highlighting its superior capabilities in interpreting temporal information.
Qualitative results of assessment tasks are illustrated in Fig. 2 and Fig. 1. First, RadarQA effectively captures the dynamic evolution of convective systems (e.g., "dilating over time while the degree of organization decreases" in Fig. 2). Second, RadarQA can also interpret key deficiencies across multiple dimensions (e.g., "struggles to accurately replicate key features such as scale changes" in Fig, 2). Additional qualitative results for assessment tasks are provided in the Appendix.
8

---

Table 4: Ablation studies of multi-stage training strategy. Frame sequence rating tasks are evaluated in average accuracy, while frame / sequence assessment tasks are assessed in GPT-4 Score. Our full 3-stage training pipeline achieves the best results.
Table 5:
Results out-of-distribution task. RadarQA is requested to evaluate radar reflectivity reconstruction task, which is unseen during training. Frame sequence rating tasks are evaluated in av- erage accuracy, while frame sequence assessment tasks are assessed in GPT-4 Score.
Stuge-1
Stago-2
Stago-3
Rating
Assessment
Metbods
Rating
X
x
X
27.79/ / 16.20
64.05 55.15
6.40/6.22
3.81 /3.92
Open Source
Qwen 2.5.V225.VLL.721 VL.
22.72/21.15
3.60/3.78
V
x
66.95 / 61.58
3
X
68. 14 62.17
6.53/6.56 6.87
API- based
GPT-4o
23.17/23.71
4.30/ 3.82
68.46 / 62.24
ol
32.28/17.33 /1.31
4.34/4.36
Ours
RadarQA
59.94/48.
6.22 /5.6
"Stage-2 is trained only on rating tasks.
Table 6:
Ablation studies of multi-dataset 9.5
GPT-4o RadarQA O Ground Truth
joint training. Training on four tasks outper- forms training on each task. Metries are average accuracy (Task-1 & Task-3) and GPT-4 Score (Task-2 & Task-4). For single-task training,
9-
8.5
7.5
7-
each task is trained on its corresponding dataset.
6.5
Training data
Task-1
Task-2
Task-3
Task-4
Frame A ssessment
Sequence Assessment
Single-task data
66.63
6.48
59.55
6.17
Figure 6: Expert Study frame assessment and sequence assessment tasks.
All-task data
6.87
62.24
6.58
Results on out-of-distribution task are illustrated in Tab. 5. We employ three models designed for radar reflectivity reconstruction, including DifISR [25], SRVIT [55], and U-Net [26], to generate out-of-distribution (OOD) samples on a different dataset for evaluation. For both the frame rating and assessment tasks, RadarQA maintains high accuracy even under the challenging OOD setting and significantly outperforms the baseline methods. For sequence rating and assessment tasks, although performance declines to some extent, RadarQA still surpasses all baselines by notable margin. This performance gap is primarily due to the lack of explicit temporal modeling in radar reflectivity reconstruction. When each frame in a sequence is predicted independently. the resulting sequence lacks temporal coherence, which may hinder the model's ss ability to make consistent assessments.
Expert study. To evaluate the alignment between RadarQA and human experts, we invited metcorologists to rate the ground truth, RadarQA, and GPT-4o on the assessment tasks on three criteria: content accuracy, information density, and coverage of expert-concerned issues. As shown in Fig. 6, both the ground truth and RadarQA outperform GPT-4o, confirming the effectiveness of the task design and the strong performance of RadarQA. Moreover, scores the sequence assessment task are generally higher than those on the frame assessment task, highlighting the value of integrating expert knowledge into the assessment process.
5.3 Ablation Studies
Training strategy. To enhance model performance, we adopt multi-stage training pipeline (see Fig. 5) comprising supervised fine-tuning, reinforcement learning, and post-training. To evaluate the effectiveness of each stage, we compare models trained with different combinations of the three training stages. First, after the Stage training, the model demonstrates relative improvement of 40% in average accuracy on rating tasks and achieves around 2.5 2.5-point increase in GPT-4 Score on assessment tasks, indicating enhanced domain understanding (i.e., #0 v.s. #1 Tab. 4). Second, combining Stage with either Stage 2 or Stage yields further improvements over using Stage I alone (i.e., #2 #3 in Tab. 4). Finally, as shown in #4 Tab. 4, the full training pipeline achieves the best performance across all four tasks, demonstrating the effectiveness of our training strategy.
Joint training on multiple tasks. To demonstrate the effectiveness of multi-tase training, we compare our jointly trained RadarQA with four single-task variants, each trained separately on a specific task. As shown in Tab. 6, RadarQA consistently outperforms all single-task models across their respective metrics, highlighting the overall efficacy of our multi-task training approach.

---

6 Conclusions and Limitations
We introduce RadarQA, an MI.LM-based model for quality analysis of weather radar forecasts. Empowered by a novel task paradigm, a high-quality dataset RQA-70K, and multi-stage training pipeline, RadarQA outperforms all baseline methods across all tasks and under out-of-distribution settings, demonstrating potential for advanced applications in meteorology.
Limitations. First, our task paradigm not yet fully unified. Extending the framework support comparisons between two predicted results can further enhance practicality. Second, the fine-graine descriptions are still not satisfactory. Finally, whether the assessment outputs can serve as feedback or rewards to improve forecasting models remains underexplored. These are left for future work.
7 Acknowledgements
This work is Supported by Shanghai Artificial Intelligence Laboratory.
References
[1] H. Agrawal, K. Desai, Y. Wang. X. Chen, R. Jain, M. Johnson, Batra, Parikh, S. Lee, and Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019.
[2] S. Antol, A, Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, VQA: Visual question answering. In ICCV, 2015.
[3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502: 2025.
[4] S. Banerjec and A. Lavic. METEOR: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL Workshops, 2005.
[5] K. Bi, L. Xie, H. Zhang, X. Chen, X. Gu, and Q Tian. Pangu-weatler A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXi::221.002556 2022.
[6] X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding. K. Dong, Q. Du. Z. Fu, et al. Deepscek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0295. 2024.
[7] J. Chen, P. Zhou, Y. Hua, D. Chong, M. Cao, Y. Li, Z. Yuan, B. Zhu, and J. Liang. Vision-language models meet metcorology: Developing models for extreme weather events detection with heatmaps. arXiv preprint arXiv:2406. 09838, 2024.
[8] K. Chen, Han, J. Gong, Bai, Ling. J.JJ. Luo, X. Chen, L. Ma, Zhang, R. Su, et. Fengwu: Pushing the skillful global medium-range weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02944 2023.
[9] X. Chen, H. Fang, T-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zimick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504.00325, 2015.
[10] Z. Chen, W. Wang, Y. Cao. Y. Liu, Z. Gao, E. Cui. J. Zhu, S. Ye, H. Tian, Z. Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 2r 2024.
[11] C. Davis, B. Brown, and R. Bullock. Object-based verification of precipitation forecasts. part i: Methodology and application to mesoscale rain areas. Monthly Weather Review, 134(7):172-17844 2006.
[12] C. Davis, B. Brown, and R. Bullock. Object-basede verification of precipitation forecasts, part ii: Application to convective rain systems. Monthly Weather Review, 134(7):1785-1795 2006.
[13] R. Donaldson, R. M. Dyer, and M. J. Kraus. An objective evaluator of techniques for predicting severe weather events. Preprints, Ninth Conf. on Severe Local Storms, Norman, OK, Amer. Meteor. Soc, 1975.
[14] J. P. Finley. Tornao predictions. American Meteorological Journal. A Monthly Review of Meteorology and Allied Branches of Study ( (1884- 1896). 1884.
[15] J. Gao, R. Liu, Y. Peng, S. Yang. J. Zhang. K. Yang. and Z. You. Deqa-doc: Adapting dega-score to document image quality assessment. In ICCV, 2025.
10

---

[16] Y. Gao, H. Wu, R. Shu, H. Dong. F. Xu. R. Chen, Y. Yan, Q. Wen, X. Hu, K. Wang, et al. Oneforecast: A universal framework for global and regional weather forecasting. arXiv preprint arXiv:2502.00338, 2025.
[17] Z. Gao, X. Shi, H. Wang. Y. Zhu, Y. B. Wang. M. Li, and D.-Y. Yeung. EarthFormer: Exploring space- time transformers for earth system forecasting. NeurIPS, 2022.
[18] Z. Gao, C. Tan. L. Wu, and S. Z. Li. Simvp: Simpler yet better video prediction. CVPR, 2022.
[19] Q Ge, W. Sun, Zhang, Li, Ji. F. Sun, S. Jui, X. Min, and Zhai. LMA-VQA: Advancing video quality assessment with large multimodal models. arXiv preprint Siii::2408.1000000 2024.
[20] T. GLM, A. Zeng. B. Xu, B. Wang, C. Zhang, D. Yin. D. Zhang, D. Rojas, G. Feng, H. Zhao, et al. ChatGLM: A family of large language models from glm-130h to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
[21] F. Gofa, D. Boucouvala, P. Louka, and H. Flocas. Spatial verification approaches tool to evaluate the performance of high resolution precipitation forecasts. Amoosperic Research, 2018.
[22] J. Gong, L. Bai, P. Yc, W. Xu, N. Liu, J. Dai, X. Yang, and W. Ouyang. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling. arXiv preprint arXiv:2402.04290, 2024.
[23] Y. Goyal, Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter; Elevating the role of image understanding in visual question answering. In CVPR, 2017.
[24] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al. Deepscck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
[25] X. He, Z. Zhou, W. Zhang. X. Zhao, H. Chen, S. Chen, and L. Bai. DiffSR: Leamning radar reflectivity synthesis via diffusion model from satellite observations. In ICASSP, 2025.
[26] K. A. Hilburn, Ebert- Uphoff. and D. Miller. Development and interpretation of neural-network-basee synthetic radar reflectivity estimator using goes-r satellite observations. Journal of Applied Meteorology and Climatology, 2020.
[27] E. J. Hu, Y. Shen, P. Wallis, Z. Alu, Y. Li, S. Wang. L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[28] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, al. GPT-4o system card. arXiv preprint arXiv:2410.21226, 2024.
[29] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai ol system card. arXiv preprint 16X12224/2.16720, 2024.
[30] Z. Jia, Z. Zhang, Qian, H. Wu, W. Sun, C. Li, X. Liu, W. Lin, G. Zhai, and X. Min. VQA²: Visual question answering for video quality assessment. arXiv preprini arXiv:2411.0379,, 2024.
[31] I. T. Jolliffe and D. B. Stephenson. Forecast verification: a practitioner's guide in atmospheric science. John Wiley & Sons, 2012.
[32] B. Li, Y. Zhang, D. Guo, R. Zhang. F. Li. H. Zhang, K. Zhang, P. Zhang. Y. Li, Z. Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0332 2024.
[33] F. Li, R. Zhang, H. Zhang, Y. Zhang, Li, W. Li, Z. Ma, and C. Li. Llava-nextiinterleave: Tackling multi image, video, and 3d in large multimodal models, arXiv preprint arXiv:2407.07895 2024.
[34] Li, X. Zhang, S. Zhao, Y. Zhang, Li, L. Zhang, and Zhang. Q-Insight: Understanding image quality via visual reinforcement learning. arXiv preprint arXiv:2503.2267. 2025.
[35] B. Lin, Y. Ye, B. Zhu, J. Cui, M. Ning, P. Jin, and L. Yuan. Vidco-llave Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 10122, 2023.
[36] C.-Y. Lin. Rouge: A package for automatic evaluation of summaries. In ACL, 2004.
[37] F. Liu, Y. Wang, T. Wang, and V. Ordonez. Visual news: Benchmark and challenges in news image captioning. arXiv preprint 2xXiv:2010.03743, 2020.
[38] H. Liu, C. Li. Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurlPS, 2023.

---

[39] Y. Liu, H. Duan, Y. Zhang, B. Li. S. Zhang. W. Zhao. Y. Yuan, J. Wang, C. He. Z. Liu, et al. MMBench: Is your multi-modal model all-around player? In ECCV, 2024.
[40] H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. H. Yang, et al. Deepseek-vl: towards real-world vision- lanoun-anguuae understanding. arXiv preprint arXiv:2403.05523, 2024.
[41] C. Ma, Z. Hua, A. Anderson-Frey, V. Iyer, X. Liu, and L. Qin. WeatherQA: Can multimodal language models reason about severe weather? arXiv preprint arXiv:2406.11217, 2024.
[42] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
[43] A. H. Murphy. The finley affair: A signal event in the history of forecast verification. Weather and forecasting, 1996.
[44] W. Palmer and R. Allen. Note the accuracy of forecasts concerinng the rain problem. US Weather Bureau, 1949.
[45] H. A. Panofsky and G. W. Brier. Some applications of statistics to meteorology. Mineral Industries Extension Services, College of Mineral Industries, Pennsylvania State University, 1958.
[46] K. Papineni, S. Roukos, T. Ward, and W.-. Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
[47] S. Ravuri, K. Lenc, M. Willson, D. Kangin, R. Lam, P. Mirowski, M. Fitzsimons, M. Athanassiadou. S. Kashem, Madge, ct al. Skilful precipitation nowcasting using deep generative models of radar. Nature, 2021.
[48] M. Rempel, F. Senf, and Deneke. Object- based metrics for forecast verification convective development with geostationary satellite data. Monthly Weather Review, 145())3161-3378, 2017.
[49] M. Robinson, Evans, and B. Crowe. En route weather depiction benefits of the nexrad vertically integrated liquid water product utilized by the corridor integrated weather system. In Conference on aviation, range and aerospace meteorology, american meteorological society, 2002.
[50] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:/707.00000 17077063777 2017.
[51] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-OKVQA: A benchmark for visual question answering using world knowledge. In ECCV, 2022.
[52] Н. Shao, S. Qian, H. Xiao, G. Song, Z. Zong, L. Wang, Y. Liu, and H. Li. Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning. In NeurlPS, 2024.
[53] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 40X.v.240..03300 2024.
[54] D. B. Stephenson, B. Casati, C. Ferro, and C. Wilson. The extreme dependency score: A non-var nonishnishing measure for forecasts rare events. Meteorological Applications: journal of forecasting, practical applications, training techniques and modelling. 2008.
[55] J. Stock, K. Hilburn, I. Ebert- Uphoff, and C. Anderson. Srvit: Vision transformers for estimating radar reflectivity from satellite observations at scale. arXiv preprint arXiv:2406. 169555 2024,
[56] K. Sun, Pan, Y. Ge, Li, H. Duan, X. Wu, R. Zhang, A. Zhou, Z. Qin, Wang, et al. Journeydb: benchmark for generative image understanding. In NeurIPS, 2023.
[57] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, Goyal, E. Hambro,
F. Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint 2302 2023.
[58] M. Veillette, S. Samsi, and C. Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite metcorology. In NewrIPS, 2020.
[59] F. Wang, M. Chen, X. He, Y. Zhang, F. Liu, Z. Guo, Z. Hu, J. Wang. J. Xu, Z. Li, et al. Omnicarth- bench: Towards holistic evaluation of earth's six spheres and cross-spheres interactions with multimodal observational earth data, arXiv preprint 25X::22505..322... 2025.
12

---

[60] W. Wang. Q. Lv. W. Yu, W. Hong, J. Qi. Y. Wang. J. Ji, Z. Yang. L. Zhao, S. XiXuan, et al. Cogvlm: Visual expert for pretrained language models. In NeurIPS, 2024.
[61] Y. Wang, M. Long. J. Wang, Z. Gao, and P. S. Yu. PredRNN: Recurrent neural networks for predictive learning using spatiotemporal lstms. In NeurIPS, 2017.
[62] Y. Wang. Y. Zeng, J. Zheng, X. Xing, J. Xu, and X. Xu. VideoCoT: A video chain-of-thought dataset with active annotation tool. arXiv preprint arXiv:2407.05355 2024.
[63] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
[64] C. J. Willmott and K. Matsuura. Advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. Climate research, 2005.
[65] H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang. C. Li, W. Sun, Q. Yan, G. Zhai, et al. Q-Bench: benchmark for general 2023.
general-purpose purpose foundation models on low-level vision. arXiv preprint arXiv:2309. 14181,
[66] H. Wu, Z. Zhang, E. Zhang, C. Chen, L. Liao, A. Wang, K. Xu, C. Li, Hou, G. Zhai, et al. Q- QnInttruct: Improving low-level visual abilities for multi-modality foundation models. In CVPR, 2024.
[67] H. Wu, Zhu, Z. Zhang, Zhang, C. Chen, L. Liao, C. Li, A. Wang, W. Sun, Q Yan, et al. Towards open-ended visual quality comparison. In ECCV, 2024.
[68] Z. Wu, X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai, H. Gao. Y. Ma, C. Wu, B. Wang, et al. Deepseek- v12: Mixture-of-experts vision- lision-language models for advanced multimodal understanding. arXiv preprint a4Xiv:2412. 10302, 2024.
[69] Xu, Z. Guo, J. He, H. Hu, T. He. S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, el al. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215. 2025.
[70] A. Yang, B. Yang, Zhang. B. Hui, Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2.5 technical report. arXiv preprint ar 21:1::4112. 2024.
[71] J. Ye, H. Xu, H. Liu, Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou. mplug-ow13: Towards long image-sequence understanding multi-modal large language models. arXiv preprint arXiv:2408.04884 2024.
[72] Q Ye, H. Xu, J. Ye, Yan, A. Hu, H. Liu, Q Qian, Zhang, and F. Huang. mplug-ow owl1: Revolutionizing multi-modal large language model with modality collaboration. In CVPR, 2024.
[73] Z. You, J. Gu, Z. Li, X. Cai, K. Zhu, C. Dong, and T. Xue. Descriptive image quality assessment in the wild. arXiv preprint arXiv:2405. 18842 2024.
[74] Z. You, Z. Li. J. Gu. Z. Yin. T. Xue, and C. Dong. Depicting beyond scores: Advancing image quality assessment through multi molli-modal language models. In ECCV, 2024.
[75] Z. You, X. Cai, J. Gu. T. Xue, and C. Dong. Teaching large language models regress accurate image quality scores using score distribution. In CVPR, 2025.
[76] P. Young. A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Asxociation for Computational Linguistics, 2014.
[77] D. Ya, X. Li, Y. Yc, B. Zhang, C. Luo, K. Dai, R. Wang, and X. Chen. Diffcast: A unified framework via residual diffusion for precipitation nowcasting. CVPR, 2024.
[78] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. Dapo: An open-source I1m reinforcement learning system scale. arXiv preprint arXiv:2503. 250..2560..1. 2025.
[79] P. Zhang, X. Dong, B. Wang, Y. Cao, C. Xu, L. Ouyang, Z. Zhao, H. Duan, S. Zhang, S. Ding, al. Internim- xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 151 12, 2023.
[80] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger. and Y. Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09:
2019.
[81] Y. Zhang, H. Yu, M. Zhang, Y. Yang, and Z. Meng. Uncertainties and error growth in forecasting the record-breakinn breaking rainfall in zhengzhou, henan 19 20 july 2021. Science China Earth Sciences, 2022.
13

---

[82] Y. Zhang, M. Long, K. Chen, L. Xing, R. Jin, M. Jordan, and J. Wang. Skilful nowcasting of extreme precipitation with nowcastnet. Nature, 2023.
[83] Z. Zhang, Z. Jia, H. Wu, C. Li, Z. Chen, Y. Zhou, W. Sun, X. Liu, X. Min, W. Lin, et al. (Bench-Vddes Benchmarking the video quality understanding of Imms, arXiv preprint arXiv:2409.20061, 2024.
[84] Z. Zhang, H. Wu, Y. Zhou, C. Li, W. Sun, C. Chen, X. Min, X. Liu, W. Lin, and G. Zhai. LMM-PCQA: Assisting point cloud quality assessment with LMM. In ACM MM, 2024.
[85] 7. Zhang, T. Kou, S. Wang, C. Li, W. Sun, W. Wang. X. Li, Z. Wang, X. Cao, X. Min, et al. 1-Fval-100K: Evaluating visual quality and alignment level for texi-to-vision content. arXiv preprint arXiv:250.00235, 2025.
[86] X. Zhao, W. Xu, B. Liu, Y. Zhou, F. Ling, B. Fei, Yue, L. Bai, W. Zhang, and X.-M. Wu. Msearth: benchmark multimodal scientific comprehension earth science. arXiv preprint arXiv:2505.20740. 2025.
[87] Q Zhong, Z. Sun, Chen, Li, and Shen. Multi model forecast biases of the diurnal variations of intense rainfall in the Bejjing-tinnjin-hbeee region. Science China Earth Sciences, 2022.
[88] Zhou, J. Wu, M. Chen, Han. Comparative study on the performance convistm and convgru classification problems-taleeee early warning of short-duration heavy rainfall as an example. Atmospheric and Oceanic Science Letters, 2024.
[89] Y. Zhou, Y. Wang, X. He, R. Xiao, Z. Li, Q. Feng, Z. Guo, Y. Yang, H. Wu, W. Huang, et al. Scientists' first exam: Probing cognitive abilities of milm via perception, understanding, and reasoning. arXiv preprint arXiv:2506. 10521, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations of the work are discussed in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those
discussed the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Experimental details are provided in Sec. Sec. 4, Sec. 5 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be a way access this model for reproducing the results a way to reproduce the model (e.g., with open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, it may be that access the model limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
Codes and datasets will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp cc/ public/guidest CodeSubmi ssi
wwtprbiissssooooooiioooooiieme onPolicy) for more details.
While we encourage the release code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/niis. clss.ccppulle htc/public/guiii
ssi oniooicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Details are provided Sec. 3, Sec. 4, Sec. and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report the experimental results following the convention in MLLM-based quality assessment research, the same as previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The information on the computer resources is shown in Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't t make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://nent /ttee://neeuipeeeeeee cctppbllc//ttthtteececeeeeo dtw.bi/ieeetihiiiiilelellleeleee
Answer: [Yes]
Justification:
We strictly adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Quality analysis of weather forecasts does not have direct negative social
impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our method is for quality assessment of weather forecasts, and our dataset is constructed based on publicly available datasets, thus does not have such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all original papers and make sure that our usage is legal.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pac re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of how we construct our datasets based on public datasets are stated in Sec. 3 and the Appendix.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We applied human annotation when constructing our dataset, as detailed in Fig. 4 and the Appendix. We provided generous compensation human annotators.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our annotation process was thoroughly explained to annotators in advance. With their informed consent and approval from relevant organizations, proceeded with the annotation process.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethies and the guidelines for their institution.
20

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLMs for data annotation and experimental evaluation, described in Sec. 3 and Sec. 5, which is conventional in the field MLLM-based quality assessment.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer LLM policy (https: htt////eerrppppppeee htt//Coofeeeecceeeeeeeeeeeeeeeee 2225//2222222 what should or should not described.
/LLM) for
21

---

Appendix
A Overview
This Appendix is structured follows. Dataset details are described in Appendix B. More ablation studies, qualitative and quantitative results are presented in Appendix
B Dataset Details
B.1 Details of Scientific Attribute Library
To facilitate dataset construction, we design scientific attribute library grounded in physical principles. This library comprises 5 super-categories and 10 sub-categories, comprising 35 attributes. Combined with the overall performance of the predictions at both the frame and sequence levels, these constitute a total of 37 key attributes used for dataset construction. The definitions of the 35 attributes in scientific attribute library are provided in detail below.
Intensity.
Miss. (a) Miss Performance. The proportion of regions with observed precipitation in the ground truth that are incorrectly predicted as "sunny" in the forecast. (b) Raw Rainfall Level. The rainfall levels the ground truth for regions where rainfall missed in the prediction. (c) Miss Rainfall Level. The rainfall levels in the prediction for regions where rainfall is missed. (d) Miss Direction. The directions the prediction which specific rainfall levels that are missed in the prediction.
FAR. (a) FAR Performance. The proportion of regions labeled as "sunny" in the ground truth but incorrectly predicted with precipitation. (b) Raw Rainfall Level. The rainfall levels in the ground truth for regions where rainfall is falsely alarmed. (c) FAR Rainfall Level. The rainfall levels in the prediction for regions where rainfall is falsely alarmed. (d) FAR Direction. The directions in the prediction in which specific talse-alarm rainfall levels that appear in the prediction.
High Value Construction. (a) High Value Retain Performance. The ability of the prediction to consistently preserve high-value regions. (b) High Value Mismatch Type (Sequence). The type of mismatch in regions with high values (i.e., precipitation at "intense" " level or above) across the prediction and ground truth sequence. (c) High Value Mismatch Direction (Sequence). The directions in which high-value regions were mismatched. (d) High Value Mismatch Performance. The ability of the prediction to predict intense precipitation levels. (e) High Value Mismatch Type (Frame). The type of mismatch in regions with high values (i.e., precipitation at "intense" level above) across the prediction and ground truth frame. (f) High Value Mismatch Direction (Frame). The directions in which high-value regions were mismatched. (g) Max Rainfall Level. The maximum precipitation level in the observation.
Precipitation Conservation.
Cumulate Precipitation. (a) Cumulate Precipitation Performance. The degree to which the cumulative precipitation predicted over the entire sequence aligns with the ground truth. (b) Cumulate Precipitation Difference. Differences between the total precipitation of the prediction and the ground truth across the sequence, indicating whether the forecast overestimates or underestimates cumulative rainfall. (c) Mismatch Direction. The directions in which the prediction fails to reconstruct the cumulative precipitation accurately.
Precipitation Dynamic Distribution.
Morphogenesis. (a) Shape Change. The change in the shape of the convective system over time the ground truth. (b) Scale Change. The change in the spatial area of the convective system across frames in the ground truth. (c) Convective Cell Change. The change in the number of convective cells. (d) Intensity Change. The change in the precipitation intensity over time. (e) Dynamic Consistency Performance. The overall consistency of dynamic evolution between the prediction and the ground truth.
Trajectory. (a) Move Direction. The primary direction of movement of the convective system in the ground truth. (b) Speed Difference. The difference in the movement speed of the convective
22

---

Table A1: Characteristics of each attribute terms of level (frame sequence), reference type (caption / comparison), annotation method (human / automation), and usage purpose (rating / assessment).
Level
Reference
Annotation
Usago
Attributes
Prame
Sequence
Caption
Comparison
Human
Automation
Rating
Assessment
Miss Performance
Raw Rainfall Level for Miss Miss Rainfall Level
Direction
FAR Performance
Raw Rainfall Level for FAR
FAR Rainfall Level
Direction
High Value Retain Performance
High Value Mismatch Type (sequence)
High Value Mismatchh Direction (xequence)
High Value Mismatch Performance
High Value Mismatch Type (Frame)
High Value Mismatch Direction (Frame)
Rainfall
Cumulate Precipitation Performance
Cumulate Precipitation Difference
Mismatch Direction
Shape Change
Scale Change
Convective Change
Inteusity Change
Dynamic Consistency Performance
Move Direction
Speed Difference
Rotation Center
Difference in Generation
Difference Dissipation
Sharpness Performance
Shape Type
0
Shape Mismatch Direction
Shape
Reason
Artifacts Diroction
Organization Degree
Distribution
Overall Perforaance (Sequence)
Overall Performunce (Frame)
system between the prediction and the ground truth. (c) Rotation Center. The spatial location that acts as the center of rotation for convective system evolution.
Convective Cycle.
Genesis. (a) Difference in Generation. The difference in the number of newly generated convective cells between the prediction and the ground truth over the entire sequence.
Dissipation. (a) Difference in Dissipation. The difference in the number of dissipated convective cells between the prediction and the ground truth throughout the sequence.
Morphology.
Sharpness. (a) Sharpness Performance. The degree of similarity between the fine-grained contours the prediction and those in the ground truth.
Shape. (b) Shape Type. The morphological pattern of the convective system in the observation. (c) Shape Mismatch Direction. The directions in which the evolution trend of the convective shape in the prediction diverges from that in the ground truth. (d) Shape Mismatch Reason. The underlying cause contributing to the mismatch in convective morphology between the prediction and observation. (e) Artifacts Direction. The directions which artificial patterns appear in the predicted sequence that do not exist in the observation. (f) Organization Degree. The temporal trend of structural organization in the ground truth reflects how orderly the convective system over time. (g) Distribution. The directional distribution of precipitation in the observation
An overview of the properties associated with each attribute is demonstrated in Tab. A..
23

---

+5min
+15min
+25min
+35min
+45min
+55min
Target
Cascast
Diffcast
Nowcastnet
PredRNN
Simvp
DGMR
Figure Al: set of example forecasts on SEVIR.
B.2 Details of Raw Data Statistics
To ensure the diversity of samples in RawRQA-20K, consider both wide range of storm event types and a diverse set of generative models. First, our RawRQA-20K covers seven storm event types, including flash flood, flood, funnel cloud, hail, heavy rain, thunderstorm wind, and tornado. Due to their strong convective nature and high impact, these storm events pose significant challenges for forecasting and contribute to a diverse sample space. The number of samples for each event type is summarized in Tab. A2, The coverage area of storm events is shown in Fig. A2.
We employ a total of seven representative nowcasting models to generate prediction samples. As illustrated in Fig. A1, these models produce diverse samples that reflect a wide range of forecast qualities. For example, Cascast tends to over ver-predict in high-value regions, yet generally exhibits superior performance detail reconstruction and dynamic consistency. contrast, DGMR often introduces substantial artifacts, which significantly degrade the overall quality. Meanwhile, PredRNN suffers from severe temporal blurring and exhibits poor performance in "high value retain".. These varied quality issues reflected in the corresponding differences across the assessment reports.

---

Table A2: Statisties of RawRQQ-20K,
Event type
Flash flood
Flood
Funnel cloud
Hail
Heavy min
Thunderstorm wind
Toraddo
# of events
218
121
58
556
55
1030
121
B.3 Details of Human Annotation Questionnaire
For the human-annotated attributes listed in Tab. A1, we employed an annotation pipeline to ensure consistency and quality. First, for each attribute, we designed a corresponding multiple-choice question, with domain experts defining clear annotation guidelines. Second, a small set of pilot samples used to evaluate annotation quality from several annotation companies. The company with the most accurate performance was selected for large-scale annotation. Third, all annotators underwent standardized training align their understanding with expert standards. Each annotator completed a trial anmotation set, which was reviewed by experts who provided feedback and corrected any misinterpretatioos. Fourth, upon completion of each annotation batch, cross- cross-validation step conducted by different annotators to ensure quality. Finally, after annotation, domain experts performed quality control by randomly sampling and reviewing 35% of the samples in each batch. A batch would be accepted only if the sampled annotations met the quality standards; otherwise, the annotators required re-annotate the entire batch.
B.4 Automated Generation
As shown in Tab. A1, 20 attributes are grounded in score-based metrics, where automated annotation provides more precise and consistent results compared manual labeling. this process, all the required thresholds or parameters are determined with the assistance of domain experts. The corresponding computation procedures for these attributes are detailed below.
False Alarm Performance. First, caleulate the false alarm rate. Let & and P denote the sets of pixels with precipitation in the ground truth and the prediction, respectively. Define Hits as H | NPI and False Alarms as (F IP \ Gl. The false alarm rate is given by:
F
false alarm rate
(A1)
H +
Thresholds [0.1, 0.2, 0.3] are selected to categorize the false alarm rate into four performance "rreat""Geeat", "Good", "Fair", "Poor").
Miss Performance. Similar to the false alarm rate, we compute the miss rate based on the binary masks. Following SEVIR, we define pixel as having precipitation if its value exceeds 16. Let G and P denote the sets of pixels with precipitation in the ground truth and prediction. Define Hits H PPP) and Misses M Pl. The miss rate is defined as:
M
miss rate
(A2)
H H+M M
Thresholds(O.1, 0.2, 0.4] are used categorize the miss rate into four performance levels. Sharpness Performance. Following SRVIT, we evaluate the sharpness of the prediction and the ground truth using the Sobel filter. Specifically, Syt and Spred denote the mean Sobel value of the ground truth and the prediction, respectively:
1
Syt N Σ Sobel(gt); Spred N
Σ t=1
Sobel(
(A3)
We then compute the relative difference:
Sped Sye
if
Sped Sg
d
(A4)
Syed
otherwise
Finally, we clip negative values zero, and define the sharpness score as:
sharpness score = max d)
(AS)
Thresholds [0.5, 0.7, 0.9] used to categorize the sharpness into four levels.
25

---

convect tive cell convective system
conve
precipi ta ietiion on eeve
coservation Anage
prediction
dynamic consistenc
00
fa se al alarm
was
uated sequence
evaluated i
image
Light precip pa
pprrcippii Decipiiation tattionto
area
Trepor
miss rate
mellate
hhhhh Value
aarm
Figure A2: Coverage area of selected storm events in our RQA- Figure A3: Wordcloud map of 70K dataset, which spans across the CONUS region.
our introduced RQA-70K dataset.
Annotation Questionnaire
Caption
1. What is the moving direction of the convective system?
4. What the convective system?
the
7. What is the shape convective system?
A
B.
D.
A.
B.
C.
A. scattered
F. multi-buoe caalti-bloc--iii like
E
P.
G
H
F.K
G.
H.
banded
multi-ur-sshie shuped
L.
J.C
center
no rotation
C. block-like
H. multi banded
How does number convective cells change? incrense
2.
does the coverage of convective system change?
5.
E. are shaped
large patch-like
J. Trregular shaped
I. spiral shaped
remain the same
incraase
C. remain the same
decrease
decrease
8. How does the shope of convective system change?
3. How does the intensity of convective system change? increase
6. How does organization degree convective system change?
merge
split
B. stretch
disappear
remain same
A. increase remainn the same
shrink
form
B. decrease
decrease
D). dilate
H. remain the
Comparison
In which directions are the diff. in shope change most severe?
1.
3.
directions
5. dissipated
artifacts?
cell the prediction is
A.
B.
C.
A. larger smaller
C. basically the same
E.
F.
G.
H.
F.
G.
H.
same
center
J. remaine the same
2. What main issue within the direct. with most diff. shope change? A. scalo diff. C. position diff. B. diff. of couvective cell numbers
4.
The scale of generated convective the prediction
6.
The movement speed of the convective cycle prediction
A. larger
C. busically the same
A. faster
C. basically the same
B. smaller
B. slower
Rating
1. What is the everall performance of the predicted sequence? great fair
2. What is the dynamic consistency
3. What overall performance
performance of the predicted sequence?
the predicted image?
B. good
great B. good
A. great fair
good
pour
C. fair
D.. poor
poor
Figure A4: Human annotation questionnaire for the 17 attributes that require manual labeling.
High Value Mismatch Performance. We first count the number of high-value pixels in both the prediction and the ground truth(i.e., pixels with intensity values greater than 219), denoted as Npred and N respectively. The relative error is computed as:
N gt
N Spred
Evel
(A6)
N
26

---

Table A3: Structure of detailed descriptions for each general attribute.
General Attriuutes
Detailed Deseription
High Value Mismatch
In the high value mismatch direction, the prediction is high volue mismatch type (over-predict under-predict). In the Miss direction, the row nainfall level is misclassified as miss rainfall level. the mismatch direction, the cumulate precipitation is cumuulate precipitation difference. In the high value mismatch direction, the prediction is high value mismarch rype (over eever-predict// / under predirt)
Miss
Cumulate Precipitation
High Value Rotain
The high value mismatch score is subsequently defined as
high value mismatch score min(1, max - E))
(A7)
Thresholds [0.3, 0.6, 0.8] used to categorize the high value mismatch into four levels.
High Value Retain Performance. The high-value retain score is computed as the average high- value mismatch score across all frames. The same thresholds [0.3, 0.6, 0.8] are used to categorize the performance into four levels.
Cumulate Precipitation Performance. First, we compute the total precipitation in the prediction and ground truth, denoted as Ppred and Pgt, respectively. We then calculate the relative precipitation error and define the cumulate precipitation score using the same method as in the computation of Enel and the high-value mismatch score. Thresholds [0.93, 0.97, 0.99] are applied to categorize the performance levels.
To provide a detailed characterization of the general attributes, we divide each image into a 3 3 grid, resulting in nine spatial regions corresponding to nine directional sectors. For each general attribute, its detailed description is formulated as combination of directional information and the associated prediction issue. For example, in the case of false alarms, typical description takes the form of "in the FAR direction, the raw rainfall level is false alarmed as the FAR rainfall level." This expression involves three distinct attributes, whose construction is detailed below.
Raw Rainfall Level. First, we compute the number of missed pixels for each rainfall intensity level. To incorporate the varying importance of different rainfall levels, we align with domain experts and assign weights [1, 2.5, 5, 10, 20], corresponding to increasing rainfall intensity from "light" to "extreme". Higher rainfall levels are given greater emphasis. We then compute the weighted sum of missed pixels for each level, ranking them descending order, and identify the rainfall level with the highest weighted missing pixel count.
Figure A5: Gridding of the image into 3 x patches, each representing directional sector.
FAR Rainfall Level. For each raw rainfall level, we examine the corresponding locations in the prediction and count the occurrences of each predicted rainfall level. The rainfall level with the highest pixel count that lighter than the raw rainfall level is selected as the FAR rainfall level.
FAR Direction. For each raw rainfall level, we compute the false alarm rate across different directions. We also count the number of pixels with raw rainfall level in each direction. To ensure both a high false alarm rate and a large false alarm area, We sort the directions by false alarm rate in descending order, and restrict our selection those whose raw rainfall level pixel counts are among the top two. The first direction satisfying this condition is selected as the FAR direction.
For other general attributes, the structure of their detailed descriptions is summarized in Tab. A3, and the construction of their underlying attributes follows a similar procedure as in FAR.
Bias from the usage of LLM. We use GPT-4o to organize annotated attributes into assessment descriptions, which may introduce potential bias, including:
Style bias. The structure of the reports may be overly uniform and fail to reflect expert diversity. Accuracy bias. The generated content does not always align with the visual information.d Redundacy bias. The presence of unnecessary information may reduce clarity. Attribute Omission bias. Less prominent yet important features may overlooked.
27

---

C More Results
Table A4: Few-shot Results on general attributes for the frame rating and frame assessment tasks. Accuracy is used the metric for the frame rating task. RadarQA surpasses all methods.
Methods
Rating
Frame Assessment
Overall
False Alarm
Miss
High Value Sharpness
BLEU
BERTScore ROUGE-L METEOR GPT4Score
GPT40
43.72
26.40
29.65
27.09
49.19
0.164
0.782
0.448
0.372
5.33
Claude3.7 sonnet
37.79
35.00
25.00
45.58
0.136
0.371
shot
Gemini2.5 pro
30.70
29.88
30.93
34,07
42.44
0.102
0,748
0.368
0.355
6.01
GPT4o
52.79
32.67
33,60
29.65
52.21
0.167
0.787
0.456
0,383
5.31
Three
Claude3.7 sonnct
28.49
32.09
13.60
24.53
26.98
0.158
0.786
0.440
0.389
5.11
shot
Gemini2.5 peo
33.72
32.32
35.81
44.41
0.140
0.364
Ours
RadarQA
61.51
65.35
67.67
69.19
78.60
0.213
0.809
0.512
0.420
6.87
Table A5:
More results on ablation studies of multi-stage training strategy on rating tasks.
Stago-1
Stage-2
Stago-3
Frame
Soquence
High Value Mismatch
Dynamic Consistency
Cumulate Precipitation
High Value
Overall
False Alarm
Miss
Sharpness
Overall
Retain
x
x
X
20.10
36.40
30.00
16.51
35.93
7,99
16,10
17.49
23.22
x
x
60.93
63.37
61.63
63.02
71.28
61.42
42.44
42,20
74.53
/
x
59.77
68.14
67.67
65.00
74.19
61.55
64.17
42.82
77.78
/
x
/
61.28
65.00
66.40
69.88
78.14
65.42
52.31
49.44
81.52
67.67
66.17
53.31
80.52
Table A6: More results on ablation studies of multi-stage training strategy assessment tasks.
Stage:1 I Stage 2 Stage.3 3
Frame
Sequence
BLEU BERTScore ROUGE L METBOR GPT-4 Score BLEU BERTSSore ROUGE L METEOR GPT-4 Score
x
x
0.122
0.75
0.389
0.332
3.81
0.09
0.745
0.281
0,342
3.92
0.195
0.498
0.417
0.212
x
0.212
0.810
0.511
0.423
6.83
0.211
0.816
0.431
0.461
6.56
0.213
0.809
0.512
0.420
6.87
0,212
0.815
0,436
0.461
6.58
Table A7: Comparison with traditional weather analysis
Table A8: Ablation studies of different
and general IQA methods on frame rating task. The threshold used for weather-related metrics is 74.
model sizes. Frame sequence rating tasks are evaluated in average accuracy, while frame / sequence assessment tasks assessed in GPT-4 Score.
Methods
Weather ralated metrics
IQA methods
Ours
CSI
POD
FAR
Bias
ACC
ETS
DISTS
LPIPS
RadurQA
Model size
Rating
Assessment
Accuracy
41.74
39.07
39.42
53.60
46.63
61.51
SRCC
0,26
0.28
0.15
0.23
0,21
0.29
0.55
0.39
0.62
3B
63.44 / 59.71
6.77 6.36
PLCC
0.27
0.16
0.20
0.29
0.56
0.43
0.64
68.46
6.57
Few-shot evaluation frame rating task and frame assessment task. We further evaluate the performance of different API-based models. As shown in Tab. A4, although other models are evaluated under few-shot settings, RadarQA consistently outperform all baselines without requiring any additional examples, demonstrating the effectiveness of RadarQA
Ablation studies multi-stage training strategy. For our multi-stage training strategy, we further examine the effectiveness of each stage across different metrics, as shown in Tab. A5 and Tab. A6. First, applying reinforcement learning significantly improves performance metrics such as false alarm and miss rates. After supervised fine-tuning, the model leverages its ability on interpreting learned from assessment tasks better rate general attributes. Finally, the full training strategy achieves the best performance on most metrics.
reasoning-relates
Comparison with domain-specite baselines. We compare RadarQA with weather-related metrics and general IQA methods. We use accuracy, PLCC, and SRCC as the evaluation metries, which reflect the consistency between the evaluation results and the expert annotations. As shown in Tab. A3, RadarQA significantly outperforms the basclines across all three metrics.
Ablation studies on different model sizes. We further evaluate the performance of different model sizes under the same training strategy using Qwen-2.5-VI. series. As shown in Tab. A8, the 3B model shows a slight drop in performance while using fewer parameters.
Qualitative results. More qualitative results of assessment tasks shown in Fig, A6 and Fig. A7.
28

---

tke obarraion soqasee, da conrecio goass ia caate and un incremin skapc ind dliates whil in dearce organizaiion deccno,, and cenerige Regorting the and rapuice Byranve Coee ceecerecy perforrncce in für ss the yoorred speed sligis with aherratione or dicrepaccec dupa change we noot praseenon wisk:
sytee The dllatii while of organiaationn decross Addisonally bleek-lkke conerage incraaure
soperze
oggath.
Segariin Dee qualit ansoussst of la lecasll sasenss Dynasic Caniaaao paróonaacce in sgrol muoeersst cligas with desraaiisa desigaasio mi dage charge.
are detetvel Candlan Prokppiadie pa Ierrance gresi escull il Bough inderprediiiion socrs the serth, nerthorol regiven High Yelao Roxin perforacced ee degool, wifl earry high rufse aoas setanet bewecver, apacticun aranche occer in the warl, where everall hightrates
stether
pursealeety sbunrvnenn, viite disipatie calls maintaas sinilar ccalis No srtifam in the poolited sopares. Canalar lat Poocp pefforaace is sift sigsifient under- prediction of prodpiattr sem in sorfeer and pereett regines High Valao Roasn ya Semenss some sab-oabe reginna sevennet bewecerr everal prodictern hish
eanenary. evaknted woathrr deccurirng sugaeee demeroinato tua dhramec coemuracyy
gins convilato sowipiaatee coorney with miner reginaa under prefiutions desp sceabee
the esaicttd
deneentonn dyruer
het enegght with
avorankes in the west While model perform will is prnr plocioo prodiction and mells high-rnhac reaiees effactieds, sech anacad chapes cenvective olla repional lnghiide inpervenant
ocorantly piodeiing comaltho protytatiie andinniingg hightolh segiee Whinecrsie uuocc slgg wilh obecred pettarna, tetatis dkfiicndes is protptation pookeise and sade eppccestiiio dacediabille fncond, coscialy inited uce toath.
Trals
Prodicion
eegurtos,
abormates eupeess,
wis mollipl locck-lkee
number conractire w ther bntmnity stable The scattrnd and
stradaros diliing uver line while lecuning los orgasiend The covengo inceno wcl.;
crnbbb mulliph Heck Aesttraaarr,, olidi dibicoss line The degree of oforgiiizaiion daross, and ls: cnver age range experdls
Regorting the qualit soserrannt of the eollard soparos. Dytume Uzeestrayy partimmanse falr, indenting mederas siggerant with obeerved dyrames The spee mocererr rrashoc the Resesstiven
Regasdin decovlaate sopuees quilly ous becveedd coro eapech Dynani Coeestzayy razches obearatiea, docepsacas purticbarly is seathwas dee to scal diffirenco. Nenty terned convective culo are snalier
misero dieropeacies
charges
conseetive
Prepptztonn
perforrance tair
duurges,
Cammaas Prodpiatan perkerraze
with supalicaal darddliiii
procipitation usme in the noeth and nemer oreas High Vake Retin perforracee is falr, wers
whdee dosiuaiin oae are larger compared to absens sliuss thrugh wrtifuut prrseet gool, with mayy Vitt
hightiolan regiinn weabbot beuavvrr ovenal prodictions high valoss snder-prdietsdd
High Reaiss perfer
partio perticalrrly eanter. xanenay.
Docipiazion pe pemaanc
the model under-gredice
noi and casl.
prodicing coral precpiitaion and intaising high-raise regions, Kry
la sunsuay, the evdlaae soqacics comsootsio firr dysassi combicacy gord rxteation of high-sakia reglnss dispto sutable micuarcee sepere saie pour
tncitad pnndetpredcction of praciptuttin cime inlomarched aigh calie reckens and
dicespendic cell for ferastioon and disipotion scalo. These linikaiion lightigh caceoouiingg Reprosenntt for erinnoal firocating souraey.
prediitation highight deficiaclee prev
Goond Trch
Grann Truil
Predktka
Prodedin
seqaene,
the rhurottic of connectiv celik that tniredh our dms The stope scamnned with malitple blado the str strecterm, anderping dikaieo slile dagrec of orgasiatiion dcecesss Adliionlly dl zange coscagge experds fronghos fessepaac. Dynanic
connectiv oyter mover sxinard with Incruding number
number and lasuadly cnnvective calls.
ccatiered with avatippl ticckllkk
strecteres roneiiing misetlllly lenged while the degres of orgasinsiio decreases. Adliicelly the raage of coverage werake stable froughod.
Tynunio Consianwcy pereensso gool alim
sopurare
Whie dee sped proseenced
reldes well, suuibrett das lo changes
dacrepasio
docoppacies stape
where chanpeo mumber of connective cilh deviate frees obsevatica. Nenty loemed
Newts berned cenvective cuth are smalee oxtparad to oberesices though di-ippting oil maistas shellar wrales aniifum peurs in Do prodetsi uquance Reyondigg Clre procppiitiin
oxns-etto vctiee colle smaller in acalc cespased to sherwal scusende visle disipating eib
malerals skeillr scales Chrreiina Powipyaatee The per Serrraaaee pour mnder- performance cvmoall Nigh-stee prodicivnn cnseruttnnteed with witemotchee sccunitgg the
Prec plopiiionn perforracee gand, nuttonet
rndepprndirion occars
regitss For High Valas Reahs
for performance fair, with parial relations of ligh---a cajens. Nuable ovarll high-rdac
Iesanerary, decovalate aoqueao danceatraio streng dysmaii combibancy bui struples with securalc proc pnceppibaiis pr cdliiiess and lish- rall- releation. White the anmonest and strectural dyeaniic well- egrocotte. high- reqerigg prceutianzs and repin
prodetsen and underppniiln ciol with vigsilicat sthanaidio cencentratil is the west.
Is lasaaayy, cralatal
lnccmiig sepeanc demenatat dheamtic conainency. hig
ender-prdiete precistaaton and quiry
geod cxerabue procptation
aepacts
aerable include prodictio proeppitatinn high-nitee regions, well ac dierrpanntec cill scale and shape changes in queciti aemas.
Figure A6: Qualitative results on sequence assessment task.
29

---

dee alaccaato leigs, proipaaee ane soual suil wit sasiern
endplaaioeen reruuity conotrooiin Bhe waah righ eo The saps the propeiase acus weathes rigite.
loot
proiptttion livel muiinng cntree rals The au sillaioo qui lo ares
tecopprred scroe space tamnnn imggaa clatcn peiciso vay
Theaultt ancuret af sags in ch achrrd Ecfiiesioo For CRASTEE ligallient eshors eree far right
The quily ancuencc the ovdeaed image rovell en ane seputt, For cata oux
where
eesiines and makkiais and prtstii cussi koding loncalic dicppici
are sadarciporad Stadarty,
prodiaation
sit la wiel lnve pred rices of light pros ppttttioon the wone whick are fenereilty
pataremec i poi dac vegocceeeen proditen prong
Ground Truth
reperted in bigher bonil sec mudoetic propptutis shappocs petferuace falr,
indrostig pertiaal conistrccy with the oleee
thrugh ditch
Ground Truth
cooileney, eng the pronuee of upuutal reproceett High velooeas kay
cuttern mall captamss ligh vake earding paffrrace
por waned hes
the
westinee
Ovendi du and mage sinvestoon daatty lor unaleity vid obarstan
Bieeskoos The
and
eslccad
nac. fibcralere rat, shipeeos high valae resaiieg the mas
tennn me loods and tice apeat auareut Andtuerity tic
dermeggntiont retecpmmciin rec suiiity whic co acome
bead ker sieibuitt with the stsuratiee tmrggs liaiiing to clfechennos eecorategial
teer demb
Thes contined
Prediction
mo secrdlli
loaking unility prrcic
ofservatee reag: proppeinue ace alargr sokk restneer poodptation righos shongatd and Inipular
Ce cherreday zage the prodiputio raall
rasilaas The propptaaoo
prodipaatio losd
pokaiily aenen
encomnndd in the car and onner rights, faminn imopater chascrs whllchegpoe and
The by aessoreid the andlaned leage roch uvra aiject, For rass are Doansty
of ritocestaan
souend
patarmenc escril
moee
wiik modernic
light
Repadlin
prodphsiiin outter endemoppnnn
Light dipiiattin enchengprhed maakini the weats and very hiary
dle fair with oor coparting of light predeptation rais
cois the
Ground
ruses lnd ciecppatio dicoaano with ee etecredun mmgg wil nove deids calc accuradyy
prodiptatinn
Ground
50
cokk onl High panleageeli retereee giott
valaas periucatty varty the methirn rodas whe anachee joutace.
aeeatly pew wil howwerr lelee gewiii sundoe the
hide copucialy camplingg
Owell,
alors
which and aoe edalkiir Koo oalig Whike of mntene an on
Protiction
dixiaato batoon de endkate image and abeerateo, High giiigg ausi fer
repoweeee soeecy..
podeladion fige aepact be
Prodiston
Be abservatie tmags, puodplatter ane a median is saak, vith the musinnn danan party
clooriio
the sedl
peoplia
proplation not
muppariii and fegeeied depps The wiee wel edlii wrrinng wine vagge eg fiom
dernng Theae skes ehhir clin and and sesterd pettonss perticlert mene alrdeed
where comeec man seatl chatering Sut iggillcat sdace
The ip ansroncc the aodlaadd koapo rovill un una arpccs. For rata ou
Formia proeig per
pereemence gend everll wes aepaicent miseen
inllutation patforuao
lovd sade-
osporada laves anagorde Fale
Sik
Fehe porfemmmen go oomel however dereeee liee
motemm
Trush
for light provlp lis the southe aver expur tul as underato sippliiie.
Chappnoe pertommane grost extriing
reckrate rite She perkir gaoe, dineestrate olipttt high
and light prodpiatios in rephie wiich
Grund Truh
and accemiily repaccaa High wae
vur cesrt volas seginss
cady repessrnad High sile curdling periecaancc & poo, as lage pf
highorac regira cornctty pro though underggrattcte erides
Onecll the an anadd iaage deneessioes poo quiliy, wilk digbes high duilldyy tn
by imayy stracheall necaracy diurepus sellddy ligat
high
Ovem thcersta reae domononae macy but porformane dee light dhooopeecco
ericatl prociptatise
These collotats pertommens maling retabii for redvandogia tnocaringg bet argutta retacmeca
farur vato
opcuntty and pabane er do weg.ticcttyyyyyy........ impart
algeeee with the dlen cation lnsig the sale vato and tiko keaan,
Prediction
per
segion
Thes
Prodiction
dieecatio Eunge the preke is aiall lociale wat suciicen eir speieel wase th gaiir ag dhole wil eieggnnird outtrrl mebll
Is descraatee tmage the procpeaaaon our anc rectamm sail with curree
cxncatrraid mot dightt tavard the purtirent The tf with dongaee dharar carterd
guily for mvrie bnoger in westmotten light
pactetalm uaca calito trea
horr
queaullyy uncorten chnhaat amage to caa acuund spects For matr wilk
encipttatie socally mileed in the mestimest whrn larule add as
wartennt and proiplattio
lnness poputtiidy Ragurting Eabe
Groand Trash
Lipe lo ripartad modia rais musi Shap and
Palee Alara Roac: pororrane, dhglgh
alrne
la vos vhick excreppenad in herry and light prochaluaion in earth.
Groand Truih
deal and poetna dis Duly seall portaane doalk and bocadia alga
digre of meetieer
atee, with mon deaa eeeten noed high-valas
ercenets trigh meicinn pervormnne ae
mith pe
acusad nady repoorrd High vala custing
revl
proppitaiio
Ovent, deceekaanii perkaaniregg mge dveem eacy gord perfemance dl
The ligheaae use fo inaee andlig light and bakie codklad wil die
bost porthits encrall proppiae sppe undonmnepptenn comppppttnggg pollase rovorably soomss tegro encm cooke ker pracie repoocecction ot posipiitts
Witk and
kook inteopppttn more
Predccion
you sattoollgg sill koncasing
hightslle adong furter
unvalinic
addiall Thes
Prediction
Figure A7: Qualitative results on frame assessment task.
30

---

Table A9: Question pool of rating task..
Question
1 Could you score the prediction besed on S(diml) S[dim2), S(dim3), and Si dim44,, and then provide overall performance lovel?
2 Please usign levels to the prediction based on the four dimensions: ((dim]] J. ([dim2), S1dim3), and S(dimn)) and give an overaall performance level.
3 How would you score the quality of the prediction on the dimensions of ((diml), J. S S1dim2), S(dim3), ), and Sidim4), and what would the overall level be?
4
Can you score the prediction using the four criteria: S [diim)) J. S(dim2), S(dim3), and 5 [[dim]] and then provide an overall level? Could you evaluate and score the prodiction using [diml) J. S(dim2). S(dim3), ), and S/dim4), J. then provide a final overall performance level?
5
6 How would you score the prediction across dimensions of ((diml) ). S(dim2), ). S(dim3), and 5 (diim4), and what would be the overall score?
7 Please score the prediction based on S [ddiml] J. S (.dim2). $1 (ddim3), and Si dimd], then provide the overall performunce level. Could you prediction?
prediction SJ (dim J. S[dim2), $1 [dimm), and SJ S(dim4), and then give an overall evaluation score the
9. How would you rate the prodiction across four dimensions, S[dim]]) $(dim2), S(dim3), and St S/dim4), and what is the overall performance level?
How would you rate the prediction on the four dimensions, 5(diml) ). S(dim2), ). Sidim3), ), and (diim4) ), and provide an overalll performance level!
Table A10: Question pool of assessment task.
Question
Please start describing the content the observation, and then evaluate quality of the prediction based S(diml) ), S[dim2), S/dim3), and 5 (dim4)) Provide comprehensive quality assessment report based on the subtasks with a summary. How would you describe the observation? Following that, could you evaluate the quality of the prediction across $[dim1). ( $(dim2). S|dim3), and S ((imm4) then give summaryy?
Provide detailed quality report of the prediction. First describe the content of the observation, then focus on {[dim]), (dim2)) SI S[dim3), ), and $1 ((dim4) performance of the prediction.
Could you describe the olservation's content, then aseess the quality prediction accurding ((dmm)) J. S(dim2), S(dim3), and S S|dim4] J in the format of a detailed report with summary? Give a report of the prediction. First describe the content of the observation, then focus on $[dim]). J. ((di22), S(dim3) J. and S [dim44] prediction. Finally, summarize your analysis.
Please descrihe the observation's content. Then, how would
the quality of the prediction hased S[diml ((dim)),
SI (,dmm)), and S( ((dim4)? ? Give a detailed report with a summary. What is your description of the observation? Aferward, could you evaluate the quality of the prediction on S/diml ), S| S[dimm)) S/dim3), and (dimm))? Please provide detailed report with summary.
Start by describing the content of the observation, then assess the prediction S(diml) ). S(dim2), S(dim3), and (dim4). Provide detailed report with summary.
How would you describe the content of the observation? Then, would you evaluate quality the prodiction on $(diml) ),
S(dim2). S (ddim3), ). and Sidim4), and summarize your findings? Give a detailed report with summary. What content description would you give for the observation? Then, how would you evaluate the quality of the prediction across S[din]], J. S (dimm)) (dim3), ). and diim4]? Provide detailed final report summary.
10
31


Reviews:
[{"review_id": "lDqAwclxjA", "reviewer": "Reviewer_78LW", "summary": "This work proposes a new benchmark for multi-modal quality analysis in weather radar forecasting, supporting both frame-wise and sequence-wise evaluation tasks. It introduces a hybrid annotation pipeline that combines large language models (LLMs) and human annotators. The authors also train a model to analyze forecasting performance based on this benchmark.", "questions": "See weaknesses", "limitations": "The proposed method appears to be more focused on dataset construction for weather forecasting rather than introducing a fundamentally new modeling approach.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a thorough analysis of the collected dataset, which appears well-suited for the weather forecasting task.\n\n2. It explores a diverse set of question types, using various input modalities—including text, single images, and image sequences—to support a range of question-answering tasks.\n\nWeaknesses:\n\n1. The technical contributions beyond dataset collection appear limited.\n\n2. Regarding the model selection, it is unclear whether different types or sizes of foundation models were explored. This makes it difficult to assess the effectiveness and advantages of the proposed training scheme", "was_revised": true, "final_justification": "The rebuttal satisfactorily addressed my concerns about this paper. However, I do not fully understand the background of the setting. I will therefore maintain my rating of borderline accept.", "rebuttal": "We sincerely thank Reviewer 78LW and the area chair for their efforts and appreciation of our comprehensive dataset analysis for the weather forecasting task, the diversity of question types, and input modalities. We will update the manuscript as suggested. Below, we address the reviewer's main concerns point by point.\n\n\n## Q1：Contributions beyond dataset collection. \n\nIn fact, **the lack of a large-scale, high-quality dataset is the main gap in the field of quality evaluation for nowcasting tasks**. Recently, many MLLM-based assessment methods, such as Q-Instruct[1], Co-Instruct[2], and DepictQA[3], have also emphasized the importance of dataset construction for quality evaluation. In the field of meteorology, existing efforts have largely focused on developing new numerical metrics, while neglecting a quality understanding and perception grounded in expert evaluation practices. In contrast, our proposed dataset, RQA-70K, addresses this gap by incorporating both frame-level and sequence-level data modalities and supporting both rating-based and assessment-based evaluation tasks, thereby bridging numerical evaluation metrics and expert-driven analysis approaches.\n\nBesides the collected dataset, our contributions mainly consist of three aspects:\n1. **A novel quality evaluation paradigm for meteorology.** By integrating the evaluation steps of meteorological experts and existing metric-based quality assessment methods, we propose four quality analysis tasks: *frame rating, frame assessment, sequence rating, and sequence assessment*. These tasks cover both frame-level and sequence-level modalities, as well as rating-based and assessment-based evaluation needs.\n2. **A well-designed data construction pipeline**. To construct our dataset, we begin by selecting seven classical nowcasting models based on various backbones to generate the RawRQA-20K dataset by using storm events from SEVIR. Second, in collaboration with domain experts, we establish a *scientific attribute library* containing a total of 37 attributes. Based on the characteristics of these attributes, we adopt a hybrid labeling strategy that combines automated and manual annotation on RawRQA-20K. Finally, we carefully design instruction prompts to guide GPT-4o in generating data for *assessment tasks* based on these attributes, and we use scripts to construct JSON-formatted data for *rating tasks*, ultimately resulting in the RQA-70K dataset.\n3. **A multi-stage training pipeline.** To train a high-performance quality evaluation model, we first employ supervised fine-tuning (SFT) to equip the model with foundational capabilities. Then, we adopt GRPO along with two carefully designed reward functions on the two rating tasks. Finally, we conduct an additional round of SFT on a small subset of RQA-70K with a lower LoRA rank to further enhance the model's performance.\n\n[1] Wu H, et al. Q-Instruct: Improving low-level visual abilities for multi-modality foundation models, CVPR 2024.\n\n[2] Wu H, et al. Towards open-ended visual quality comparison, ECCV 2024.\n\n[3] You Z, et al. Depicting beyond scores: Advancing image quality assessment through multi-modal language models, ECCV 2024.\n\n## Q2：More foundation models. \n\nWe adopt Qwen2.5-VL-7B-Instruct as our base model, following many recent quality evaluation studies [1,2,3]. To provide a more comprehensive comparison, here we additionally conduct experiments using Qwen2.5-VL-3B-Instruct. The experimental results below show that the 3B model shows a slight drop in performance while using fewer parameters.\n\n| Accuracy                    | Overall | False Alarm | Miss   | High Value | Sharpness | Overall | Dynamic Consistency | Cumulate Precipitation | High Value Retain |\n|----------------------------|---------|-------------|--------|-------------|-----------|---------|----------------------|-------------------------|--------------------|\n|          | Frame|Frame|Frame|Frame|Frame|Sequence|Sequence|Sequence|Sequence|\n| Qwen2.5-VL-3B Instruct     | 60.93   | 58.95       | 57.33  | 65.81       | 74.19     | 64.42   | 53.43                | 46.19                   | 74.78              |\n| Qwen2.5-VL-7B Instruct     | 61.51   | 65.35       | 67.67  | 69.19       | 78.60     | 66.17   | 53.31                | 48.94                   | 80.52              |\n\n\n\n\n| Frame / Sequence               | BERTScore     | BLEU          | METEOR        | ROUGE_L       | GPT4Score   |\n| ---------------------- | ------------- | ------------- | ------------- | ------------- | ----------- |\n| Qwen2.5-VL-3B Instruct |0.811 / 0.815       |0.212 / 0.216       |0.425 / 0.465       |0.513 / 0.434       |6.77 / 6.36      |\n| Qwen2.5-VL-7B Instruct | 0.809 / 0.815 | 0.213 / 0.212 | 0.420 / 0.461 | 0.512 / 0.436 | 6.87 / 6.58 |\n\n\n[1] Li W, Zhang X, Zhao S, et al. Q-Insight: Understanding image quality via visual reinforcement learning[J]. arXiv preprint arXiv:2503.22679, 2025.\n\n[2] Wu T, Zou J, Liang J, et al. VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank[J]. arXiv preprint arXiv:2505.14460, 2025.\n\n[3] Zhang X, Li W, Zhao S, et al. VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning[J]. arXiv preprint arXiv:2506.18564, 2025."}, {"review_id": "1kM3waCTel", "reviewer": "Reviewer_AvwT", "summary": "Thanks to the authors for their contribution. The paper introduces RadarQA, a Multi-modal Large Language Model (MLLM)-based system designed for quality analysis of weather radar forecasts. The proposed RadarQA employs visual and textual modalities to assess static (single frame) and temporal (sequence) radar predictions. The contributions can be summarized as follows-\n* A task policy covering four evaluation types for static and sequence rating and assessment\n* RQA-70k dataset annotated by a hybrid annotation pipeline combining human labeling with automated heuristics \n* A multi-stage training pipeline for model improvement and comprehensive evaluation of the results for quality analysis in weather prediction", "questions": "My questions are inline below-\n\n* The training strategy seems quite computationally intensive. I'm curious how this approach is reproducible across other weather associated tasks. \n* The authors mention the importance of human annotation and how human annotation was used in curating the proposed RadarQA. However, the paper doesn't describe the process followed for human annotation. How were these annotators recruited? What were their demographics and background information? Ideally, this information is important for evaluating biases associated with data collection and curation. \n* If the overall objective of the work is to perform quality analysis, I'm curious what the effect of using the radar outputs as images instead of raw values. It is observed that converting radar outputs to RGB or image format has an implicit loss of information. How is that being addressed in the proposed workflow? \n* The authors are recommended to provide dataset samples and fine-tuning instruction prompts as a part of the text or appendix so that the work is reproducible. \n* Training configuration for LoRA finetuning is missing\n* I'm curious to see such low Bleu and Meteor scores. Is this justified for such a kind of evaluation, or is it due to custom outputs being evaluated? I missing be missing something here and would appreciate some insights here.", "limitations": "Yes. The authors have highlighted limitations of their work in the manuscript.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The proposed work addresses an important problem of assessing the quality of weather radar forecasts using MLLMs. I liked the task's policy formulation for including descriptive and domain-informed feedback. The training strategy is logical, and proper ablation studies are thoroughly described. The annotation pipeline includes human annotation, which is promising. Overall, the paper is well articulated.\n\nShortcomings\n\nSince neither the proposed dataset nor the codes are openly available, the validation of the dataset and results is restricted. Additionally, the background for defining task attributes based on the scientific attributes library is not clear to me.", "was_revised": true, "final_justification": "The authors have addressed the concerns in their rebuttals and explained quantitatively how the conversion of raw data to images is lossless. The authors are also committing to release their dataset and code publicly.", "rebuttal": "We sincerely thank Reviewer AvwT and the area chair for the thoughtful feedback and appreciation for our training strategy and ablation studies. We also appreciate the positive comments on the annotation pipeline and clarity of the overall presentation. We will continue to refine the manuscript accordingly. Below, we address the reviewer's concerns in detail.\n\n## Q1：Public availability of the proposed dataset and codes. \n\nWe commit to releasing all datasets, codes, and model weights upon acceptance of the paper to support community usage and reproducibility.\n\n\n## Q2：Background of task attribute definition based on the scientific attributes library.\n\nOur model is designed to evaluate the quality of short-term precipitation forecasting. \n\nShort-term precipitation forecasting [1, 2] is one of the most fundamental tasks in the meteorological domain. In our experiments, forecasting models are tasked with predicting the next 12 frames of precipitation based on the previous 10 frames. These predictions are intended to support early warning systems and meteorological agencies in making timely decisions to mitigate the impact of severe convective weather events.\n\nTo evaluate the quality of such forecasts, meteorological experts typically rely on a combination of professional knowledge and formula-based indicators, aiming to ensure the reliability of assessments. \n\nInspired by this evaluation paradigm, we collaborated with a frontline meteorologist with nearly 20 years of experience to construct a *scientific attribute library* for assessing forecast quality from both physical and perceptual standpoints, ensuring professionalism.\n\nSome attributes are derived from metrics, aligning with expert use of objective indicators, while others are designed based on perceptual and domain-specific meteorological knowledge, reflecting expert experience. This library allows for a comprehensive evaluation of model outputs that mirrors expert judgment. The detailed definitions and properties of all attributes can be found in the supplementary material.\n\n[1] Gong J, Bai L, Ye P, et al. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling[J]. arXiv preprint arXiv:2402.04290, 2024.\n\n[2] Zhang Y, Long M, Chen K, et al. Skilful nowcasting of extreme precipitation with NowcastNet[J]. Nature, 2023, 619(7970): 526-532.\n\n\n## Q3: Resource-intensive training strategy and reproducibility across diverse weather forecasting scenarios.\n\nWe appreciate the reviewer's question. The proposed training strategy is indeed applicable to other weather-related tasks. \n\nFirst, fine-tuning a 7B-parameter model is not so costly, and can be reliably performed on two NVIDIA RTX 4090 GPUs, with inference requiring only a single GPU. \n\nSecond, smaller models can also be used depending on the application scenario. We conducted experiments under the same setting using Qwen2.5-VL-3B-Instruct in the table below. Although there is a slight performance drop, it still demonstrates promising potential for quality assessment. \n\n| Accuracy                    | Overall | False Alarm | Miss   | High Value | Sharpness | Overall | Dynamic Consistency | Cumulate Precipitation | High Value Retain |\n|----------------------------|---------|-------------|--------|-------------|-----------|---------|----------------------|-------------------------|--------------------|\n|          | Frame|Frame|Frame|Frame|Frame|Sequence|Sequence|Sequence|Sequence|\n| Qwen2.5-VL-3B Instruct     | 60.93   | 58.95       | 57.33  | 65.81       | 74.19     | 64.42   | 53.43                | 46.19                   | 74.78              |\n| Qwen2.5-VL-7B Instruct     | 61.51   | 65.35       | 67.67  | 69.19       | 78.60     | 66.17   | 53.31                | 48.94                   | 80.52              |\n\n\n| Frame / Sequence               | BERTScore     | BLEU          | METEOR        | ROUGE_L       | GPT4Score   |\n| ---------------------- | ------------- | ------------- | ------------- | ------------- | ----------- |\n| Qwen2.5-VL-3B Instruct |0.811 / 0.815       |0.212 / 0.216       |0.425 / 0.465       |0.513 / 0.434       |6.77 / 6.36      |\n| Qwen2.5-VL-7B Instruct | 0.809 / 0.815 | 0.213 / 0.212 | 0.420 / 0.461 | 0.512 / 0.436 | 6.87 / 6.58 |\n\nThird, data is the key factor to adapt this MLLM-based method to diverse weather forecasting scenarios. Researchers can follow our proposed dataset construction pipeline to construct new datasets. \n\nFinally, with constructed datasets, the proposed training strategy is feasible for a wide range of tasks. \n\n## Q4: Annotator profiles. \n\nFirst, we invited meteorological experts to define the annotation guidelines and standards. \n\nSecond, we recruited a total of 16 annotators with undergraduate-level education or higher, all with backgrounds in meteorology and prior annotation experience. \n\nThen, annotators were asked to conduct a pilot annotation phase, during which their results were reviewed and validated by experts. \n\nAlso, feedback from this phase was used to refine and clarify the annotation rules, aiming to minimize annotator bias and ensure consistency throughout the labeling process.\n\n## Q5: Potential information loss. \n\nWe appreciate the reviewer's concern. \n\nFirst, the conversion from raw radar data to RGB image is almost lossless. We need to convert the raw radar data (float32) into RGB format (uint8). Our statistics below with RMSE, MAE, R2, PSNR, and SSIM metrics show that this conversion is almost lossless. \n\n| RMSE  | MAE   | R2    | PSNR  | SSIM  |\n| ----- | ----- | ----- | ----- | ----- |\n| 0.001 | 0.0008 | 0.996 | 57.004 | 0.999 |\n\n\nSecond, for better visualization and to facilitate understanding of different precipitation levels, we follow previous meterological works [1,2] to divide the precipitation into seven discrete levels (*i.e.*, sunny, light, moderate, heavy, very heavy, intense, and extreme), which is a common and standard visualization process in the meteorological field. \n\n[1] Gong J, Bai L, Ye P, et al. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling[J]. arXiv preprint arXiv:2402.04290, 2024.\n\n[2] Robinson M, Evans J, Crowe B. En route weather depiction benefits of the NEXRAD vertically integrated liquid water product utilized by the corridor integrated weather system[C]//10th conference on aviation, range and aerospace meteorology, american meteorological society, portland, or. 2002.\n\n\n## Q6：More details about the dataset samples and the fine-tuning instruction prompts.\n\nWe commit to releasing all datasets, codes, and model weights upon acceptance of the paper to support community usage and reproducibility. The system prompt and instruction-tuning prompt used in our experiments will also be fully released. \n\nSome samples from RadarQA, along with the corresponding prompt information, are also provided in the supplementary material (Figure A7, Figure A8, and Tables A6–A9). \n\n## Q7: Detailed LoRA fine-tuning training configuration.\n\nWe appreciate the reviewer's feedback. We have provided a description of the training configuration in the main paper (Page 7, Lines 274–280). Below, we further supplement it with additional implementation details.\n\nIn the first supervised fine-tuning stage, we set `lora_rank` and `lora_alpha` set to 8 and 32, respectively, applied to all linear layers. The model is trained for 5 epochs on 8 NVIDIA A800 GPUs, with a batch size of 2 per GPU and `gradient_accumulation_steps` set to 8. DeepSpeed ZeRO Stage 2 is employed to optimize memory usage. We set `VIDEO_MAX_PIXELS` to 602112 and `FPS_MAX_FRAMES` to 12 during training.\n\nIn the second reinforcement learning stage, we add new LoRA parameters, with `lora_rank` and `lora_alpha` set to 8 and 32. The batch size per GPU is set to 4, with gradient accumulation steps of 8, training for 1 epoch. We utilize vLLM to accelerate inference, generating 8 samples per iteration for GRPO reinforcement learning. Training is conducted using DeepSpeed ZeRO Stage 3. \n\nIn the third post-training stage, new LoRA parameters (`lora_rank=8` and `lora_alpha=32`) are added. We randomly select 10,000 samples from RQA-70K for a small-scale supervised fine-tuning. The training parameters are similar to those in the first stage, except that lora_rank is set to 4.\n\n## Q8: Reasons for low BLEU and METEOR scores.\n\nWe appreciate the reviewer's feedback. \n\nFirst, one reason for the relatively low BLEU scores is the metric's sensitivity to exact n-gram matches. BLEU was originally designed for machine translation. As a result, if the predicted sentences use synonyms or have different word order compared to the references, BLEU may penalize them heavily, even if the semantic meaning is similar[1].\n\nSecond, the inherently challenging nature of our task contributes to lower scores across BLEU and METEOR. Each sample of the assessment tasks contains complex, informative meteorological descriptions, which makes it more difficult to achieve high scores, especially compared to less domain-specific tasks. For example, in WeatherQA[2], a domain-specific task focusing on severe weather reasoning, even after fine-tuning, MLLMs still yield relatively low BLEU and METEOR scores, similar to those in our setting. This further highlights the inherent difficulty of domain-specific tasks for current MLLMs.\n\nFinally, despite the relatively low BLEU and METEOR scores, the overall performance of RadarQA shows consistent improvements over baselines, which demonstrates strong potential for weather quality analysis.\n\n[1] Celikyilmaz A, Clark E, Gao J. Evaluation of text generation: A survey[J]. arXiv preprint arXiv:2006.14799, 2020.\n\n[2] Ma C, Hua Z, Anderson-Frey A, et al. Weatherqa: Can multimodal language models reason about severe weather?[J]. arXiv preprint arXiv:2406.11217, 2024."}, {"review_id": "WFAedPVwF3", "reviewer": "Reviewer_Htbn", "summary": "This paper introduces RadarQA, a multi-modal large language model-based method for quality analysis of weather radar forecasts. The authors argue that traditional score-based metrics (like CSI, POD, FAR) fall short of expert-level analysis in descriptive properties, interpretability, and miss the dynamic evolution of weather systems. \n\nTo address this, they propose: A new task definition with four sub-tasks spanning frame-level and sequence-level analysis, each with both rating (quantitative) and assessment (descriptive) components.\n\nThen, the author proposed RQA-70K dataset constructed based on SEVIR weather data, multiple forecasting models, and a hybrid annotation pipeline combining human expert labels with automated metrics. RadarQA model trained via a three-stage pipeline: supervised fine-tuning, reinforcement learning with GRPO, and post-training. The experiment results demonstrating better performance over existing MLLMs, including GPT-4o and Claude-3.7.", "questions": "See in \"Weakness\".", "limitations": "Yes. In \"Conclusion and Limitation\" section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1.  The paper highlights a genuine gap between automated metrics and expert meteorological analysis. This is a crucial application scenario（AI in meteorology / AI for Science). The motivation is clearly articulated with concrete examples showing how descriptive analysis provides insights beyond numerical scores. \n\n2. The four-task matrix (frame/sequence × rating/assessment) is thoughtfully designed to mirror how meteorologists evaluate forecasts.  The scientific attribute library incorporating physics-informed attributes (morphology, intensity, atmospheric physics properties) shows domain expertise and goes beyond simple metrics. \n\n3. The key contribution of this paper is the RQA-70K dataset. The dataset construction process takes the hybrid annotation approach, combining human expertise with automated metrics. The use of GPT-4o for generating assessment reports from attributes is clever, addressing the challenge of obtaining consistent expert annotations. The dataset will benefit to the field. \n\n**Weaknesses:**\n1. While the paper compares against several MLLMs, it lacks comparison with traditional weather analysis methods or weather-specific ML models. A comparison with more traditional methods provides a better understanding of how MLLMs differ from previous methods. How much improvement has been made with the MLLM method, and what level of performance can be achieved by MLLMs.\n\n2. The RL post-training part is not clear enough and lacks RL-specific analysis. What's your objective function or optimization goal? Just simple \"Format Reward\" and \"Accuracy Reward\" with a binary reward. But the structure output is a complex task, maybe this reward doesn't guide the model towards a better JSON structure. Accuracy reward doesn't consider attribute dependencies or correlations. Also, single epoch training may be insufficient for meaningful policy improvement. \n\n3. Figure 2 is kind of a mess and hard to capture the key information. For example, what does each color represent? The author could provide a label in the figure.", "was_revised": true, "final_justification": "Thanks for the author's detailed response.\n\nThe author's responses to Q1, Q2, and Q3 are convincing. The rebuttal addressed my concern.\n\nI have updated my score to \"Borderline Accept\".", "rebuttal": "We sincerely thank Reviewer Htbn for detailed feedback. We greatly appreciate the recognition of our clearly motivated problem setting, dataset construction strategy. We are glad that the reviewer found our four-task matrix well aligned with real-world forecasting practices. Below, we address the reviewer's main concerns point by point.\n\n\n## Q1: Comparison with traditional weather analysis methods and weather-specific ML models.\n\n\nFirst, we compare our method with meteorology-specific metrics and image quality assessment (IQA) metrics on the overall Frame Rating task in the table below. The accuracy, PLCC, and SRCC reflect the consistency between the evaluation results and the expert annotations. RadarQA significantly outperforms the baseline traditional metrics across all three evaluation metrics. While these traditional metrics are reasonable, there remains a noticeable gap between their outputs and expert judgment. These baseline metrics are meteorology-specific ones, including CSI, POD, FAR, bias, and ETS (with thresholds 74/160/181), and IQA metrics including PSNR, SSIM, LPIPS, and DISTS. \n\n\n| Metric       | Accuracy | PLCC | SRCC |\n|--------------|----------|------|------|\n| **Weather-related metrics** |          |      |      |\n| csi_74       | 41.74    | 0.27 | 0.26 |\n| pod_74       | 42.79    | 0.29 | 0.28 |\n| far_74       | 39.07    | 0.16 | 0.15 |\n| bias_74      | 39.42    | 0.20 | 0.23 |\n| acc_74       | 39.53    | 0.22 | 0.21 |\n| ets_74       | 43.60    | 0.29 | 0.29 |\n| csi_160      | 44.42    | 0.43 | 0.41 |\n| pod_160      | 45.70    | 0.46 | 0.44 |\n| far_160      | 38.60    | 0.10 | 0.12 |\n| bias_160     | 46.28    | 0.40 | 0.42 |\n| acc_160      | 38.02    | 0.06 | 0.05 |\n| ets_160      | 41.51    | 0.39 | 0.34 |\n| **IQA methods** |          |      |      |\n| psnr         | 41.05    | 0.28 | 0.25 |\n| ssim         | 42.44    | 0.33 | 0.32 |\n| lpips        | 46.63    | 0.43 | 0.39 |\n| dists        | 53.60    | 0.56 | 0.55 |\n| **Ours**     |          |      |      |\n| RadarQA      | 61.51    | 0.64 | 0.62 |\n\n\n\nSecond, to the best of our knowledge, there are no existing works in the meteorological domain that apply machine learning models for quality evaluation. Traditionally, meteorological forecast quality assessment relies on two approaches: expert subjective judgment and objective metric-based scoring. \n1. **Expert subjective judgment** requires high costs and specialized domain knowledge.\n2. **Objective metric-based scoring** cannot comprehensively evaluate forecast results, lacking understanding and assessment of proprietary meteorological patterns (*e.g.*, convective systems).\n\nTherefore, we leverage the power of MLLMs to effectively integrate expert knowledge (text-based quality assessments) with objective metrics (level-based quality rating) for quality evaluation, enabling a comprehensive assessment that combines domain expertise and quantitative indicators at a lower cost.\n\n## Q2: RL training details.\n\nWe appreciate the reviewer's question. We have provided descriptions of the RL training details in the main paper (Page 7, Lines 258-269). We will elaborate on it below for completeness.\n\n**Overall RL method**. In the RL post-training stage, we adopt the GRPO algorithm to optimize the model's performance on the rating task. Given a query $q$, the model with old parameters $\\pi_{old}$ generates $N$ different responses ${o_1, o_2, ..., o_N}$. Then, a custom reward model is used to compute the corresponding rewards ${r_1, r_2, ..., r_N}$. Unlike other online RL methods (e.g., PPO), GRPO utilizes group-wise averaging to estimate the relative advantage without requiring an additional value model. The computation is performed as follows:\n\n$$\nA_i=\\frac{r_i-\\text{mean}(\\{r_1,r_2,...,r_N\\})}{\\text{std}(\\{r_1,r_2,...,r_N\\})}\n$$\n\nBased on the advantage function, the objective function can be expressed as:\n\n$$\n\\mathcal{J}(\\theta)=\\mathbb{E} \\left[ \\min(\\rho_i A_i,\\ \\text{clip}(\\rho_i, \\ 1 - \\delta, \\ 1 + \\delta)\\ A_i) - \\beta \\cdot D_{\\mathbb{KL}}(\\pi_{\\theta_{new}}||\\pi_{ref})\\right]\n$$\n\nHere，$\\rho_i=\\frac{\\pi_{\\theta_{new}}(o_i|q)}{\\pi_{\\theta_{old}}(o_i|q)}$, $q\\sim{Q}$ and $o_i\\sim\\pi_{\\theta_{old}}$.\n\n**Format reward**. The format reward function uses regular expressions to determine whether the model's output can be correctly parsed. If the format of output can be successfully parsed into a JSON format and all keys exactly match those in the ground truth, the reward is 1; otherwise, the reward is 0.\n\n**Accuracy reward**. Let $n$ be the total number of keys in the ground truth (i.e. `n = len(gt_dict.keys())`). If the extracted answer has $k$ keys whose values match those in the ground truth, then the reward is $k/n$. Since RL training is only applied in rating tasks where the answer could be \"poor\", \"fair\", \"good\", and \"great\", it is easy to compare the predicted answer to the ground truth. The output is first parsed into a valid JSON format to extract the answer. If the parse fails, the reward is also 0.\n\n**Number of training epochs**. A shown in the table below, training with 1 epoch has led to good enough performance, and increasing the number of training epochs does not lead to significant performance improvements. Therefore, we choose to train for 1 epoch to balance computational cost and model performance.\n\n| Epochs |Overall | False Alarm | Miss | High Value | Sharpness | Overall | Dynamic Consistency | Cumulate Precipitation | High Value Retain |\n|--------|-------------|-------------|------|-------------|-----------|--------------|----------------------|-------------------------|--------------------|\n|         | Frame     | Frame     | Frame     | Frame     | Frame     | Sequence | Sequence | Sequence | Sequence |\n| 1      | 59.77       | 68.14       | 67.67 | 65.00       | 74.19     | 61.55        | 64.17                | 42.82                   | 77.78              |\n| 2      | 61.05       | 68.37       | 66.63 | 66.51       | 77.09     | 61.80        | 41.20                | 38.83                   | 77.40              |\n| 3      | 60.47       | 68.02       | 66.86 | 67.21       | 76.63     | 61.05        | 42.20                | 37.20                   | 76.40              |\n\n\n## Q3: Labeling in Figure 2.\n\nWe appreciate the reviewer's suggestion. In Figure 2, we use different colors to highlight some key information. We will simplify the use of colors to make it easier to read. We will also add a label to improve clarity and understanding."}, {"review_id": "vyhO0GeKRo", "reviewer": "Reviewer_6kbe", "summary": "this paper proposes RadarQA, a multi-modal LLM system to judge quality of weather-radar forecasts; they make new 4-task paradigm (frame / sequence rating + assessment), build big RQA-70K data with human-and-script labels, and train model in three stage (SFT + GRPO RL + post-finetune). extensive experiments show large gain over open-source and closed models, even on out-of-distribution radar-SR task.\n\nI really like how you combine the radar physics understanding with language evaluation, it gives a richer insight than just pure numbers,, it feels very helpful to meteorologists and I think it’s a big step forward and quite encouraging.  \n\n\nborderline accept: I feel the benefits slightly outweigh the concerns, so I encourage accepting..", "questions": "maybe add a concise, reader-friendly summary and reduce jargon and unpack acronyms\n\nalso maybe acknowledge that the underlying radar‐to‐forecast architecture builds on established vision-LLM methods, but emphasize your unique adaptations (e.g. handling of spatiotemporal radar echoes, specialized preprocessing)?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "* The four-task paradigm you propose covers both static frames and dynamic sequences and gives both ratings and narrative, it’s kind of complete and well thought out, making the system versatile for different needs, and that design is very neat.  \n* Building the RQA-70K dataset with around seventy thousand samples labeled on fifteen frame and twenty-two sequence attributes is an impressive effort, and using a mix of human annotation and scripts makes it both efficient and reliable, so nice work on that.  \n* Training in three stages with fine-tuning, RL with GPT-based rewards, and post-fine-tuning to polish the style shows very good engineering, and the ablation studies clearly show each stage brings improvement, which is very convincing.  \n* Seeing the jump from around sixty-one to sixty-six percent accuracy on rating tasks and getting a GPT-4 score of 6.9 compared to 5.3 for other models is quite strong, and the out-of-distribution robustness even on super-resolution tasks is really promising, great results!  \n* Noting that the assessment generation also uses GPT-4 labels raises a little concern about possible bias and cost, but it still gives consistent high-quality text, so maybe just discuss that more in limitations.  \n* It would be even stronger to see more human evaluations rather than just GPT-4 scoring to avoid inflating the results, you know, some blind human review would add more confidence and make it feel more solid.  \n* The reliance on proprietary models and heavy compute, like eight A800 GPUs and generating twenty thousand storms, might be a barrier for smaller labs, so maybe you could suggest lighter versions or share code to help reproducibility, which would be awesome.  \n* The paper is detailed and sometimes dense with acronyms and appendix details, so reading can feel a bit tiring; maybe adding a clearer summary or reducing jargon would make it more accessible to broader audience.  \n* While the idea nicely applies MLLM techniques to weather radar, the core algorithm isn’t brand-new, it’s more an application than a fundamental breakthrough, but it’s still valuable and useful, so good job on practical impact.  \n* It might help to compare against stronger domain-specific baselines, like specialized CNN or quality control networks, to show even more advantage beyond general vision-LLMsthis would boost the significance.  \n* Missing an ethical and social impact discussion feels like a small gap because radar forecast assessments could influence public safety decisions, so a short caution section would be encouraging to see.", "was_revised": true, "final_justification": "I appreciate the rebuttal from the authors and my main points are addressed with some satisfaction. Therefore, raising the score.", "rebuttal": "We sincerely thank Reviewer 6kbe for the constructive comments and suggestions on RadarQA. We are glad that the completeness and versatility of our four-task paradigm was appreciated. We are encouraged by the reviewer's recognition of our strong performance in both in-distribution and out-of-distribution settings. We address the reviewer's comments in more detail below.\n\n## Q1: Potential bias and cost of using GPT-4-generated labels.\n\nThank you for the reviewer’s suggestion. \n\nFirst, the key attributes of our dataset is annotated by meteorological experts or calculated by meteorological metrics. GPT-4o is only used to organize these attributes into a fluent language. Therefore, the potential bias introduced by GPT-4o is very small. \n\nSecond, the total cost of GPT-4o labeling was approximately $700. We will include a discussion of the limitations associated with GPT-4o annotations in the final version of the manuscript.\n\n## Q2: Blind human review.\n\nWe conduct human experts evaluation in the \"Expert study\" section (Page 9, Line 326-332) of the submission, where our method show clear advantage over GPT-4o. Specifically, we first randomized the ground truth, RadarQA’s outputs, and GPT-4o’s outputs. Then, we invite nine meteorology experts to participate in the expert study. Based on the observed images, model predictions, and different quality assessment reports, the experts considered the following three dimensions to provide an overall score: (1) Accuracy of the textual content, (2) Whether the issues of concern to the experts are mentioned in the report, (3) The density of effective information. \n\n## Q3: Reproducibility challenges due to reliance on proprietary models and high computational cost.\n\nFirst, we will release all codes, collected datasets, and model weights for reproducibility. \n\nSecond, we highlight that fine-tuning a 7B-parameter model is not so costly, and can be reliably performed on 2 NVIDIA RTX 4090 GPUs, with inference requiring only a single NVIDIA RTX 4090 GPU. \n\nThird, we also trained a smaller 3B model by using Qwen2.5-VL-3B-Instruct as the base model under the same training settings. The results are given below. \n\n| Accuracy                    | Overall | False Alarm | Miss   | High Value | Sharpness | Overall | Dynamic Consistency | Cumulate Precipitation | High Value Retain |\n|----------------------------|---------|-------------|--------|-------------|-----------|---------|----------------------|-------------------------|--------------------|\n|          | Frame|Frame|Frame|Frame|Frame|Sequence|Sequence|Sequence|Sequence|\n| Qwen2.5-VL-3B Instruct     | 60.93   | 58.95       | 57.33  | 65.81       | 74.19     | 64.42   | 53.43                | 46.19                   | 74.78              |\n| Qwen2.5-VL-7B Instruct     | 61.51   | 65.35       | 67.67  | 69.19       | 78.60     | 66.17   | 53.31                | 48.94                   | 80.52              |\n\n| Frame / Sequence               | BERTScore     | BLEU          | METEOR        | ROUGE_L       | GPT4Score   |\n| ---------------------- | ------------- | ------------- | ------------- | ------------- | ----------- |\n| Qwen2.5-VL-3B Instruct |0.811 / 0.815       |0.212 / 0.216       |0.425 / 0.465       |0.513 / 0.434       |6.77 / 6.36      |\n| Qwen2.5-VL-7B Instruct | 0.809 / 0.815 | 0.213 / 0.212 | 0.420 / 0.461 | 0.512 / 0.436 | 6.87 / 6.58 |\n\n\n## Q4: Comparison with stronger domain-specific baselines.\n\nWe compare RadarQA with domain specific baselines in the table below. The results show that RadarQA significantly outperforms all selected baselines, highlighting its effectiveness in aligning with human experts.\n\nThe domain specific baselines are:  **Weather specific metrics** including CSI, POD, FAR, bias, and ETS, with thresholds set at 74 and 160, and 181, and **General FR-IQA methods** including PSNR, SSIM, LPIPS and DISTS.\n\n\n| Metric       | Accuracy | PLCC | SRCC |\n|--------------|----------|------|------|\n| **Weather-related metrics** |          |      |      |\n| csi_74       | 41.74    | 0.27 | 0.26 |\n| pod_74       | 42.79    | 0.29 | 0.28 |\n| far_74       | 39.07    | 0.16 | 0.15 |\n| bias_74      | 39.42    | 0.20 | 0.23 |\n| acc_74       | 39.53    | 0.22 | 0.21 |\n| ets_74       | 43.60    | 0.29 | 0.29 |\n| csi_160      | 44.42    | 0.43 | 0.41 |\n| pod_160      | 45.70    | 0.46 | 0.44 |\n| far_160      | 38.60    | 0.10 | 0.12 |\n| bias_160     | 46.28    | 0.40 | 0.42 |\n| acc_160      | 38.02    | 0.06 | 0.05 |\n| ets_160      | 41.51    | 0.39 | 0.34 |\n| **IQA methods** |          |      |      |\n| psnr         | 41.05    | 0.28 | 0.25 |\n| ssim         | 42.44    | 0.33 | 0.32 |\n| lpips        | 46.63    | 0.43 | 0.39 |\n| dists        | 53.60    | 0.56 | 0.55 |\n| **Ours**     |          |      |      |\n| RadarQA      | 61.51    | 0.64 | 0.62 |\n\n\n## Q5: Ethical and social impact discussion. \n\nWe appreciate the suggestion. We acknowledge that weather prediction assessment may have a quite huge influence in humans' daily life and production. We will include a dedicated caution section addressing these considerations in the final version to ensure responsible use and awareness of the limitations of our work."}]

Author rebuttal:

Meta-review: Summarize the scientific claims and findings:

This paper proposes RadarQA, a multi-modal large language model (MLLM)-based framework for quality analysis of weather radar forecasts. The authors define a four-task evaluation paradigm (frame-level vs. sequence-level × rating vs. assessment) that integrates quantitative scoring with descriptive assessment, aiming to bridge the gap between traditional numerical metrics (e.g., CSI, POD, FAR) and expert-driven meteorological evaluations. They introduce RQA-70K, a large-scale dataset created via a hybrid annotation pipeline that combines human expert labeling, automated heuristics, and GPT-4o-assisted text generation. A three-stage training strategy (supervised fine-tuning, GRPO-based reinforcement learning, and post-training) is applied to a Qwen2.5-VL backbone, resulting in performance gains over both general-purpose MLLMs (e.g., GPT-4o, Claude-3.7) and domain-specific baselines, including in out-of-distribution settings such as radar super-resolution tasks.

Strengths of the paper:

- Addresses a real gap between numerical metrics and expert meteorological interpretation, with well-articulated rationale and domain-specific grounding.

- The four-task matrix closely mirrors expert workflows, covering both static and dynamic aspects of radar forecast quality.

- The construction of the RQA-70K dataset is a substantial contribution. The hybrid annotation pipeline, combining expert-defined physical attributes, automated scripts, and LLM-assisted text generation, is a robust and scalable approach for creating high-quality, domain-specific data.

- The paper includes extensive experiments, including ablations of the training pipeline, comparisons against strong open- and closed-source MLLM baselines, and out-of-distribution tests. The results are consistently positive and convincing.

Weaknesses:

- Clarity of rl training details: The design and justification of the specific reward functions for the RL stage, while addressed in the rebuttal, could be more thoroughly explained and analyzed in the main paper to alleviate concerns about their simplicity.

- Algorithmic novelty: The core technical approach adapts existing MLLM fine-tuning and reinforcement learning techniques (SFT, GRPO) to a new domain. 

Reasons for decision:

The decision to Accept this submission as a poster is based on its significant contributions to applied AI and the meteorological sciences. The primary reasons are: (1) The work directly enables a valuable application (expert-level forecast analysis). It has clear potential for real-world impact in weather prediction and safety. (2) The creation and promised release of the RQA-70K dataset is a major benefit to the research community that will facilitate future work in this area. (3) The model demonstrates clear and substantial improvements over very strong baselines across a comprehensive set of tasks, validated by both automated metrics and human experts.

Summary of rebuttal and discussion:

The author rebuttal was comprehensive and effectively addressed the majority of reviewer concerns: (1) The authors clarified that their contributions include a novel evaluation paradigm and a multi-stage training pipeline, contextualizing their dataset effort alongside recent trends in quality-focused MLLM research. (2) Reviewers requested more details on model selection, LoRA configuration, and RL rewards. The authors provided extensive additional experimental results, full training hyperparameters, and a detailed mathematical breakdown of the reward functions, satisfying these queries. (3)  A key concern was the lack of public code/data and the use of GPT-4o for annotations. The authors made a firm commitment to release all code, data, and model weights upon acceptance, which was a decisive factor for one reviewer to increase their score. They also acknowledged potential biases from GPT-4o and agreed to discuss them in the limitations. (4) A request for comparisons against traditional meteorological metrics was fully addressed with new results showing RadarQA's superior alignment with expert judgment.

The rebuttal process strengthened the submission. Initial concerns about scope, reproducibility, and evaluation were convincingly mitigated. The reviewers converged towards a positive consensus, recognizing the work's value as a solid technical contribution with high practical utility and a valuable resource for the community.
Decision: Accept (poster)