Title: RadarQA: Multi-modal Quality Analysis of Weather Radar Forecasts
Venue: NeurIPS 2025
Accepted: yes
Keywords: Quality Assessment

Multimodal model

AI4Science
Abstract: Quality analysis of weather forecasts is an essential topic in meteorology. Although traditional score-based evaluation metrics can quantify certain forecast errors, they are still far from meteorological experts in terms of descriptive capability, interpretability, and understanding of dynamic evolution. With the rapid development of Multi-modal Large Language Models (MLLMs), these models become potential tools to overcome the above challenges. In this work, we introduce an MLLM-based weather forecast analysis method, RadarQA, integrating key physical attributes with detailed assessment reports. We introduce a novel and comprehensive task paradigm for multi-modal quality analysis, encompassing both single frame and sequence, under both rating and assessment scenarios. To support training and benchmarking, we design a hybrid annotation pipeline that combines human expert labeling with automated heuristics. With such an annotation method, we construct RQA-70K, a large-scale dataset with varying difficulty levels for radar forecast quality evaluation. We further design a multi-stage training strategy that iteratively improves model performance at each stage. Extensive experiments show that RadarQA outperforms existing general MLLMs across all evaluation settings, highlighting its potential for advancing quality analysis in weather prediction.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthis paper proposes RadarQA, a multi-modal LLM system to judge quality of weather-radar forecasts; they make new 4-task paradigm (frame / sequence rating + assessment), build big RQA-70K data with human-and-script labels, and train model in three stage (SFT + GRPO RL + post-finetune). extensive experiments show large gain over open-source and closed models, even on out-of-distribution radar-SR task.\n\nI really like how you combine the radar physics understanding with language evaluation, it gives a richer insight than just pure numbers,, it feels very helpful to meteorologists and I think it’s a big step forward and quite encouraging.  \n\n\nborderline accept: I feel the benefits slightly outweigh the concerns, so I encourage accepting..\n\nSTRENGTHS AND WEAKNESSES:\n* The four-task paradigm you propose covers both static frames and dynamic sequences and gives both ratings and narrative, it’s kind of complete and well thought out, making the system versatile for different needs, and that design is very neat.  \n* Building the RQA-70K dataset with around seventy thousand samples labeled on fifteen frame and twenty-two sequence attributes is an impressive effort, and using a mix of human annotation and scripts makes it both efficient and reliable, so nice work on that.  \n* Training in three stages with fine-tuning, RL with GPT-based rewards, and post-fine-tuning to polish the style shows very good engineering, and the ablation studies clearly show each stage brings improvement, which is very convincing.  \n* Seeing the jump from around sixty-one to sixty-six percent accuracy on rating tasks and getting a GPT-4 score of 6.9 compared to 5.3 for other models is quite strong, and the out-of-distribution robustness even on super-resolution tasks is really promising, great results!  \n* Noting that the assessment generation also uses GPT-4 labels raises a little concern about possible bias and cost, but it still gives consistent high-quality text, so maybe just discuss that more in limitations.  \n* It would be even stronger to see more human evaluations rather than just GPT-4 scoring to avoid inflating the results, you know, some blind human review would add more confidence and make it feel more solid.  \n* The reliance on proprietary models and heavy compute, like eight A800 GPUs and generating twenty thousand storms, might be a barrier for smaller labs, so maybe you could suggest lighter versions or share code to help reproducibility, which would be awesome.  \n* The paper is detailed and sometimes dense with acronyms and appendix details, so reading can feel a bit tiring; maybe adding a clearer summary or reducing jargon would make it more accessible to broader audience.  \n* While the idea nicely applies MLLM techniques to weather radar, the core algorithm isn’t brand-new, it’s more an application than a fundamental breakthrough, but it’s still valuable and useful, so good job on practical impact.  \n* It might help to compare against stronger domain-specific baselines, like specialized CNN or quality control networks, to show even more advantage beyond general vision-LLMsthis would boost the significance.  \n* Missing an ethical and social impact discussion feels like a small gap because radar forecast assessments could influence public safety decisions, so a short caution section would be encouraging to see.\n\nQUESTIONS:\nmaybe add a concise, reader-friendly summary and reduce jargon and unpack acronyms\n\nalso maybe acknowledge that the underlying radar‐to‐forecast architecture builds on established vision-LLM methods, but emphasize your unique adaptations (e.g. handling of spatiotemporal radar echoes, specialized preprocessing)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces RadarQA, a multi-modal large language model-based method for quality analysis of weather radar forecasts. The authors argue that traditional score-based metrics (like CSI, POD, FAR) fall short of expert-level analysis in descriptive properties, interpretability, and miss the dynamic evolution of weather systems. \n\nTo address this, they propose: A new task definition with four sub-tasks spanning frame-level and sequence-level analysis, each with both rating (quantitative) and assessment (descriptive) components.\n\nThen, the author proposed RQA-70K dataset constructed based on SEVIR weather data, multiple forecasting models, and a hybrid annotation pipeline combining human expert labels with automated metrics. RadarQA model trained via a three-stage pipeline: supervised fine-tuning, reinforcement learning with GRPO, and post-training. The experiment results demonstrating better performance over existing MLLMs, including GPT-4o and Claude-3.7.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.  The paper highlights a genuine gap between automated metrics and expert meteorological analysis. This is a crucial application scenario（AI in meteorology / AI for Science). The motivation is clearly articulated with concrete examples showing how descriptive analysis provides insights beyond numerical scores. \n\n2. The four-task matrix (frame/sequence × rating/assessment) is thoughtfully designed to mirror how meteorologists evaluate forecasts.  The scientific attribute library incorporating physics-informed attributes (morphology, intensity, atmospheric physics properties) shows domain expertise and goes beyond simple metrics. \n\n3. The key contribution of this paper is the RQA-70K dataset. The dataset construction process takes the hybrid annotation approach, combining human expertise with automated metrics. The use of GPT-4o for generating assessment reports from attributes is clever, addressing the challenge of obtaining consistent expert annotations. The dataset will benefit to the field. \n\n**Weaknesses:**\n1. While the paper compares against several MLLMs, it lacks comparison with traditional weather analysis methods or weather-specific ML models. A comparison with more traditional methods provides a better understanding of how MLLMs differ from previous methods. How much improvement has been made with the MLLM method, and what level of performance can be achieved by MLLMs.\n\n2. The RL post-training part is not clear enough and lacks RL-specific analysis. What's your objective function or optimization goal? Just simple \"Format Reward\" and \"Accuracy Reward\" with a binary reward. But the structure output is a complex task, maybe this reward doesn't guide the model towards a better JSON structure. Accuracy reward doesn't consider attribute dependencies or correlations. Also, single epoch training may be insufficient for meaningful policy improvement. \n\n3. Figure 2 is kind of a mess and hard to capture the key information. For example, what does each color represent? The author could provide a label in the figure.\n\nQUESTIONS:\nSee in \"Weakness\".\n\nLIMITATIONS:\nYes. In \"Conclusion and Limitation\" section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThanks to the authors for their contribution. The paper introduces RadarQA, a Multi-modal Large Language Model (MLLM)-based system designed for quality analysis of weather radar forecasts. The proposed RadarQA employs visual and textual modalities to assess static (single frame) and temporal (sequence) radar predictions. The contributions can be summarized as follows-\n* A task policy covering four evaluation types for static and sequence rating and assessment\n* RQA-70k dataset annotated by a hybrid annotation pipeline combining human labeling with automated heuristics \n* A multi-stage training pipeline for model improvement and comprehensive evaluation of the results for quality analysis in weather prediction\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed work addresses an important problem of assessing the quality of weather radar forecasts using MLLMs. I liked the task's policy formulation for including descriptive and domain-informed feedback. The training strategy is logical, and proper ablation studies are thoroughly described. The annotation pipeline includes human annotation, which is promising. Overall, the paper is well articulated.\n\nShortcomings\n\nSince neither the proposed dataset nor the codes are openly available, the validation of the dataset and results is restricted. Additionally, the background for defining task attributes based on the scientific attributes library is not clear to me.\n\nQUESTIONS:\nMy questions are inline below-\n\n* The training strategy seems quite computationally intensive. I'm curious how this approach is reproducible across other weather associated tasks. \n* The authors mention the importance of human annotation and how human annotation was used in curating the proposed RadarQA. However, the paper doesn't describe the process followed for human annotation. How were these annotators recruited? What were their demographics and background information? Ideally, this information is important for evaluating biases associated with data collection and curation. \n* If the overall objective of the work is to perform quality analysis, I'm curious what the effect of using the radar outputs as images instead of raw values. It is observed that converting radar outputs to RGB or image format has an implicit loss of information. How is that being addressed in the proposed workflow? \n* The authors are recommended to provide dataset samples and fine-tuning instruction prompts as a part of the text or appendix so that the work is reproducible. \n* Training configuration for LoRA finetuning is missing\n* I'm curious to see such low Bleu and Meteor scores. Is this justified for such a kind of evaluation, or is it due to custom outputs being evaluated? I missing be missing something here and would appreciate some insights here.\n\nLIMITATIONS:\nYes. The authors have highlighted limitations of their work in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
This work proposes a new benchmark for multi-modal quality analysis in weather radar forecasting, supporting both frame-wise and sequence-wise evaluation tasks. It introduces a hybrid annotation pipeline that combines large language models (LLMs) and human annotators. The authors also train a model to analyze forecasting performance based on this benchmark.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper provides a thorough analysis of the collected dataset, which appears well-suited for the weather forecasting task.

2. It explores a diverse set of question types, using various input modalities—including text, single images, and image sequences—to support a range of question-answering tasks.

Weaknesses:

1. The technical contributions beyond dataset collection appear limited.

2. Regarding the model selection, it is unclear whether different types or sizes of foundation models were explored. This makes it difficult to assess the effectiveness and advantages of the proposed training scheme

QUESTIONS:
See weaknesses

LIMITATIONS:
The proposed method appears to be more focused on dataset construction for weather forecasting rather than introducing a fundamentally new modeling approach.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03