WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning
Authors: Xiangyu Zhao
Zhiwang Zhou
zhangwenlong
Yihao Liu
Xiangyu Chen
Junchao Gong
Hao Chen
Ben Fei
Shiqi Chen
Wanli Ouyang
Xiao-Ming Wu
LEI BAI
Venue: ICLR 2025 Poster (2025)
Keywords: AI for Science
Weather foundation model
in-context learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: The Earth's weather system involves intricate weather data modalities and diverse weather understanding tasks, which hold significant value to human life. 
Existing data-driven models focus on single weather understanding tasks (e.g., weather forecasting). 
While these models have achieved promising results, they fail to tackle various complex tasks within a single and unified model. 
Moreover, the paradigm that relies on limited real observations for a single scenario hinders the model's performance upper bound.
Inspired by the in-context learning paradigm from visual foundation models and large language models, in this paper, we introduce the first generalist weather generalist foundation model (WeatherGFM) to address weather understanding tasks in a unified manner. 
Specifically, we first unify the representation and definition for diverse weather understanding tasks.
Subsequently, we design weather prompt formats to handle different weather data modalities, including single, multiple, and temporal modalities. 
Finally, we adopt a visual prompting question-answering paradigm for the training of unified weather understanding tasks. 
Extensive experiments indicate that our WeatherGFM can effectively handle up to 12 weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing. Our method also showcases generalization ability on unseen tasks. The source code is available at https://github.com/xiangyu-mm/WeatherGFM.

Published as a conference paper at ICLR 2025
WEATHERGEM: LEARNING A WEATHER GENERALIST FOUNDATION MODEL VIA IN-CONTEXT LEARNING
Xiangyu Zhao?: !. Zhiwang Zhou', Wenlong Zhang' S Yihao Liu', Xiangyu Chen', Junchao Gongl.. HHo Chen', Ben Fei', Shiqi Chen', Wanli Ouyang',
Xiao-Ming Wuz Lei Bai'
'Shanghai AI Laboratory 'The Hong Kong Polytechnic University *Shanghai Jiao Tong University "Shanghai Metcorological Service
zhangwen o jlab. org. en, xiao-ming wu@polyu. edu
ABSTRACT
The Earth's weather system involves intricate weather data modalities and di- verse weather understanding tasks, which hold significant value to human life. Existing data-driven models focus single weather understanding tasks (e.g., weather forecasting). While these models have achieved promising results, they fail tackle various complex tasks within single and unified model. Moreover, the paradigm that relies on limited real observations for a single scenario hin- ders the model's performance upper bound. Inspired by the in-context learning paradigm from visual foundation models and large language models, in this pa- per, we introduce the first weather generalist foundation model (WeatherGFM) to address weather understanding tasks in unified manner. Specifically, we first unify the representation and definition for diverse weather understanding tasks. Subsequently, we design weather prompt formats to handle different weather data modalities, including single, multiple, and temporal modalities. Finally, we adopt a visual prompting question-ans answering paradigm for the training of unified weather understanding tasks. Extensive experiments indicate that our WeatherGFM can effectively handle up 12 weather understanding tasks, including weather fore- casting, super- super-resolution, weather image translation, and post-processing. Our method also showcases generalization ability on unseen tasks. The source code is available at
com/ xi xiangyu mmn/ mmm/Wes WeatherGGM
INTRODUCTION
Modeling Earth weather systems involves series of complex subprocesses that are intended trans- form intricate Earth observation data into applications like weather forecasting (Chen et al.) 2023a balp Bi
al. (023), downscaling (Chen al.] (2022). assimilation (Huang et al.) (024), retrieval (Liu et al.
2011). and bias correction (Gong et al. [2024). During the past decade, many data-driven machine
learning methods have been investigated for various weather understanding tasks and delivering desirable performance on specific tasks. For example, recent studies using large-scale training data (e.g., ERA5 reanalysis data (Hersbach et al., 20200) have exceeded the accuracy of conventional nu- merical weather forecasts. However, current weather foundational models face challenges regarding generalizability and data scale limitations. On the one hand, the Earth observation system consists of a variety of observation devices, such as satellites, radar, and weather stations, which produce diverse modalities of data. Consequently, designing specific model for a single-task scenario is highly complex, time-consuming, and labor-intensive. On the other hand, large-scale data in fields such as computer vision can be obtained at a low cost, whereas weather understanding tasks face an intrinsic bottleneck data scale due to restrictions on individual scenes and single observation devices as shown in Table I For instance, local short-term precipitation forecasting models can only utilize a finite range of observational data.
"This work was primarily conducted during the author's 's internship at the Shanghai Artificial Intelligence Lahoratory.

---

Published as a conference paper at ICLR 2025
A significant trend in Al research is the development of foundation models, shifting towards large- scale pre-trate vre-rrining and in-context learning. This paradigm enables unified processing of multitude of complex tasks and generalization unseen tasks. For example, large language models (LLMs) can perform a variety of language-centric tasks (e.g., sentiment analysis, question answering and machine translation) by combining language input-output examples with new query inputs (prompts) without optimizing model parameters (Brown 2020). Similarly, vision foundation models (Wang al. 2023b; Liu et al. 2023b; Chen et al. 2024b) employ visual prompts with query inputs to carry out diverse image-centre tasks, such as semantic segmentation. depth estimation, and image restoration. These studies highlight the significant potential of generalist foundational models.
The study of foundation models remains largely limited in weather understanding, with the majority focused on Computer Vision and Natural Language Processing. While there has been some progress with large foundation models in weather and climate, the focus is mainly on weather forecasting and downscaling tasks. For example, Climax (Nguyen et al., 2023 uses a pre-training tee-training--ietunnnn paradigm for weather forecasting and downscaling, Aurora (Bodnar et al. (2224) employs LoRA to unify weather forecasting and quick prediction of atmospherie chemistry. However, as shown in Table I1 these studies do not take into account the modeling of multi-modalities and multi-tasks. This poses a challenge: Is it possible to design a universal foundation model capable of handling a variety of complex weather understanding tasks and data modalities?
In this paper, we first propose a weather generalist foundation model, WeatherGFM, to uniformly address variety of complex weather understanding tasks and data modalities. Unlike prior studies that focused on weather forecasting, our proposed method can expand the task scope to weather forecasting, weather super-resolution (i.e., weather downscaling) (Veillette al.) 2020), weather image translation (similar to retrieval in weather) (Veillette et al. 2020). and post-processing (Gong al., 2024). These tasks all belong to the domain of weather understanding, but their modalities are distinct. Specifically, Sequence modal data can be utilized for weather forecasting, such as short-term predictions based on radar data. Multi-modal data can employed for weather image translation, such as converting multi-modal satellite data to generate radar data. Single-modal data can be applied to various common scenarios, such as radar image super-resolution and post- post-processing. To unify the diverse weather data modalities into a general representation, we introduce weather prompt format that assigns different prompt phrases to various modalities. By leveraging in-context learning, our WeatherGFM achieves a promising in-context ability both various seen tasks and unseen tasks. The significance of our work can be summarized as:
We propose the first weather generalist foundation model (i.e., WeatherGFM), which can handle more than 12 weather understanding tasks.
Our weather prompt design supports a diversity of weather data modalities, including time-series, multi-modal, and single-modal data.
Our WeatherGFM with in-context learning first demonstrates the generalization ability to unseen weather understanding tasks.
2 RELATED WORK
Weather understanding and beyond. Over the past decade, machine learning techniques (Zhang ct al. 2022 2023 2024) have consistently attracted attention in the field of weather and climate. Numerous data-driven machine learning models have been proposed address classical tasks in weather understanding (Veillette et al.) 2020). such as forecasting, super-resolution, image translation, and prosesroc.ssing. Weather forecasting (Bi et al., (203)) aims to predict future observations from past data. Weather super-resolution tasks, I.c., weather downscaling Chen et al. (2022) focus on recovering high-resolution data from low-resolution observations. Weather image translation tasks (Stock et al. 2024) involves converting existing observational data into desired target modalities, such as transforming satellite observations into ground- weather radar data. Post-processing tasks seek to enhance existing model results, such as bias correction and deblurring (Gong et al. 2024). Despite significant advancements, current methods often rely on specialized datasets and customized single-tast models for certain scenarios. Consequently, single-task models struggle to exhibit strong generalization abilities and fail to capture the interconnections between diverse tasks, which hinders the establishment of simulations for the Earth system.
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of task-specific models and general models across Earth science and computer vision domains. Our proposed WeatherGFM stands out with its capability to handle multiple data, and demonstrate general adaptability underscoring its strength in acquiring challenging to-access weather data.
tasks, process -o-engginng---sssss
Data Acquisition
Multi-taska
Multi-modal
Generalist
Cuteguryy
Method
Supported Tacks
Difticulty
support?
support?
support?
HAT Chen
2023b
Low--oot
Image spperircctttion (SR)
IPT
Cl
3021
Low-oost
Image restonation Dersin, Dallnre
Requires fin--unnng
Computer
Image restoration
Painter
Wing al.
2023a
Low-oost
Vision
Segmentttion, Keypaist detection
PramptCiPP Liu
Low-eost
Image restoratios, Deviit,
GenLV
Low cost
Image resturaion, onhuncement, transltioo
Profiff
2024
Highsoos
Weather foroessting
Caucas
Gemp
Highsoet
Pest-preessing
Climax
Nguyen et al.
2025
High-cost
Weuther lorecasting, resolution
Requires line caning
Earth
Weather forecasting
Sciesce
Aurura
Bodnar
2024
Hligh-ostt
Requires fine-cuning
Astospheric chemisty peodiction
WeotherGIM
Weather forecusting. Weather image SR
Highsost
(ouns)
Weather image translation. Post processing
Weather foundation model. The rise of foundation models (Liu et al. 2024| Zhao et al. 2024a b Xu et al.] 2023) in Natural Language Processing and computer vision has sparked interest in their application for weather and climate. Large foundation models, enhanced through pre-training, improve the generalization of AI climate models and can be fine-tuned for specific tasks. Pathak ct al. (2022) proposed FourCastNet, a climate pre-trained model using Vision Transformer for high- resolution predictions and rapid inference through supe-super pre-training and autoregressive fine-tuning. Pangu-Weather
Weather (Bi et (023) utilizes a 3D Earth-specific Transformer for accurate global predictions. ClimaX (Nguyen et al. 2023) introduces supervised pre-training to weather prediction, offering flexibility for diverse forecasting tasks. A pre-training foundation model usually requires mask modeling for pre-training and then undergoes fine-tuning on specific tasks, such as fine- tuning the pre-trained model on weather forecasting, remote sensing classification and segmentation
tasks Bodnar et al. (2024); Cong et al. (2022) ; Noman et al.] (2024): Li al. (2024).
Visual in-context learning. In recent advancements, visual in-context learning has emerged as promising research area, inspired by the success of language models like GPT-3 (Brown) 2020).
These models adapt to various NLP tasks using prompts or in-context examples without extensive retraining. Similarly, in the vision domain, models such as MAE-VQGAN (Hojel et al., 2024) and Painter (Wang al.) (0235) have begun exploring in-context learning. However, challenges persist, especially in low-level tasks requiring detailed pixel manipulation. To address this, PromptGI (Liu et al.] 2023b) and GenLV have incorporated in-context learning concepts into their designs to unify low-level vision tasks with diverse input and output modalities, aiming to develop generalist models. Vision-language models like Unified-IO (Lu et al., 2022) and Unified-IO 2 (Lu al., 2024 have made significant progress in integrating multiple tasks, highlighting the potential for unified approaches across modalities. Additionally, compositional visual reasoning, exemplified by Visual Programming (Gupta & Kembhavil 2023), aligns with in-context learning goals by emphasizing visual task synthesis. ViperGPT (Surís al. (003) further demonstrates foundational models for visual reasoning, employing computational techniques similar to our objectives, though without relying on programmatic inputs. These collective efforts pave the way for more sophisticated and versatile visual in-context learning frameworks.
METHOD
3.1 UNIFIED REPRESENTATION OF WEATHEE UNDERSTANDING TASKS
Weather understanding tasks involve processing multi-source observational data Veillette et al. 2020), such as geostationary satellites (GEOS), polar-orbiting satellites (POES), weather radars, and ground observation stations. Each task (c.g., weather forecasting, spatial and temporal super-resolution. weather image translation, and post- post-processing) utilizes different types of input and output data. To address this challenge, we first developed a unified data representation that standardize these
3

---

Published as a conference paper at ICLR 2025
Weather Forecasting
Raba Eurapslacce
Sustae Znsupokiion
Weather Image Translation
GOES VaRke
GEOS Suellite
Tnrper
PEOS Saciliie
Wester Rader
Weather Super-Resolution Treqool
Post-processing
Spati
Detker
Ground Statien
Ingel
Figure 1: Illustration of the unified representation framework for weather understanding tasks.
diverse tasks. Unlike traditional methods that rely on task-specific models for each distinct task, we introduce a universal foundational model capable of addressing various weather understanding tasks through a single and general solution.
As shown in Figure 1 several key weather understanding tasks be framed using different types of input and output data. For instance, the weather spatial super-resolution (SR) task generates high- resolution image H from a low-resolution image LRR while weather temporal super-resolution predicts a high- nigh-resolution image x' based on two consecutive observed input images ZLR and "LE where represents particular moment in time. The weather temporal super-resolution task aims to restore the missing observed data in time (. Weather forecasting relies on sequence of observed data points {a+ x² ..... _x") that are gathered over the past time steps. These observed data points serve as condition, enabling the prediction of future data points such as points [x'+1, ,x'+2.... The image translation task focuses on converting an input image from one modality (c.g., satellite image) to another modality (e.g., radar image). Formally, can represent these tasks projections from the source input data Xs to the target output data Xy:
f++1 t-1
Xs XT.
When Xs unn and X7 Tnn, the task corresponds to spatial SR. Similarly, when Xs = (xr, ..2.....,') and X7 = [='+1,='+ ...... the task represents weather forecasting. As these tasks differ in their input and output formats, as well as sequence lengths, the key challenge lies in unifying them within one coherent data representation.
3.2 WEATHERGFM: A WEATHER GENERALIST FOUNDATION MODEL
We present the Weather Generalist Foundation Model (WeatherGFM) to tackle the challenges inherent in a range of weather understanding tasks. Through in-context learning, WeatherGFM can uniformly handle various weather understanding tasks involving multiple data modalities.
Weather prompt designing. In large language models and vision foundation models, task prompts commonly provide specific task-related input-output pairs. As shown in Figure 2 in machine translation (Stahlberg 2020), the model is given English French text pairs as prompts. The model can perform machine translation tasks based on these sample prompts for given input. In visual tasks (Wang et al.) 2023a), the visual prompt imagel may be natural image, and image2 is the corresponding segmented image. The model will conduct the segmentation task for new input image3 to obtain the segmented image.
Following this paradigm, we designed weather prompts for weather understanding tasks. Since the input for weather understanding tasks involves multiple modalities, such as single weather observation variable, multiple different weather variables, and time-series weather variables, proposed three prompts to handle different modalities of input. In Figure 2 weather prompt1 is similar to visual prompts, converting single modality image into target image. In weather prompt2, the input modality can be two different channel satellite observation images (c.g., IR069 and IR107 data), and the output be weather radar observation data for image translation tasks. In weather
4

---

Published as a conference paper at ICLR 2025
Text Prompt: (example: sea otter, loutre de merr)
query: cheese
output: fromage
Visual Prompt:
(example: imagel, image2]
query: image3
output: image4
Weather Prompts
Weather Prompt1:
texample: imagel, image2]
query; image3
output: image4
Weather Prompt2: (example: imagel jimage2, image33 query: image4,image5
output: image6
Weather Prompt3:
(example: sequeneel, sequenee2)
query: sequence3
output: sequence4
Figure 2: Comparison of weather prompts with text and visual prompts design.
Trank T'imesariin hm Medal
WeetherGFM
Ouig..
Pronptiapd
Treept Expe
lope
Theeoreie
Sical Paats
Vodaa Trandaroo
Lubo Decokk
d-hundd waaler wnderkandtnggtteeee tedee
Taiget
Teckk2
Tuski:
Trekk: Single Medal
Micidd MoodhMolk Nol
Entodlaog
firet an
Targee
Vidill
Nedial
leget
T'rpe
Tinks
Thirsig p0000
X
Mtewell
Xix
Lozs
Maddet
tepil
Micuet Ture
Tagi Pleaar
Vinetter CPM
paidi
?
Ground patch
Predictd patch
Frduced
Figure 3: Overall approach of our weather generalist foundation model (WeatherGFM),
prompt3, time-series prompts can be input to perform weather roreaesting-related tasks. With these forms of prompt design, our method can handle most weather understanding tasks.
Weather in-context learning. Inspired by the success of in-context learning in large language models (Dong al. 2022) and vision foundation models (Wang al., 2023a), propose unify the weather understanding problem as the visual prompting question-answer paradigm. as illustrated in Eq. 2 Specifically, given a visual question-anssee answer prompt pair (P.n. Ptorget) as task-guided prompt and query input Xins the model is expected to perceive the context of the prompt (i.e., what task represents). Consequently, the model can perform the corresponding operations on the query with the prompt. This process can be formulated as follows:
Xtarget = (F P Xini 0):
(2)
where F, represents a universal foundation model parameterized by 0. P.n and Prorget denotes the input and target of task prompts. We can determine what task will be performed on the input X by selecting the task-specific prompt P... and Ptarget, and then obtain the target Xtarget for the corresponding task through the model F-.
Mixed-modal mask modeling, Upon redefining the output spaces of the aforementioned representa- tive vision tasks, is observed that both the input and output of these tasks are in the form of images as transformers-basee architectures could provide flexibility by treating the image-like data as a set of tokens. Therefore, we build the WeatherGFM architecture on Vision Transformers (ViT) and propose mixed-moda masked image modeling (MMIM) pipeline train multiple weather understanding tasks as shown in Figure (2023a): Liu et al. (2023b); Chen et al. (2024a), introduce mixed- modality masking various weather modalities for visual iudestion-and-answer modeling in weather understanding tasks. This process can be formulated follows:
Inspired by the concept of Visual Question Answering WWang et al.
P Purrget= X target = (r(Pin M (Ptaagget), Xin, M(X M(Xtarget): 0);
(3)
where we randomly conduct mask operation M on the prompt target P Ptarget as well as the ground truth X target according the mask ratio. Meanwhile, the prompt input Pt,, and the input query X

---

Published as a conference paper at ICLR 2025
will be retained entirely. Piarget and Xtrgge represent the predicted target output of model F,. The optimization objectives are as follows:
8 fotal Lzz(Parget: Piarget) + L2( (Xttaggtt Xtarget).
(4)
where we use MSE (mean square enror) loss L2 train the weather generalist foundation model. In the inference stage, we keep the P... Prarget- and X intact while the target image is fully masked. This target full masking strategy allows generalist foundation models generate the corresponding target through a visual question-and-answer format. Our WeatherGFM comprises two main elements: the format for input data and the architectural design.
Given an input shape (C, HI, W), ViT predicts an output of shape (C', HI,, W' where C' represents the input channels and C' represents the output channels. As shown in Figure 3 different tasks have different channels. The model tokenizes the input into a sequence of patches, with each patch having a size of C' p², where p is the patch size. Unlike RGB-based image data, where the channels are fixed, the number of physical variables in climate and weather data can vary between different datasets and tasks. To adapt the ViT to different weather-related downstream tasks, designed task-specifi patch embedding layers within the architecture. After the patch embedding layer, we use an MLP layer to align the embeddings of different tasks to the same space:
Input format:
7c = PatchEmbedc(x)), x CCHxW ZC R R
(5)
2g MLN((LN(2))),
20 E RND
where N, D denotes the number of input tokens and the transformer dimension, respectively. For the masked area, we follow previous works (Liu et al. 2023a) to use a learnable token vector to replace each masked patch. We adopt the block-wise masking strategy, taking the masking ratio as 75%.
Architecture: A vanilla vision Transformer (ViT) is adopted as the backbone architecture. consists of task-specific patch-embedded layers and several alternating layers made of Multi-Head Self-Attention (MHSA) and MLP blocks. Layer Normalization (LN) is applied before every block, and residual connections are applied after every block. As shown in Figure 3 this process can be formulated as follows:
Ze MHSA(LL(ZZ Ze-1, I...L,
(6)
= MLP(LN(2)) + +2,, 2
where L denotes the number of layers. After the attention layers, we employ a prediction head and then unpatchify the output of the prediction head. The prediction head is one-layer MLP with hidden dimension of 1024.
4 EXPERIMENTS
4.1 WEATHER UNDERSTANDING TASKS
We incorporate up to 12 tasks including diverse weather forecasting, weather super-resolution, weather image translation and weather processog tasks into experiments. Specifically, leverage the Storm EVent ImageRy (SEVIR) (Veillette et al.) 2020). ERAS (Hersbach et al.) (2020).
POMINO-TROPOMA product (Liu al. 2020) and GEOS-CH (Keller et al. 2021) datasets train
and evaluate our WeatherGFM. We provide a detailed introduction in Appendix and B
4.2 IMPLEMENTATION AND EVALUATION
Training details. During training, we resize the weather images of different resolutions to a resolution of 256x256 and input them into the model in accordance with the combination mode of P... Pout, X4n, Xout in the task-specific prompt format, resulting in a x 256 x 256 total input resolution. The LI loss is employed as the loss function. For optimization, the AdamW optimizer with a cosine learning rate scheduler is utilized. The base learning rate is le4. The batch size is 20.
Evaluation metrics. Besides RMSE and ACC, we also include the Critical Success Index (CSI), which is commonly used in weather understanding tasks (c.g., precipitation nowcasting). Details can be found in Appendix C.4

---

Published as a conference paper at ICLR 2025
Thacorto Prange
Bepul
Thaeedio Ougpc
Trats
Rah
Prospe Iaper
Proepe Taraet
Ispat
Ourput
Grousd Trath
CROW
EEDIA PORD 1083-000
Prosept Irgat
Prongt Tuger
Jspal
Oslpat
Grossd Trath
Spainal SE
Spaaii
GEOS-IN GEOD-BR
Figure 4: Visual results of the weather understanding tasks by our WeatherGFM.
7

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative results on weather understanding tasks. single-task model. f: trained with all weather understanding tasks. x: continual training with REAS dataset. RMSE, ACC and CSI are calculated as the quantitative metric. lower RMSE and higher CSI/ACC indicate better results. For Weather Forecasting tasks, we report RMSE scores along with the average ACC score for forecast horizons ranging from 6 to 168 hours. The best results are highlighted bold, and the second-best results are underscored.
Weither super-esolltion (SR)
Thek some
Sutellite Spatial SR
Radar Temporal SR
Rudar Sputial SR
UNe!
0.932
0.650
0.912
0.739
0,485
0.182
0.034
0.650
0.675
0,400
0.184
0.047
0.900
0.333
0.285
0.061
0.120
0.830
0.637
0.358
WeatherGFM
Woathe Forecawting
Post-prozssing
Tisk nome
Sutellite extrepoaation
Radar extrauulation
Deblur
CS1--4000
CSL-66000
RMSE
CSL/44
CSU211
UNe!
1,033
0.617
0.900
0.815
0.353
0.082
0.007
0.713
0.457
0.145
0.027
0:408
0,540
0.913
0.490
0,640
0.079
0.007
0.163
0.591
0.104
WeatherGFM
Weather image toaslation
Task name
GOES2Rabar
GOES IR2GOES IR
RMSE
CSU/1
CSI/60
CST1-6000
CSU--4000
UNe!
0.821
0.222
0.570
0180
0.153
0.079
0.915
0.929
0.741
0.638
0.078
VIT
0.445
0,502
0.436
0.180
0.131
0.042
0.257
0.987
0,972
0,809
0.136
0.447
Tusk nume
GOES- IR2GOER Visible
GOESIPOES
Mecics
RMSE
CSU2000
CSU3200
CSI4400
CSU5600
CSU6800
RMSE
CSV1
CSUS
CSV/1
CSU15
UNa
0.915
0.422
0.285
0.179
0.040
0.799
0.360
0.202
VIT
WeatherGFM
0.439
0.580
0.439
0.298
0.165
0.068
0.302
0.682
0.562
0.382
0.197
Horecasting 12M
Forecasting
Lead Time [h..
6h
24h
72hs
120h
168h
6h
24h
72h
120h
168h
ACC
IFS
0.97
1.02
1,30
1.21
2.23
0.79
1.11
1.92
2.89
3,81
0.836
ClimaX
1.11
1.19
1.47
1.33
2.17
1.04
1.31
2.02
2.29
3.35
0.824
WeatherGFM
LO8
123
1.56
1.68
1.76
1.12
1.26
1.99
2,61
3.11
0.848
Table Standard deviation of the performance computed based on 20 different prompts. Avg. CSI denotes the mean of the CSI score across thresholds [16, 74, 133, 160, 181, 219].
GOES2Radar
Radar extrapolation
GOES-IR2GOES-IE
Radar Spatial
Radar Temporal SR
Deblur
Avg. RMSE
0.0087 0.0187
0.0012
0.0481 0.0284
0.0001
0.0002 0.0010
0.0016
Avg. CSI
0.0201
0.0006
0.0047
4.3 EXPERIMENTAL RESULTS
Currently, there is no general weather foundation model that can comprehensively handle all the discussed weather understanding tasks simultaneously. Although many machine learning methods have been investigated for single tasks, they generally adopt different backbone networks and design strategies tailored them. For a fair comparison, have trained series of basclines (i.c., single-task model) for each weather understanding task under a consistent training setup, including commonly used UNet (Trebing al. (021) and ViT (Nguyen al. 2023) networks. Notably, the purpose of this paper is not to achieve state-of-the-an performance on every task.
The weather generalist foundation model can achieve strong universal capabilities. As seen in Table 2 our WeatherGFM, equipped with a straightforward VIT backbone, shows impressive performance and adaptability in 12 weather understanding tasks. It is not only capable of conducting
8

---

Published as a conference paper at ICLR 2025
Time ocries Prompt Imput
Time secies Prompe Target
Timo scries Input
Time sones Outpet 1
Ground Truhh
Time-series Fronpt Input
Time-sories Foumpt Tarpet
Time-sefies Input
Time series Oulpul 2
Ground Truth
Figure 5: Case studies of our WeatherGFM with different prompts in the radar extrapolation task.
weather forecasting and super-resolution tasks but is also proficient in dealing with weather image translation and post-processing tasks. Overall, our WeatherGFM achieves promising performance on a diversity of weather understanding tasks.
The weather generalist foundation model outperforms single-task models. In Table 2 we notice that our WeatherGFM achieves results that outperform the baseline in weather forecasting, weather super-resolution, and image translation tasks. For instance, in radar extrapolation tasks, our WeatherGFM with universal ViT-based model outperforms the single-task VIT model. This indicates that unified approach to weather understanding tasks can potentially break the performance upperbound of single-task models.
In-context learning can generate correct outputs across a variety of data modalities and tasks. As depicted in Figure 4 our WeatherGFM effectively carries out a wide array of weather understanding tasks multi-modal weather data. In practical scenarios, weather forecasting and weather image transformation represent two substantially different tasks due to differences in temporal modalities. Despite their intricacies, our WeatherGFM with in-context learning can successfully recognize distinct task types, highlighting its significant generalization capacity.
4.4 ABLATION STUDIES AND EXPLORATIONS
Exploration of different task prompts. To investigate the impact of various visual prompts on quantitative performance, we randomly select 20 meteorological prompts for each task and calculate their quantitative metrics on the test set. [able(]presentt the standard deviation of performance for each task across the 20 distinct meteorological prompts. We note that weather super-resolution tasks minimally affected by the randomness of weather prompts, whereas weather forecasting tasks and image transformation tasks exhibit more significant variability, reaching approximately 0.02 in CSL. Figure iillustrats that for certain weather events, employing different prompts yields more precise outputs. This indicates that our method can comprehend specific weather cases based on weather prompts rather than being black box model incapable of interactive operations.
Exploration of generalizability. To assess the generalizability of our framework, we also utilize the ERA5 dataset (1.406259) (Hersbach al. 2020). The WeatherGFM model used for the ERA5 comparison is further trained on the original model by incorporating the ERAS data, mixed with the previous tasks, as an additional task for continual training. With the introduction of new meteoro- logical variables, we add a new patch embedding layer for each of these variables to the original model. Following the approach ClimaX (Nguyen al., al.]|202. 2023), we use cross-attention module to aggregate all variable embeddings into a single vector. This allows us to handle the task using the single-modal mode in WeatherGFM. Specifically, we employ an MLP layer to align the embeddings for this task. In line with ClimaX (Nguyen et al. 2023). we use 48 ECMWF (European Centre for Medium- Medium--annge Weather Forecasts) variables as input and evaluate the performance of WeatherGFM using the temperature at 2 meters above ground (T2m). We consider seven lead times: 6 hours and 1, 3, 5, 7 days, covering a range from nowcasting to short- and medium-range forecasting. Instead of training separate models for each target variable, WeatherGFM is trained once to predict all variables across all lead times simultancously. During fine-tuning, we randomize the lead time from and Table show that our approach significantly outperforms ClimaX in 6 hours to 7 days. Table and

---

Published as a conference paper at ICLR 2025
Tarme
Trahh
Tapa
Dhapee
Pa
Oater
Gread
Figure 6: Visual results of our WeatherGFM on OOD tasks.
apsappaation
0.00
2011
622
3003
384
Figure 7: The effect of model sizes. ViT-ST: single-task ViT trained on 0.5 million samples. Base: our WeatherGFM with 100 M parameters trained on 4 million samples. Large: our WeatherGFM with 330 M parameters trained on million samples.
the 120-hour and 168-hour forecast results, even exceeding the ECMWF IFS method. More results can be found in Appendix D
Exploration of aut-of-distribution tasks. To evaluate the generalization ability of our WeatherGFM, we have devised variety of out-of- training phase, including GEOS-IRE 107 extrapolation. weather image translation GEOS-IR IRS70 to GEOS-IRO69 weather temporal SR at 15 minutes and GEOS-visible satellite extrapolation. As shown in Figure 6 our WeatherGFM generates correct outputs for the first three tasks, which are similar to the training distribution. However, the model encounters difficulties with the more challenging task of multiple-modal satellite spatial SR, where its outputs fail to provide effective metcorological information. These OOD tests demonstrate the model's ability to identify tasks outside the training distribution from new prompts, showcasing a degree of generalization.
(OOD) tasks that were not encountered during the
The scaling law for weather foundation models. To evaluate the impact of data and model scale on performance, we compared single-task models, the base version of our WcatherGFM. and its large version. We established baseline using 30M parameter ViT under single-task with 0.5 million samples. Subsequently, in multi-task setting with 4 million samples, our model was configured with a base version of 110M and a large version 330M parameters. Figure 7 illustrates that improvements in performance In specific tasks like radar super-resolution, we observe that scaling up both the data and the model essential for performance gains.
various tasks are achieved with the increase of model and data scale.
CONCLUSION
We introduce WeatherGFM, the first generalist foundation model for weather. By utilizing a unified representation across multiple weather understanding tasks and employing multi-modal prompt design, WeatherGFM effectively addresses range tasks, including weather forecasting, super- resolution, image translation, and post-processing through in-context learning. We conduct com- prehensive explorations of the model's 's adaptability to various tasks, its generalization capabilities to unseen tasks, and its scaling behavior with respect to data and model size. We hope this study paves the way for the development of future large-scale generalist foundation models in weather and climate.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by Shanghai Artificial Intelligence Laboratory and the China Postdoctoral Science Foundation (Grant No. 2024M761547).
REFERENCES
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- arXiv preprint arXiv:2205.01580, 2022.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Accurate medium- range global weather forecasting with 3d neural networks, Nature, 619(7970)5333-538, 2023,
Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: foundation model of the atmosphere. arXiv preprint 2xXv:2245..30663 2024.
Tom B Brown, Language models are few-shot learners. arXiv preprint arXiv:2005. 14100, 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12299-12310, 2021.
Kang Chen, Tao Han, Junchao Gong, Lei Bai, Fenghua Ling, Jing-Jia Luo, Xi Chen, Leiming Ma, Tianning Zhang, Rui Su, et al. Fengwu: Pushing the skillful global medium-rang weather forecast beyond 10 days lead. arXiv preprint arXiv:2304.02946 2023a.
Xiangyu Chen, Xintao Wang, Wenlong Zhang, Xiangtao Kong, Yu Qiao. Jiantao Zhou, and Chao Dong. Hat: Hybrid attention transformer for image restoration. arXiv preprint arXiv:2309.05239 2023b.
Xiangyu Chen, Yihao Liu, Yuandong Pu, Wenlong Zhang, Jiantao Zhou, Yu Qiao, and Chao Dong. Learning a low-level vision generalist via visual task prompt. arXiv preprint arXiv:240S. 14060000 2024a.
Xiangyu Chen, Yihao Liu, Yuandong Pu, Wenlong Zhang, Jiantao Zhou, Yu Qiao, and Chao Dong. Learning a low-level vision generalist via visual task prompt. arXiv preprint 2rXiv:2408.0880 2024b.
Xuanhong Chen, Kairui Feng, Naiyuan Liu, Bingbing Ni, Yifan Lu, Zhengyan Tong, and Ziang Liu. Rainnet: A large-scale imagery dataset and benchmark for spatial precipitation downscaling. Advances in Neural Information Processing Systems, 35:9797-9812 2022.
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi- spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint 00X0222301.00234, 2022.
Zhihan Gao, Xingjian Shi, Boran Han, Hao Wang, Xiaoyong Jin, Danielle Maddix, Yi Zhu, Mu Li, and Yuyang Bernie Wang. Prediff: Precipitation nowcasting with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Junchao Gong, Lei Bai, Peng Ye, Wanghan Xu, Na Liu, Jianhua Dai, Xiaokang Yang, and Wanli Ouyang. Cascast: Skillful high-resolution precipitation nowcasting via cascaded modelling. arXiv preprint arXiv:2402.04220 2024.
Tammay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11953-14961 2023.
11

---

Published as a conference paper at ICLR 2025
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):1999-2000
2020.
Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar. Finding visual task vectors. arXiv preprint 2404::2200.05729, 2024.
Langwen Huang, Lukas Gianinazzi, Yuejiang Yu, Peter D Dueben, and Torsten Hoefler. Diffda: diffusion model for weather-scale data assimilation. arXiv preprint arXi::2401.05932, 2024.
Christoph A Keller, K Emma Knowland, Bryan Dunean, Junhua Liu, Daniel Anderson, Sampa Das, Robert A Lucchesi, Elizabeth W LLudggren, Julie M Nicely, Eric Nielsen, et al. Description of the nasa geos composition forecast modeling system geos-cf v1. O. Journal of Advances in Modeling Earth Systems, 13(4):22020M50024113 2021.
Xuyang Li, Danfeng Hong, and Jocelyn Chanussot. S2mae: A spatial-spectral pretraining foundation model for spectral remote sensing data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24088-24097 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
Mengyao Liu, Jintai Lin, Hao Kong. K Folkert Boersma, Henk Eskes, Yugo Kanaya, Qin He, Xin Tian, Kai Qin, Pinhua Xie, et al. new tropomi product for tropospheric no columns over cast asia with explicit acrosol corrections. Atmospheric Measurement Techniques, 13(8):4247-4259, 2020.
Yihao Liu, Xiangyu Chen, Xianzheng Ma, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Uni- fying image processing as visual prompting question answering. arXiv preprint arXiv:2310.105133 2023a.
Yihao Liu, Xiangyu Chen, Xianzheng Ma, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Uni- fying image processing visual prompting question answering. arXiv preprint vxXiv:2300.105/1 2023b.
Yuanbo Liu, Qiaoni Fu, Ping Song, Xiaosong Zhao, and Cuicui Dou. Satellite retrieval of precipitation: An overview. Advances in Earth Science, 26(11): 1):1 162, 2011.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference Learning Representations, 2022.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26439-26455, 2024.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint 2XX1:23011.10433 2023.
Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Shahbaz Khan. Rethinking transformers pre-training for multi-spectral satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27811 1-27819, 2024.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcast- net: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint ar0iv: 1214, 2022.
Shukla, P. B., Pal, K. P., Joshi, and C. P. Extrapolation of sequence of geostationary satellite images for weather nowcasting. Geoscience and Remote Sensing Letters IEFE, 2011.
12

---

Published as a conference paper at ICLR 2025
Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-41 418, 2020.
Jason Stock, Kyle Hilburn, Imme Ebert-Uphoff, and Charles Anderson. Srvit: Vision transformers for estimating radar reflectivity from satellite observations scale, arXiv preprint arXiv:2406.16955, 2024.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 11888-11898, 2023.
Kevin Trebing, Tomasz Stanczyk, and Siamak Mehrkanoon. Smaat-unet: Precipitation nowcasting using a small attention-unet architecture. Puttern Recognition Letters, 145:178-186, 2021.
Thomas Vandal, Evan Kodra, Sangram Ganguly, Andrew Michaelis, Ramakrishna Nemani, and Auroop R Ganguly. Deepsd: Generating high resolution climate change projections through single image super-resolution. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 1663-1672, 2017.
Mark Veillette, Siddharth Samsi, and Chris Mattioli. Sevir: storm event imagery dataset for deep learning applications in radar and satellite meteorology. Advances in Neural Information Processing Systems, 33:22009-22011 2020.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6830-6839, 2023a.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6830-6839, 2023b.
Li Xu, Bo Liu, Ameer Hamza Khan, Lu Fan, and Xiao-Ming Wu. Multi-modal pre-training for medical vision-language understanding and generation: An empirical study with a new benchmark. In Conference Health, Inference, and Learning, pp. 117-132. PMLR, 2023.
Wenlong Zhang, Guangyuan Shi, Yihao Liu, Chao Dong, and Xiao-Ming Wu. A closer look at blind super-resolution: Degradation models, baselines, and performance upper bounds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 527-536, 2022.
Wenlong Zhang, Xiaohui Li, Xiangyu Chen, Qiao, Xiao-Ming Wu, and Chao Dong. Seal: A frame- work for systematic evaluation of real-world super-resolution. arXiv preprint arXiv:2309.03020
2023.
Wenlong Zhang, Xiaohui Li, Guangyuan Shi, Xiangyu Chen, Yu Qiao, Xiaoyun Zhang, Xiao-Ming Wu, and Chao Dong. Real-world image super-resolution as multi-task learning. Advances in Neural Information Processing Systems, 36, 2024.
Xiangyu Zhao, Bo Liu, Qijiong Liu, Guangyuan Shi, and Xiao-Ming Wu. Easygen: Easing mul- timodal generation with bidiffuser and llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 1351-1370, 2024a.
Xiangyu Zhao, Yuehan Zhang, Wenlong Zhang, and Xiao-Ming Wu. Unifashion: A unified vision- language model for multimodal fashion retrieval and generation. arXiv preprint arXiv:2408. 1305, 2024b.
13

---

Published as a conference paper at ICLR 2025
A DATASETS
SEVIR. The Storm EVent ImageRy dataset (SEVIR) (Veillette al. 2020) is a spatiotemporally aligned dataset that contains over 10,000 weather events represented by five spatially and temporally aligned sensors. These sensors consist of three channels (C02, CO9, C13) from the GOES- 16 satellite, one NEXRAD derived vertically integrated liquid (VIL.) mosaic variable, and lighting detections from the GOES GLM sensor. Each SEVIR event spans 4 hours with 5-minute intervals, sampled randomly (with oversampling of events with moderate and high precipitation) using the NOAA Storm Event Database. In our task, we uniformly resize the resolution of images from different modalities to 256x256. Moreover, we filter the events within the SEVIR dataset and pick out those events that include both the three channels of the GOES- satellite and the one variable derived from weather radar. Ultimately, the dataset utilize comprises 11 11,508 events with four distinct sensing modalities. Among them, 11 11,308 events are selected as the training set, while 100 events are designated as the validation set and 100 events are designated as the test set. Consequently, the training set contains a total of 2.2M images, while the validate/test set has a total of 19.6K images. We provide detailed introduction in Appendix B
POMINO-TROPOME GEOS-CF. In addition, we add a weather image translation task for en- vironment monitoring: Translate geostationary NO,, data to oobar-or satellites NO₂ data (GEOS2POES- (0) based on POMINO-TROPOMI product (Liu et al. 2020) and GEOS-CF dataset (Keller et al., (02T). In this task, the input images are sourced from GEMS as well as the GEOS-CF datasets, while the output images are obtained from the TROPOMI dataset. The original image has a resolution of 1400x800. We also divide it into grids of 256x256 with a sliding step size of 128. Each original image can thus be segmented into 45 pieces of 256x256 pictures. We utilize the observational data from January 2021 to April 2022. After processing, each modality has 20,000 images with a resolution of 256x256. Among them, we allocate 18,000 images as the training set, 1,000 images as the validation set, and 1,000 images as the test set.
ERA5, ERAS (Hersbach et al. 2020), developed by the European Centre for Medium- Rangg Weather Forecasts (ECMWF), is a global atmospheric reanalysis dataset that provides detailed information on the Earth's climate and weather conditions from 1940 to the present. It includes a diverse set of variables such as temperature, humidity, precipitation, wind speed and direction, mean sea level pressure, and more. In line with the ClimaX framework, our input selection consists of 48 variables in total: 6 atmospheric variables across 7 vertical levels, 3 surface variables, and 3 constant fields.
B DETAILS OF WEATHER UNDERSTANDING TASKS
Table 4: Overview of model inputs, outputs, and prompt formats
Task
Dotasee
Proceet Format
Pronupt Laput
Procupt Ocsput
Input
Oonput
Spatial SR
Single Modal
VIL
Sttllltte Spatial
Sevir
Single Modal
BR-089
L8
BR.069
Radie Temporal SR
Sevir
Single Modal
(0,50mmin)
(0.6llmin)
VII. 30mii
Deblur
Sevir
Singls Modal
VIL (Canhhoomerr)
VIL (Gronnd Truth)
VIL (Larhhformer)
VIL (Ground Trath)
GiOOS-IR2Rade
Scvir
Cross Modal
IR-499.
VII.
IR-069.1R- 107
VII.
GEOS IR2GEOS-IR
Sevir
Cross Modal
IR-089
IR- 107
TR. TR-069
IR-107
GEOS IR2GEUS Vis
Sevir
Cross Midal
IR 499
VIS
IR. 069
VIS
GROS:POES-NOX
POMINO
Cross Modal
GEMS, GROS-CF
TROPONE
GIEMS. GROS-CC
TROPOMI
Sucllite extrapolation
Sevir
Time sertes Medal
IR R8-099 1.300,00 60 90min)
IR 069 120, (80min)
IR: 069 (0.30 90min)
069 180min)
Radir extrapolation
Sevir
Modil
(0,30,60
VIL (120, (8018))))))
(0,301,0, (91,.0))Ummm))
VIL. (120, 150mii))
Weather T2M
FRAS
Modal
variables
varisbble
48 variablls
12mm variable
Weather Forecasting U10
ERAS
Cross
variables
U10 vanable
48 vuriables
variable
Weather forecasting, Radar echo extrapolation aims to forecast data for the subsequent 1-2 hours utilizing observations from past moments (Gao et al., 2024). This task, similar to precipitation nowcasting, plays significant role in predicting local weather conditions. It can directly impact traffic plans, disaster warnings, and energy management. Likewise, meteorological satellite image extrapolation is crucial for monitoring and analyzing meteorological conditions. Based on the SEVIR dataset, we consider two weather forecasting tasks: radar echo extrapolation Gong et al. (2024) and satellite image extrapolation (Shukla et al., 2011). Our weather prediction tasks incorporate observations from the hour before (0, 30, 60, and 90 minutes past) and the hour ahead (120 and 180 minutes into the future) for both radar and satellite IR-069 extrapolation. Consequently, for this task,
14

---

Published as a conference paper at ICLR 2025
the SEVIR data was extracted and processed to generate 135,696 sequences for training, along with an independent set of 1,200 sequences to validate/test the fitted model.
Weather super-resolution (SR). Weather spatial super-resolution task (Veillette et al. 2020) gen- crates a high-resolution image from a row-resolution(L) image, while temporal super-resolutin predicts a high-resolution(HHR image based two consecutive observed input images. We take into consideration three weather super-resolution tasks: spatial SR for satellite IR-069, spatial SR for radar VIL, and temporal SR for radar VIL with a one-hour interval. We utilize the SEVIR dataset as the source of the HR image. To obtain the LR image, we employ the "Bicubic" interpolation approach, which is commonly used in vision image SR. In the context of meteorology. this analogous to statistical downscaling, as described in statistical downscaling (Vandal et al.] 2017). Specifically, for the VII. image, given that its original resolution 384x384, we resize it to 256x256 to serve as the HR image and resize it to 64x64 to function as the LR image, thereby implementing 4x super-resolution task. For the IR-069 image, since its original image has resolution of 196x196, we resize it to 256x256 to be the HR image and resize it to 64x64 to be the LR image, thus carrying out a 3x super-resolution task. For each spatial SR for satellite tasks, the SEVIR data was extracted and processed to yield 542,784 images for training, along with an independent set of 4,800 images for validating/testing.. For the weather temporal SR, we use the radar VIL image at 1 hour (0 and 60 minutes) as the input to predict the radar VIL image at 30 minutes. For the temporal SR task, the SEVIR data was further extracted and processed to generate 407,08 007,088 sequences for training, along with an independent set of 3,600 sequences to validate/test the fitted model.
Weather image translation. Weather image translation involves converting observation data (e.g.. satellite data) to a desired weather image (Veillette et al. 2020). For example, depictions of storms obtained from weather radar are extremely important. However, most areas of the world do not have access to ground-based radar. is useful for generating weather radar images of storm depictions from satellite observation (Veillette et al.) (020). We consider three weather image translation tasks based on SEVIR dataset: translate geostationary IR-069 to geostationary IR-107 data (GEOS-R22GOOS- geostationary IR-069 to geostationary Visible data (GEOS-IR2GEOS Vis), translate geostationary IR-069 and IR-107 to radar VIL data (GEOS-IR2Rdda). In addition, we add a weather image translation task for environment monitoring: Translate geostationary NO₂ data to polar-orbiting product (Liu et al. (020).
satellites NO₂ data (GEOS2POES-NO) 2 based on POMINO-TROPOM For the image translation tasks based on SEVIR dataset, we split SEVIR into 542,784 training samples, 4,800 validation samples and 4,800 test samples. For translating geostationary NO₂ data,
Weather post-processing:
Post-processing (e-g., bias correction) aims minimize eliminate systematic biases in model outputs and observational data, which emerge due to uncertainties in weather models and measurement errors. Various methods, including statistical, machine learning, and deep learning techniques, can be employed for post-processing, tailoring the approach based on the specific application and data characteristics. By minimizing or eliminating systematic biases, post-pn post-processing improves the quality and reliability of weather and climate data. In our experiment, we consider a classic post-pre post-processing task: Debluring for radar VIL nowcasting. We employ the output of Earthformer and the corresponding high-quality image as a training sample. Deblurring aims to learn how to map from the output of Earthformer to the corresponding high-quality image.
C IMPLEMENTATION DETAILS
C.I IMPLEMENTATION DETAILS AND HYPERPARAMETERS
The hyperparameters for WeatherGFM in our experiments is shown in Table refhyperparameters. The L1 loss is employed as the loss function. For optimization, the AdamW optimizer with cosine learning rate scheduler is utilized. The base learning rate is lc-4. The batch size is 20 and the accumulation gradient iterations are 4. We use 16 Nvidía A100 GPUs for training. A total of 50 epochs are executed. We leverage fp16 floating point precision in our model.
C.2 VIT HYPERPARAMETERS
We borrow our ViT implementation from (Beyer et al. 2022). We use the following by hyperparameters for ViT in all of our experiments.
15

---

Published as a conference paper at ICLR 2025
Table 5: Default hyperparameters of WeatherGFM
Hyperparameter
Meaning
Large
Base
p
Patch size
16
16
Encoder dimension
Encoder Embedding dimension
1024
768
Decoder dimension
Decoder Embedding dimension
512
512
Encoder depth
Number of Encoder blocks
24
12
Decoder depth
Number of Encoder blocks
8
8
Encoder Heads
Encoder's attention heads
16
12
Decoder Heads
Decoder's attention heads
16
16
MLP ratio
The hidden dimension of the MLP layer in ViT block
4
Masked ratio
Percentage of the masked target data
75%
75%
Table 6: Hyperparameters of VIT
Hyperparameter
Mcaning
Value
p
Patch size
16
Dimension
Embedding dimension
512
Depth
Number of Encoder blocks
16
Heads
Encoder's 's attention heads
8
MLP dim
Encoder's attention heads
1024
C.3
UNET HYPERPARAMETERS
We use the following hyperparameters for UNet in all of our experiments.
C.4
CSI METRIC
The CST (Critical Success Index) is a commonly used metric used in weather understanding tasks (e.g.. precipitation nowcasting). The definition of CSI is:
Hits
CSI
Hits + Misses + F.Alamms
To count the Hits (truth=1, I) Misses (truth=1, pred=0) and F.Alarms (truth=0, pred=1), 1), the prediction and the ground-truth are normalized using mean variiacc normalization and binarized at different thresholds. Following SEVIR (Veillette et al. 2020), for radar output tasks, we have established thresholds at [16, 74, 133, 160, 181, 219]. GEOS-visible output tasks are assigned thresholds of [2000, 3200, 4400, 5600, 6800]. The GEOS-IR100 output tasks operate with thresholds set to [-6000, -4000, 0, 2000]. Lastly, the GEOS -RRO9 output task employs thresholds of [-4000, -5000, -6000, -7000].
Table 7: Hyperparameters of UNet
Hyperparameter
Meaning
Value
Padding size
Padding size of each convolution layer
1
Kernel size
Kernel size of each convolution layer
3
Stride
Stride of each convolution layer Number output channels for Down and Up blocks
1
Channel multiplications
[1,
2, 4, 8, 8]
Blocks
Number of blocks If use attention in Down and Up blocks
3
Use attention
False
Dropout
Dropout rate
0
Inner channel
Number channels in the intermediate layers
64
16

---

Published as a conference paper at ICLR 2025
D EXTENDABILITY
Tables 8 and Table 9 show the comparison of RMSE and ACC with different lead times for IFS, ClimaX, and our WeatherGFM on the variables t2m and U10. Our method achieves comparable results to ClimaX at other lead times. It's worth noting that Climax fine-tunes the pre-train model each lead time, meaning Climax requires models for weather forecasting at lead times. In contrast, our universal model can handle all lead time tasks with just a single model, without the need for task-specific manintnnng. Moreover, the Climax method trained for 100 epochs using 80 V100 GPUs, while our generalist model trained for 20 epochs using 8 A100 GPUs. This indicates that our generalist model converges much faster than Climax. On the other hand, Aurora, focusing on forecasting tasks, employs resolutions of 0.25 and 0.1%, which often produce better quantitative results than 1.4%. For comparison with ClimaX, we selected the 1.4 setting for our atmospheric forecasting experiments. Our generalist model has outperformed ClimaX on multiple variables in these experiments.
Table 8: Comparison of RMSE and ACC across different lead times for IFS, ClimaX, and ours WFAtherGFM on 12m variable. ClimaX views predicting at each lead time as a separate task and fine-tunes separate model for every individual task. In contrast, WeatherGFM utilizes single model to deal with all of these tasks. Aurora is forecasting foundation model that is trained using higher resolution atmospheric data from ERA5 (0.25%), making results not directly comparable.
Lead Time
RMSE
ACC 1
|hr.]
IFS
Aurora (0.25%)
ClimaX
WeatherGFM
IFS
ClimaX
WeatherGFM
6
0.97
0.53
1.11
1.08
0.99
0.98
0.98
24
1.02
0.68
1.19
1.23
0.99
0.97
0.97
72
1.30
0.96
1.47
1.56
0.98
0.96
0,96
120
1.71
1.32
1,83
1.68
0.96
0.94
0.95
168
2.23
1.73
2.17
1.76
0.93
0.91
0.94
Table 9: Comparison of RMSE and ACC across different lead times for IFS, ClimaX,and ours WeatherGFM on U10 variable. WeatherGFM utilizes a single model to deal with different lead time tasks.
Lead Time
RMSE 1
ACC 1
[hr.]
IFS
Aurora (0.25%)
ClimaX
WeatherGFM
IFS
ClimaX
WeatherGFM
6
0.79
0.69
1.04
1.12
0.98
0.97
0.97
24
1.11
0.97
1.31
1.26
0.97
0.95
0.95
72
1.92
1.56
2,02
1.99
0.89
0.87
0.88
120
2.89
2.27
2.79
2.61
0.76
0.74
0.79
168
3.81
2.98
3.35
3.11
0.58
0.59
0.65
E EFFECTS OF MULTI-TASK TRAINING
In order to assess the influence of multi-task training on performance, we contrasted the versions of our WeatherGFM that were trained on 4 tasks and 10 tasks respectively. Additionally, for the purpose of comparison with WeatherGFM, we also made use of the single-task Vision Transformer (VTT). As shown in the table 12] WeatherGFMM4taske SR, GOES2GOES, GOES2Radar, and Radar Spatial SR. It uses more data and encompasses more tasks than the ViT-ST specialized model, which trained on these four tasks separately. However, WeatherGF utilizes less data and fewer tasks compared to WeatherGFM-10ke trained on ten tasks. The results indicate that for radar image generation tasks (Radar Temporal SR, GOES2Radar, and Radar Spatial SR), both WeatherGFM-4ttss and WeatherGFM-100a outperform ViT-ST. This is likely because most of the selected tasks are related to radar image generation. However, for the satellite image generation task (GOES2GOES), WeatherGFM-4tasks
trained on four tasks: Radar Temporal
which
17

---

Published as a conference paper at ICLR 2025
Table Influence of employing different visual prompts on different tasks. The color red used
for the poorest-perfrming prompts, and green is used for the best-performing prompts.
Prompts
Idxo
1dx1
Idx2
Idx3
1dx4
Idx5
Idx6
Idx7
1dx8
Idx9
RMSE
0.4759
0.471
0.5117
0.4748
0.4691
0.4725
0.4726
0.4692
0.4722
0.473
GOESZRalar
0.3452
0.3302
Rodar Extrapolation
RMSE
0.4912
0,4908
0.4891
0.4925
0.4933
0.4937
0,4933
0.493
0.4933
0.4951
CS1
0.3401
0,2497
0.2534
0.2484
0.2467
0.2462
0.2467
0.2453
0.2467
GOES AR2GOES-EI IR
RMSE
0.283
0.2947
0.2593
0.3791
0.2424
0.2989
0.3213
0.2811
0,4372
0.387
CSI
0.6851
0.6907
0.6963
0.6539
0.8958
0.7525
0.6835
0.7314
0.6240
0.6529
0.1334
0.1335
Rodar Spatial SR
CSI 0,7208
0.7206
0.7209
0.7208
0.7228
0.7222
0.7221
0.7227
0.7217
0.7225
RMSE
0.3166
0.3166
0.3164
0.3164
0.3168
0.3172
0.3168
0.3166
0.317
0.3172
Radar Temporal SR
CSI
0.3387
0.3395
0.3389
0.3359
0.3398
0.3366
0.3376
0.3376
0.3372
0.3388
Dehlur
RMSE
0.2272
0,2236
0.2236
0,2258
0.2253
0.2274
0.2273
0.2286
0.2241
0.2233
CSI
0.3412
0.3405
0.3420
0.3413
0. 0.3407
0.3406
0,3408
0.3408
0.3410
0.3419
Prompts
Mdx10
Idx11
Idx12
Idx13
1dx14
Mx115
Idx16
Idx17
Idx18
MX119
RMSE
0.4711
0,4722
0.4731
0,4706
0.4743
0.4752
0,4747
0.4764
0,4735
0.4725
GOESZRalar
CSI
0.3260
0.3252
0.3277
0.3266
0.3264
0.3297
0.3248
0.3247
0.3276
0.3265
Radar
RMSE
0.4927
0,4931
0.4934
0,493
0.4938
0.4929
0,4932
0.4934
0.4937
0,4934
Extrapolation
CSI
0,2478
0,2474
0.2483
0,2483
0.2493
0,2488
0.2474
0.2477
0.2479
0.2474
RMSE
0.4052
0.3017
0.2917
0,3136
0.3027
0,2828
0.2886
0.3189
0.3147
0.3102
GOES IR2GOE--IR
CSI
0.6451
0.7057
0.7034
0.6930
0.7064
0.7044
0.7076
0.6899
0.6982
0.7040
RMSE
0.1332
0.1333
0.1332
0,1332
0.1332
0.1332
0.1332
0.1333
0.1332
0.1332
Radar Spatial
0.7221
0.7218
0.7216
0.7219
0.2218
0.7215
0.7224
0.7213
RMSE
0,3166
0,3168
0,3167
0,3168
0.3166
0,3167
0.3166
0,3169
0.317
0.3166
Radar Temporal SR
0.3375
0.3374
RMSE
0.2252
0,2233
0.2272
0.2257
0.2259
0.2257
0.2264
0.2267
0.2235
0.2263
Dehlur
CSI
0.3405
0.3405
0.3413
0.3417
0.3416
0.3418
0.3416
0.3417
0.3414
0.3415
Table 11: Comparison of the models with random prompts, high-quality prompts, and searched prompts according to RMSE.
Tasks
GOES2Radar
Extrapolation
Metrics
CS177
CSV113
CSI/160
CSV/181
CS1/219
CSL/4
CSI/133
CSI/160
CSV181
CSI/219
maddmm prompts
0.389
0.238
0.194
0.15
0.048
0.423
0.187
0.106
0.069
0.017
high prompts
0.399
0.249
0,208
064
0.057
0.426
0.191
0.11
0.072
0.019
searched prompts
0.401
0.24
0.199
0.155
0.049
0.425
0.188
0.108
0.071
0.018
Table 12: The effect of multi-task training. VIT-ST: single-task VIT. WeatherGFM-4Taske our WeatherGFM trained on 4 tasks. WeatherGFF-IOTTakke our WeatherGFM trained on all tasks.
Tasks
GOES2Radar
Radar Temporal SK
Mesrics
RMSE
CS1/74
CSI/133
CSV/81
CSI/219
RMSE
CS1/74
CSI/133
CSI/881
CS1/19
WeabheGFM-rTaae
0.443
0.353
0.576
0.355
0.209
0.074
NeuberGFM-100Teee
0.436
0.447
0.266
0.157
0.053
0.327
0.597
0.376
0.217
0.073
Ticks
Radar Spatial
GOES2GOES
Metries
RMSE
CS1/4
CSV/13
CSV/181
CSI/119
RMSE
CSI-6K
CSI/4K
CSDD
CSI2K
VIT-ST
0.120
0.820
0.703
0.573
0.387
0.257
0.987
0.972
0.809
0.136
WeatheGFM--Taac
0.120
0.831
0.714
0.574
0.380
0.317
0.994
0.968
0.766
0.148
WeaterGFM 10Tasks
0.121
0.831
0.712
0.570
0.375
0.310
0.993
0.968
0.808
0.222
18

---

Published as a conference paper at ICLR 2025
Prompt Targetl
Ground
Prompt Inputl
Inpul
Outputl
Truth
Prompt Target2
Ground
Prompt Tnput2
Input
Outpur2
Truth
Figure 8: Case studies of our WeatherGFM using different prompts in the GEOS-IR2Radee task.
Reder Extrepatetion
COES2Nwaar
Rudee Temporal Si
Reder Scatiel SRt
6.71
3770
323
0.44
6.0
6-1
3320
VIT.77
Large
Large
3ame
Larpe
VISST
Large
Figure 9: RMSE performance comparison across different model configurations.
does not perform as well. This suggests that multi-task learning of similar tasks can enhance the model's performance on those tasks.
F MORE DETAILS OF SCALING LAW
In Figure we present the results of the RMSE metric that compares single-task models, the base version of our WeatherGFM, and its large version. It can be observed that increasing the capacity of the model generally leads to better performance when the data size remains constant. In contrast, for smaller models, an increase in training data may result in poorer performance. We hypothesize that this could due to the specificity different tasks within the training data, which makes it more challenging for the model to fit effectively.
G EFFECTS OF VISUAL PROMPTS
To assess the effectiveness of visual prompts within the WeatherGFM framework, conducted a series of tests across various scenarios, including GOES2Radar, Radar Extrapolation, GOES- IR2GOES-IR, Radar Spatial SR, Radar Temporal SR, and Deblur tasks. In main paper, detail a thorough process of prompt selection, curating 20 unique prompts for each task and subsequently reporting the most favorable quantitative outcomes. The comprehensive results of these prompt variations on several representative tasks are presented in the tables. By examining Table I0] we observed that prompts have a significant impact on the GOES2Radar and Radar Extrapolation tasks. Consequently, we conducted further experiments on these two tasks. The results are shown in Table 11 In these experiments, "random prompt" refers to prompts selected randomly, while "high prompt" refers to prompts selected from high-quality prompt base derived from radar data. Specifically, we grouped 100 events and selected samples that contained values exceeding a threshold of 50 within each group, totaling 113 samples. Prompts were then randomly selected from this high-quality base for the experiments. "Searched prompt" refers to a method where prompts are selected based on the RMSE metric caleulated from the input images find similar prompts.
19

---

Published as a conference paper at ICLR 2025
Table 13: Quantitative evaluation on OOD tasks. denotes that WeatherGFM has not been trained or fine-tuned for the tasks listed below and conducts generalized inference directly. indicate that UNet and ViT undergo supervised training on the corresponding training dataset.
Tasks
IR107 Sacelitte extraplotion
IR107 IR069
RMSE
CS1/4K
CS10
CS1/2K
CSI/4K
CSU-6K
UNet"
0.991
0.695
0.642
0.074
0.942
0.642
0.910
VII"
0.413
0.899
0,776
0.245
0.212
0.958
0.986
WeatheGFPM
0.389
0.774
Tasks
Temporal SR at 15min
Metrics
RMSE
CSI/16
CS1/74
CSI/133
CSI/160
CSI/181
CSL211
UNa
0.676
0.211
0.627
0.428
0.351
0.262
0.083
VII*
0.761
WeMtherGFM
0.272
0,814
0.703
0.507
0.419
0.336
0.117
Table 14: More quantitative results on weather understanding tasks. We also report the results of the POD and FAR metries.
Downscaling
Forecasting
Task name
Satellite Spatial
Radar Temporal
Radar Spatial SR
Radar Extrapolation
Metries
AVG. POD
AVG. FAR
AVG. POD
AVG. FAR
AVG. POD
AVG. FAR
AVG. POD
AVG. FAR
UNet
1,0000
0.7466
0,4380
0,1738
0.6568
0.1702
0.3207
0.1751
VIT
0,9941
0.0560
0.4520
0.0139
0.7219
0.0047
0.2749
0.0228
WeatherGFM
0.9973
0.05400
0,4362
0.0111
0.7320
0,0039
0,3028
0.01620
Inversion
Forecasting
Task name
GOES2Radar
GOES IR2GOES Visible
GOES2POES NO2
Satellite Extrapolation
Metries
AVG. POD
PVD
FAR
FAR
AVG. POD
AVG. FAR
AVG. POD
AVG. FAR
AVG. POD
AVG. FAR
UNet
0,4691
0,1821
0,4260
0,1358
0.5993
0.2548
1,0000
0.7841
VIT
0.3537
0.0229
0.4626
0.0829
0.5421
0,1686
0.9599
0.3514
WFMtherGFM
0.3524
0.0147
0,3630
0.1388
0.5023
0.0024
0.9578
0.2379
H ADDITIONAL QUANTITATIVE RESULTS
In this section, we present more quantitative results on various weather understanding tasks. Table 13 provides quantitative evaluation on OOD tasks. Table 14 provides a detailed comparison of these metrics for the UNet, ViT, and WeatherGFM models. AVG.POD and AVG.FAR represent the mean scores across different thresholds for various tasks. For radar output tasks, the thresholds are [16, 74, 133, 160, 181, 219]; for GEOS-visible output tasks, the thresholds are [2000, 3200, 4400, 5600, 6800]; and for GEOS-IRO61 output tasks, the thresholds are [-4000, -5000, -6000, -7000].
I MORE VISUAL RESULTS
To comprehensively assess the performance of WeatherGFM, we present range of qualitative visual results across various tasks, including weather forecasting, weather super-resolution, and weather image translation. Additionally, we conduct a comparative evaluation against the unified ViT-large model, well as single-task ViT and UNet models. The visual outputs and comparisons are thoughtfully illustrated in Figure 10 WeatherGFM's proficiency in generating visually appealing outputs is readily evident in the presented results. Notably, the visual quality surpasses that of the baseline models. However, the significance of WeatherGFM's capability extends beyond visual quality. Its distinctive strength lies in its ability effectively handle wide array of image enhancement tasks and image detection. This marks a noteworthy distinction from traditional models, which often struggle to concurrently address such a diverse spectrum of tasks.
20

---

Published as a conference paper at ICLR 2025
Input
U-Net
VIT
WeatherGFM
Ground Truth
Saith estriphbtios Citrgehatin
Roter
Input
U-Net
WMatherGFM
Ground
CHIDNAL NGED
Rader Immpara
CRRN your NO1
Input
U-Net
VIT
WeatherGFM
Ground Truth
Mader Spattal
SaNII SpiiiiS
CRON-HR GEOSIR
Figure 10: Visual results of the weather understanding tasks by our WeatherGFM.
21


Reviews:
[{"review_id": "lifRlf8ZEb", "reviewer": "Reviewer_vbFm", "summary": "This paper introduces WeatherGFM, a weather foundation model inspired by in-context learning approaches used in visual and language models. WeatherGFM aims to address multiple weather-related tasks within a unified framework by converting diverse tasks with various data types into general weather prompts. The model employs a masked modeling approach for training, while fully masking the targets during inference. Experimental results on SEVIR dataset show that WeatherGFM can handle up to ten different weather tasks, including forecasting, super-resolution, image translation, and post-processing. The model also demonstrates its generalization capabilities on unseen tasks.", "questions": "- Why does WeatherGFM show weaker performance in weather image translation compared to other tasks?\n- In Table 1, there is a misuse of double lines for \"GOES2Radar, CSI/219.\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed weather prompt design effectively handles a wide range of weather-related tasks.\n- By utilizing an in-context learning approach, WeatherGFM can generalize to unseen tasks without requiring fine-tuning.\n- WeatherGFM demonstrates its scalability across different model sizes.\n- The model successfully handles ten various tasks on the SEVIR dataset using a single, unified framework.", "weaknesses": "- In Section 3.2, it appears that WeatherGFM uses task-specific patch embedding layers, which may limit the framework's generalization ability. For instance, with the current design, WeatherGFM may struggle to generalize to new, unseen tasks that require novel sensor channels.\n- The impact of increasing dataset size is unclear in Figure 7. Here, the authors compare ViT-ST with WeatherGFM-Base to illustrate the scalability of WeatherGFM. However, since ViT-ST is trained on a single task while WeatherGFM is trained on multiple tasks with more data and parameters, it’s unclear whether the performance gains are due to dataset scalability or multi-task training or larger number of parameters. Additionally, on radar spatial super-resolution (SR), scaling the dataset size or using multi-task training appears to degrade performance. A more detailed analysis and discussion on the effects of dataset scalability would be valuable.\n- It would be beneficial to compare WeatherGFM’s performance against other climate foundation models (e.g., Aurora, ClimaX) using the ERA5 dataset. One of WeatherGFM’s key strengths is its ability to unify multi-task, multi-modal datasets. Thus, similar to Aurora, pretraining WeatherGFM on a combination of ERA5 and other heterogeneous datasets could potentially enhance its performance. It would be impressive if WeatherGFM could achieve comparable results to ClimaX or Aurora without fine-tuning, especially by leveraging a larger dataset and multi-task training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v639ISUjqE", "reviewer": "Reviewer_Broi", "summary": "This paper presented a weather generalist foundation model based on in-context learning, which unified a wide range of weather understanding tasks, like weather forecasting, wealth image translation, wealth super-resolution, deblur, etc.\nIt first defines the visual prompt template and then does in-context learning to achieve such a unified model.\nThe experiments demonstrate that this unified model achieved comparable results with specialist models, e.g., ViT and Unet.", "questions": "What is the motivation for designing Mixed-modal mask modeling? It seems like there should be some references that motivate this design.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A good contribution of this work is to apply in-context learning to weather-understanding tasks for building a unified model and confirming its effectiveness.\n2. The proposed model demonstrates comparable results with specialist models on ten weather understanding tasks,\n3. Mixed-modal mask modeling is interesting.", "weaknesses": "1. The motivation of the training objective in Figure 3 is unclear. The eq.5 is mismatched with Figure 3. There should be more details about the training objective. X_T in eq.5 lacks a definition for your universal representation.\n2. Details missing. For example, the patch embedding layer is task-specific. Does this mean that there is a hard code to select which patch embedding layer is used to handle different tasks or data modalities? If so, can this model be claimed as a unified model?\n3. The motivation of such a unified model is actually unclear to real-world applications. There is no obvious improvement over the conventional specialist model. In addition, there are not many combinations of tasks here. What is the essential advance of the proposed model compared with per-task specialists?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzftIZlYu4", "reviewer": "Reviewer_CC3i", "summary": "The authors propose a generalist foundation model for weather tasks that includes in-context learning that can handle multiple weather tasks.", "questions": "See Weaknesses. \n\nAre there ablations on the length of the input prompt sequence and how that affects performance?\nKeeping text bold in the introduction paragraphs hampers readability.\nLine 159-180: Define what t is and what its range is", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "**(S1)**: The problem of creating a more general weather foundation model is important, and it is valuable that the authors are investigating this area. \n\n**(S2)**: Demonstrating in-context learning is also useful for weather foundation models, although I admit I haven’t looked too closely into whether there exists prior work that tackles this.", "weaknesses": "**(W1)**: Experimental comparisons are missing. The authors don’t seem to compare WeatherGFM on the same tasks used in the Climax or Aurora papers, and it is unclear why this comparison isn’t made. The main claim is that WeatherGFM is a more general foundation model, and yet the only comparison is made with simple task-specific baselines such as a ViT or UNet. There needs to be compelling evidence that WeatherGFM can outperform/compete with Climax or Aurora. Moreover, only 3 qualitative examples on out-of-distribution (OOD) examples are given in Figure 6. Quantitative evaluations on OOD tasks are quite important for a foundation model.\n\n**(W2)**: Experimental results are weak. For the image translation tasks in Table 2, it is unclear that WeatherGFM can outperform baselines on these datasets. The authors do mention that their goal is not to achieve SoTA performance on each task— this is fine, but since comparison with other SoTA methods is missing, the results on the tasks described are not compelling enough. Moreover, in weather forecasting and super-resolution, the performance of the ViT baseline is nearly the same as WeatherGFM, which suggests only a minute (if any) improvement with the author’s designed modifications and pre-training strategy. Moreover, details on how these baseline ViT or UNets were trained are lacking, which are very crucial to determine if the experiments are fair.\n\n**(W3)**: Lack of significant novelty. While the authors do propose changes to the ViT pre-training setup used in other Weather foundation model setups, these are not significant. The authors’ claims of supporting multi-modality isn’t well supported considering other foundation models already do support inputs from different weather sensors and earth observation (EO) systems. The only difference here seems to be satellite imagery which I don’t think is too different from the setup of the other foundation models, and the authors don’t provide any evidence that the techniques of Climax or Aurora wouldn’t also extend to this additional input. Masked image modelling for EO data and designing patch embeddings for different sensory inputs is an established technique, eg: [1].\n\n**(W4)**: Lack of presentation clarity. I think this paper is tackling an important problem, but does not clearly motivate its solution. \n* For example, figure 3 is quite unclear and it is very hard to understand what exactly is going on (in terms of the pre-training strategy or the architectural modifications). The masked/predicted/ground-truth patch disambiguation is not clear to me. It seems that this figure contains multiple subfigures that should clearly be disambiguated and explained.\n* The actual pre-training method is not described extensively. Specifically, lines 236-253 need to be clearly explained and expanded, since this seems to be the crux of the paper’s modifications. Instead, too much space is given to explaining the architecture of a ViT (lines 254-280), which is not required in the main text of the paper.\n* Qualitative outputs from the model are not compared with qualitative outputs from other baselines. The authors demonstrate the outputs of WeatherGFM but don’t compare these on tasks with other weather foundation models. This is related to (W1).\n* Critical details on training and inference are missing. What is the patch size? What kind of compute was used for training? How much compute? Was the dataset combined? Any dataset augmentation? etc. etc. etc. these details are important for reproducibility but are missing from the paper and appendix.\n\nOverall, given the weaknesses describe above, I don't think this paper is ready for acceptance in this conference. \n\nReferences:  \n[1] SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery, NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFUiOWPdnn", "reviewer": "Reviewer_qe7B", "summary": "The paper presents WeatherGFM, a pioneering general-purpose weather foundation model that unifies diverse weather-related tasks through contextual learning. By integrating different weather data formats, WeatherGFM is trained with visual prompts to handle a wide range of tasks, including forecasting, super-resolution, image translation, and post-processing. The model is designed to overcome the limitations of task-specific models by leveraging a single architecture capable of understanding and adapting to varying weather data patterns. Extensive experiments show that WeatherGFM achieves state-of-the-art performance across multiple tasks and exhibits impressive generalization on unseen tasks, demonstrating its potential as a robust, adaptable solution for complex weather modeling challenges.", "questions": "1. Can you explain in detail which part of the module WeatherGFM shows its advantages in various tasks?\n2. Can you simply explain the significance of CSI threshold setting?", "limitations": "", "rating": 10, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1.\tThe proposed WeatherGFM can handle different weather data modalities and task objectives in a unified manner\n2.\tIn the unseen understanding task, the WeatherGFM still shows a certain generalization.\n3.\tThe introduction of the model input format is very clear, and the classification and identification of various weather image tasks are fully explained", "weaknesses": "1.\tA mixed-modal masked image modeling (MMIM) pipeline, which is the core innovation part of the proposed framework of WeatherGFM, has not been introduced in this paper. Other parts of  WeatherGFM architecture are actually closer to the comparative ClimaX model framework. It will lack the innovative elaboration of the model framework.\n2.\tCSI is the main evaluation index in this paper, but ACC is also a very important index to evaluate the correlation of weather. Should this index be taken into account? Moreover, the setting of CSI threshold value needs some explanation in this paper to explain its important reference for this kind of weather situation classification.\n3.\tAccording to the comparative results of Table 2 experiments in this paper, in fact, the basic Vision Transformer can also realize a variety of understanding tasks of various data sets in the form of input format proposed by the author, and the effect of some tasks is better than that of WeatherGFM, so the advantages of this model in multi-task are specifically highlighted.\n4.\tThere are two clerical errors: (a) \"the\" should be capitalized. (b) The basic learning rate in line 366 should be 1e-4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)