WeatherGFM: Learning a Weather Generalist Foundation Model via In-context Learning

## Abstract

Abstract The Earth’s weather system encompasses intricate weather data modalities and diverse weather understanding tasks, which hold significant value to human life.
Existing data-driven models focus on single weather understanding tasks (e.g., weather forecasting).
Although these models have achieved promising results, they fail to tackle various complex tasks within a single and unified model.
Moreover, the paradigm that relies on limited real observations for a single scenario hinders the model’s performance upper bound.
In response to these limitations, we draw inspiration from the in-context learning paradigm employed in state-of-the-art visual foundation models and large language models. In this paper, we introduce the first generalist weather foundation model (WeatherGFM), designed to address a wide spectrum of weather understanding tasks in a unified manner.
More specifically, we initially unify the representation and definition of the diverse weather understanding tasks. Subsequently, we devised weather prompt formats to manage different weather data modalities, namely single, multiple, and temporal modalities. Finally, we adopt a visual prompting question-answering paradigm for the training of unified weather understanding tasks.
Extensive experiments indicate that our WeatherGFM can effectively handle up to ten weather understanding tasks, including weather forecasting, super-resolution, weather image translation, and post-processing. Our method also showcases generalization ability on unseen tasks.

## 1 Introduction

Modeling Earth weather systems involves a series of complex subprocesses that are intended to transform intricate Earth observation data into applications like weather forecasting (Chen et al., ; Bi et al., ), downscaling (Chen et al., ; Xu et al., ; Ling et al., ; Liu et al., ), assimilation (Huang et al., ; Wang et al., ), retrieval (Liu et al., ), and bias correction (Gong et al., ; ). During the past decade, many data-driven methods have been investigated for various weather understanding tasks and delivering desirable performance on specific tasks. For example, recent studies using large-scale data (e.g., ERA5 reanalysis data (Hersbach et al., )) have exceeded the accuracy of conventional numerical weather forecasts.
However, current weather foundational models face challenges regarding generalizability and data scale limitations.
On the one hand, the Earth observation system consists of a variety of observation devices, such as satellites, radar, and weather stations, which produce diverse modalities of data.
Consequently, designing a specific model for a single-task scenario is highly complex, time-consuming, and labor-intensive.
On the other hand, large-scale data in fields such as computer vision can be obtained at a low cost, whereas weather understanding tasks face an intrinsic bottleneck in data scale due to restrictions on individual scenes and single observation devices as shown in Table 1. For instance, local short-term precipitation forecasting models can only utilize a finite range of observational data.

A significant trend in AI research is the development of foundation models, shifting towards large-scale pre-training and in-context learning. This paradigm enables unified processing of a multitude of complex tasks and generalization to unseen tasks. For example, large language models (LLMs) can perform a variety of language-centric tasks (e.g., question answering, machine translation and dialogue state tracking (Feng et al., )) by combining language input-output examples with new query inputs (prompts) without optimizing model parameters (Brown, ). Similarly, vision foundation models (Wang et al., ; Chen et al., ) employ visual prompts with query inputs to carry out diverse image-centric tasks, such as semantic segmentation, depth estimation, and image restoration. These studies highlight the significant potential of generalist foundational models.

The study of foundation models remains largely limited in weather understanding, with the majority focused on Computer Vision and Natural Language Processing. While there has been some progress with large foundation models in weather and climate, the focus is mainly on weather forecasting and downscaling tasks. For example, Climax (Nguyen et al., ) uses a pre-training-finetuning paradigm for weather forecasting and downscaling. Aurora (Bodnar et al., ) employs LoRA to unify weather forecasting and quick prediction of atmospheric chemistry. However, as shown in Table 1, these studies do not take into account the modeling of multi-modalities and multi-tasks. This poses a challenge: Is it possible to design a universal foundation model capable of handling the variety of complex weather understanding tasks and data modalities?

**Table 1: Comparison of existing task-specific and general models across Earth science and computer vision fields. Our WeatherGFM demonstrates its ability to handle multi-tasks, multi-modal data, and general capabilities, showcasing its strength in acquiring hard-to-access weather data.**
| Category | Method | Data Acquisition<br>Difficulty | Supported Tasks | Multi-tasks<br>support? | Multi-modal<br>support? | Generalist<br>support? |
| --- | --- | --- | --- | --- | --- | --- |
| Computer<br>Vision | HAT (Chen et al., ) | Low-cost | Image super-resolution (SR) | ✗ | ✗ | ✗ |
| IPT (Chen et al., ) | Low-cost | Image restoration, Derain, Dehaze | ✓ | ✗ | Requires fine-tuning |  |
| Painter (Wang et al., ) | Low-cost | Image restoration<br>Segmentation, Keypoint detection | ✓ | ✗ | ✓ |  |
| PromptGIP (Liu et al., ) | Low-cost | Image restoration, Derain, Dehaze | ✓ | ✗ | ✓ |  |
| GenLV (Chen et al., ) | Low-cost | Image restoration, enhancement, translation | ✓ | ✗ | ✓ |  |
| Earth<br>Science | Prediff (Gao et al., ) | High-cost | Weather forecasting | ✗ | ✗ | ✗ |
| Cascast (Gong et al., ) | High-cost | Post-processing | ✗ | ✗ | ✗ |  |
| Climax (Nguyen et al., ) | High-cost | Weather forecasting, Weather image SR | ✓ | ✗ | Requires fine-tuning |  |
| Aurora (Bodnar et al., ) | High-cost | Weather forecasting<br>Atmospheric chemistry prediction | ✗ | ✓ | Requires fine-tuning |  |
| WeatherGFM<br>(ours) | High-cost | Weather forecasting, Weather image SR<br>Weather image translation, Post-processing | ✓ | ✓ | ✓ |  |

In this paper, we first propose a weather generalist foundation model, WeatherGFM, to uniformly address a variety of complex weather understanding tasks and data modalities.
Unlike prior studies that focused on weather forecasting, our proposed method can expand the task scope to weather forecasting, weather super-resolution (i.e., weather downscaling), weather image translation (i.e., synthetic weather radar) (Veillette et al., ), and post-processing (Gong et al., ).
These tasks all belong to the domain of weather understanding, but their modalities are distinct.
Specifically, Sequence modal data can be utilized for weather forecasting, such as short-term predictions based on radar data. Multi-modal data can be employed for weather image translation, such as converting multi-modal satellite data to generate radar data. Single-modal data can be applied to various common scenarios, such as radar image super-resolution and post-processing.
To unify the diverse weather data modalities into a general representation, we introduce a weather prompt format that assigns different prompt phrases to various modalities.
By leveraging in-context learning, our WeatherGFM achieves a promising in-context ability on both various seen tasks and unseen tasks. The significance of our work can be summarized as:

- •
We propose the first weather generalist foundation model (i.e., WeatherGFM), which can handle more than ten weather understanding tasks.
- •
Our weather prompt design supports a diversity of weather data modalities, including time-series, multi-modal, and single-modal data.
- •
Our WeatherGFM with in-context learning first demonstrates the generalization ability to unseen weather understanding tasks.

## 2 Related Work

Weather understanding and beyond.
Over the past decade, machine learning techniques (Zhang et al., ; ; ; Wang et al., ) applied in vision tasks have continuously drawn attention within the field of weather and climate. Numerous data-driven machine learning models have been proposed to address classical tasks in weather understanding (Veillette et al., ), such as forecasting, super-resolution, image translation, and post-processing. Weather forecasting (Bi et al., )) aims to predict future observations from past data. Weather super-resolution tasks, i.e., weather downscaling,  Chen et al. () focus on recovering high-resolution data from low-resolution observations. Weather image translation tasks (Stock et al., ) involves converting existing observational data into desired target modalities, such as transforming satellite observations into ground-based weather radar data. Post-processing tasks seek to enhance existing model results, such as bias correction and deblurring (Gong et al., ; ).

Despite significant advancements, current methods often rely on specialized datasets and customized single-task models for certain scenarios. Consequently, single-task models struggle to exhibit strong generalization abilities and fail to capture the interconnections between diverse tasks, which hinders the establishment of simulations for the Earth system.

Weather foundation model.
The rise of foundation models in Natural Language Processing and computer vision has sparked interest in their application for weather and climate. Large foundation models, enhanced through pre-training, improve the generalization of AI climate models and can be fine-tuned for specific tasks.  Pathak et al. () proposed FourCastNet, a climate pre-trained model using Vision Transformer for high-resolution predictions and rapid inference through self-supervised pre-training and autoregressive fine-tuning. Pangu-Weather (Bi et al., ) utilizes a 3D Earth-specific Transformer for accurate global predictions. FengWu series (Chen et al., ; Han et al., ) solve the forecast problem from a multi-modal and multi-task perspective. ClimaX (Nguyen et al., ) introduces supervised pre-training to weather prediction, offering flexibility for diverse forecasting tasks. W-MAE (Man et al., ) employs unsupervised training via a Masked Auto-Encoder, while MetePFL (Chen et al., ) and FedWing (Chen et al., )advocate for prompt-based federated learning to enhance collaborative training while protecting privacy. Recent advancements, such as OceanGPT (Bi et al., ), extend LLM capabilities to ocean-related tasks, and ClimateBERT (Webersinke et al., ) focuses on processing climate-related texts from extensive sources.

Visual in-context learning.
In recent advancements, visual in-context learning has emerged as a promising research area, inspired by the success of language models like GPT-3  (Brown, ). These models adapt to various NLP tasks using prompts or in-context examples without extensive retraining. Similarly, in the vision domain, models such as MAE-VQGAN (Hojel et al., ) and Painter  (Wang et al., ) have begun exploring in-context learning. However, challenges persist, especially in low-level tasks requiring detailed pixel manipulation. To address this, PromptGIP (Liu et al., ) and GenLV have incorporated in-context learning concepts into their designs to unify low-level vision tasks with diverse input and output modalities, aiming to develop generalist models.
Vision-language models (Liu et al., ; ; Zhao et al., ; ) like Unified-IO (Lu et al., ) and Unified-IO 2 (Lu et al., ) have made significant progress in integrating multiple tasks, highlighting the potential for unified approaches across modalities. Additionally, compositional visual reasoning, exemplified by Visual Programming (Gupta & Kembhavi, ), aligns with in-context learning goals by emphasizing visual task synthesis. ViperGPT (Surís et al., ) further demonstrates foundational models for visual reasoning, employing computational techniques similar to our objectives, though without relying on programmatic inputs. These collective efforts pave the way for more sophisticated and versatile visual in-context learning frameworks.

## 3 Method

### 3.1 Unified representation of weather understanding tasks.

Figure: Figure 1: Illustration of the unified representation for weather understanding tasks.
Refer to caption: x1.png

Weather understanding tasks involve processing multi-source observational data (Veillette et al., ), such as geostationary satellites (GEOS), polar-orbiting satellites (POES), weather radars, and ground observation stations. Each task (e.g., weather forecasting, spatial and temporal super-resolution, weather image translation, and post-processing) utilizes different types of input and output data. To address this challenge, we first developed a unified data representation that can standardize these diverse tasks. Unlike traditional methods that rely on task-specific models for each distinct task, we introduce a universal foundational model capable of addressing various weather understanding tasks through a single and general solution.

As shown in Figure 1, several key weather understanding tasks can be framed using different types of input and output data. For instance, the weather spatial super-resolution (SR) task generates a high-resolution image $x_{HR}$ from a low-resolution image $x_{LR}$, while weather temporal super-resolution predicts a high-resolution image $x^{t}_{HR}$ based on two consecutive observed input images $x^{t-1}_{LR}$ and $x^{t+1}_{LR}$. Weather forecasting relies on a sequence of observed data points $\{x^{1},x^{2},\ldots,x^{t}\}$ to predict future data points $\{x^{t+1},x^{t+2},\ldots\}$. the image translation task focuses on converting an input image from one modality (e.g., satellite image) to another modality (e.g., radar image). Formally, we can represent these tasks as projections from the source input data $X_{S}$ to the target output data $X_{T}$:

$$ $\tau:X_{S}\xrightarrow{}X_{T}.$ (1) $$

When $X_{S}=x_{LR}$ and $X_{T}=x_{HR}$, the task corresponds to spatial SR. Similarly, when $X_{S}=\{x^{1},x^{2},\ldots,x^{t}\}$ and $X_{T}=\{x^{t+1},x^{t+2},\ldots\}$, the task represents weather forecasting. As these tasks differ in their input and output formats, as well as sequence lengths, the key challenge lies in unifying them within one coherent data representation.

### 3.2 WeatherGFM: Weather Generalist Foundation Model

We present the Weather Generalist Foundation Model (WeatherGFM) to tackle the challenges inherent in a range of weather understanding tasks. Through in-context learning, our WeatherGFM can uniformly handle various weather understanding tasks involving multiple data modalities.

Weather prompt designing.
In large language models and vision foundation models, task prompts commonly provide specific task-related input-output pairs. As shown in Figure 2, in machine translation (Stahlberg, ), the model is given English to French text pairs as prompts. The model can perform machine translation tasks based on these sample prompts for a given input. In visual tasks (Wang et al., ), the visual prompt image1 may be a natural image, and image2 is the corresponding segmented image. The model will conduct the segmentation task for a new input image3 to obtain the segmented image.

Figure: Figure 2: Comparison of weather prompts with text and visual prompts design.
Refer to caption: x2.png

Following this paradigm, we designed weather prompts for weather understanding tasks. Since the input for weather understanding tasks involves multiple modalities, such as a single weather observation variable, multiple different weather variables, and time-series weather variables, we proposed three prompts to handle different modalities of input.
In Figure 2, weather prompt1 is similar to visual prompts, converting a single modality image into a target image. In weather prompt2, the input modality can be two different channel satellite observation images (e.g., IR069 and IR107 data), and the output can be weather radar observation data for image translation tasks. In weather prompt3, time-series prompts can be input to perform weather forecasting-related tasks. With these forms of prompt design, our method can handle most weather understanding tasks.

Weather in-context learning.
Inspired by the success of in-context learning in large language models (Dong et al., ) and vision foundation models (Wang et al., ), we propose to unify the weather understanding problem as the visual prompting question-answer paradigm, as illustrated in Eq. 2. Specifically, given a visual question-answer prompt pair $(P_{in},P_{out})$ as a task-guided prompt and a query input $X_{in}$, the model is expected to perceive the context of the prompt (i.e., what task it represents). Consequently, the model can perform the corresponding operations on the query with the prompt. This process can be formulated as follows:

$$ $X_{out}=F_{\tau}(X_{in},P_{in},P_{out};\theta);$ (2) $$

where $F_{\tau}$ represents a universal foundation model parameterized by $\theta$.
We can determine what task will be performed on the query sample by selecting the task-specific prompt.

Figure: Figure 3: Overall approach of our weather generalist foundation model (WeatherGFM).
Refer to caption: x3.png

Mixed-modal mask modeling.
Upon redefining the output spaces of the aforementioned representative vision tasks, it is observed that both the input and output of these tasks are in the form of images as transformers-based architectures could provide flexibility by treating the image-like data as a set of tokens. Therefore, we build the WeatherGFM architecture on Vision Transformers (ViT) and propose a mixed-modal masked image modeling (MMIM) pipeline to train multiple weather understanding tasks as shown in Figure 3. Our WeatherGFM comprises three main elements: the format for input data, the architectural design, and loss function.

Input format: Given an input of shape $(C,H,W)$, ViT predicts an output of shape $(C^{{}^{\prime}},H^{{}^{\prime}},W^{{}^{\prime}})$, where $C$ represents the input channels and $C^{{}^{\prime}}$ represents the output channels. As shown in Figure 3, different tasks have different channels. The model tokenizes the input into a sequence of patches, with each patch having a size of $C\times p^{2}$, where $p$ is the patch size. Unlike RGB-based image data, where the channels are fixed, the number of physical variables in climate and weather data can vary between different datasets and tasks. To adapt the ViT to different weather-related downstream tasks, we designed task-specific patch embedding layers within the architecture. After the patch embedding layer, we use an MLP layer to align the embeddings of different tasks to the same space:

$$ $\displaystyle\mathrm{z_{C}}$ $\displaystyle=\mathrm{PatchEmbed_{C}(x)},\mathrm{x}\in\mathbb{R}^{C\times H \times W},$ $\displaystyle\mathrm{z_{C}}\in\mathbb{R}^{N\times D},$ (3) $\displaystyle\mathrm{z_{0}}$ $\displaystyle=\mathrm{MLP_{C}(LN(\mathrm{z_{C}}))},$ $\displaystyle\mathrm{z_{0}}\in\mathbb{R}^{N\times D}$ $$

where $\mathrm{N,D}$ denotes the number of input tokens and the transformer dimension, respectively. For the masked area, we follow previous works (Liu et al., ) to use a learnable token vector to replace each masked patch. We adopt the block-wise masking strategy, taking the masking ratio as 75%.

Architecture: A vanilla vision Transformer (ViT) is adopted as the backbone architecture. It consists of task-specific patch-embedded layers and several alternating layers made of Multi-Head Self-Attention (MHSA) and MLP blocks. Layer Normalization (LN) is applied before every block, and residual connections are applied after every block. As shown in Figure 3, this process can be formulated as follows:

$$ $\displaystyle\mathrm{z_{\ell}^{{}^{\prime}}}$ $\displaystyle=\mathrm{MHSA(LN(\mathrm{z_{\ell-1}}))}+\mathrm{z_{\ell-1}},\ell= 1...L,$ (4) $\displaystyle\mathrm{z_{\ell}}$ $\displaystyle=\mathrm{MLP(LN(\mathrm{z_{\ell}^{{}^{\prime}}}))}+\mathrm{z_{ \ell}^{{}^{\prime}}},\ell=1...L,$ $$

where L denotes the number of layers. After the attention layers, we employ a prediction head and then unpatchify the output of the prediction head. The prediction head is a one-layer MLP with a hidden dimension of 1024.

Loss Function: After generate the output $X_{out}$, optimization objectives are as follows:

$$ $\displaystyle\mathrm{L}_{\theta}(X_{out},X_{T})$ $\displaystyle=\left\|X_{out}-X_{T}\right\|^{2}.$ (5) $$

where we use MSE (mean square error) loss to train the weather generalist foundation model.

## 4 Experiments

### 4.1 Weather Understanding Tasks.

To show the versatility of our proposed method, we incorporate up to 10 tasks including diverse weather forecasting,
weather super-resolution, weather image translation and weather post-processing tasks into our experiments.

SEVIR.
The Storm EVent ImageRy dataset (SEVIR) (Veillette et al., ) is a spatiotemporally aligned dataset that contains over 10,000 weather events represented by five spatially and temporally aligned sensors. These sensors consist of three channels (C02, C09, C13) from the GOES-16 satellite, one NEXRAD derived vertically integrated liquid (VIL) mosaics variable, and lighting detections from the GOES GLM sensor. Each SEVIR event spans 4 hours with 5-minute intervals, sampled randomly (with oversampling of events with moderate and high precipitation) using the NOAA Storm Event Database. The SEVIR benchmark supports scientific research on multiple meteorological applications, including future prediction, image-to-image translation, super-resolution, etc. A detailed description of these tasks can be found in Appendix A.
In our task, given that the WeatherGFM has the ability to generate images of diverse modalities, we uniformly sample the resolution of images from different modalities to 256×256. Moreover, we perform filtering on the events within the SEVIR dataset and pick out those events that include both the three channels of the GOES-16 satellite and the one variable derived from weather radar. Ultimately, the dataset we utilize comprises 11,508 events with four distinct sensing modalities. Among them, 11,308 events are selected as the training set, while 100 events are designated as the validation set and 100 events are designated as the test set. Each SEVIR event covers a period of 4 hours with increments of 5 minutes, which implies that there are 49 images for each modality within a single event. Consequently, the training set contains a total of 2.2M images, while the validate/test set has a total of 19.6K images. Regarding the demarcation of the training set and the test set for each task, we provide a detailed introduction in Appendix A.

POMINO-TROPOMI, GEOS-CF.
In addition, we add a weather image translation task for environment monitoring: Translate geostationary $NO_{2}$ data to polar-orbiting satellites $NO_{2}$ data (GEOS2POES-$NO_{2}$) based on POMINO-TROPOMI product (Liu et al., ) and GEOS-CF dataset (Keller et al., ). In this task, the input images are sourced from GEMS as well as the GEOS-CF datasets, while the output images are obtained from the TROPOMI dataset.
The original image has a resolution of 1400×800. We also divide it into grids of 256×256 with a sliding step size of 128. Each original image can thus be segmented into 45 pieces of 256×256 pictures.
We utilize the observational data from January 2021 to April 2022. After processing, each modality has 20,000 images with a resolution of 256×256. Among them, we allocate 18,000 images as the training set, 1,000 images as the validation set, and 1,000 images as the test set.

**Table 2: Quantitative results on weather understanding tasks. #: single-task model. †: trained with all ten weather understanding tasks. RMSE and CSI are calculated as the quantitative metric. A lower RMSE and a higher CSI indicate better results. The best results are highlighted in bold, and the second-best results are underscored.**
|  | Weather super-resolution (SR) |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Task name | Satellite Spatial SR | Radar Temporal SR | Radar Spatial SR |  |  |  |  |  |  |  |  |
| Metrics | RMSE | CSI/-4000 | CSI/-6000 | RMSE | CSI/74 | CSI/160 | CSI/219 | RMSE | CSI/74 | CSI/160 | CSI/219 |
| UNet^# | 0.932 | 0.650 | 0.912 | 0.739 | 0.485 | 0.182 | 0.034 | 0.650 | 0.675 | 0.400 | 0.184 |
| ViT^# | 0.047 | 0.987 | 0.990 | 0.333 | 0.591 | 0.285 | 0.061 | 0.120 | 0.830 | 0.637 | 0.358 |
| WeatherGFM^† | 0.042 | 0.988 | 0.996 | 0.327 | 0.597 | 0.287 | 0.073 | 0.121 | 0.831 | 0.644 | 0.375 |
|  | Weather Forecasting | Post-processing |  |  |  |  |  |  |  |  |  |
| Task name | Satellite extrapolation | Radar extrapolation | Deblur |  |  |  |  |  |  |  |  |
| Metrics | RMSE | CSI/-4000 | CSI/-6000 | RMSE | CSI/74 | CSI/160 | CSI/219 | RMSE | CSI/74 | CSI/160 | CSI/219 |
| UNet^# | 1.033 | 0.617 | 0.900 | 0.815 | 0.353 | 0.082 | 0.007 | 0.713 | 0.457 | 0.145 | 0.027 |
| ViT^# | 0.408 | 0.840 | 0.943 | 0.490 | 0.440 | 0.079 | 0.007 | 0.163 | 0.594 | 0.291 | 0.104 |
| WeatherGFM^† | 0.347 | 0.863 | 0.951 | 0.467 | 0.465 | 0.128 | 0.021 | 0.264 | 0.629 | 0.255 | 0.082 |
|  | Weather image translation |  |  |  |  |  |  |  |  |  |  |
| Task name | GOES2Radar | GOES-IR2GOES-IR |  |  |  |  |  |  |  |  |  |
| Metrics | RMSE | CSI/16 | CSI/74 | CSI/160 | CSI/181 | CSI/219 | RMSE | CSI/-6000 | CSI/-4000 | CSI/0 | CSI/2000 |
| UNet^# | 0.821 | 0.222 | 0.370 | 0.180 | 0.153 | 0.079 | 0.915 | 0.929 | 0.741 | 0.638 | 0.078 |
| ViT^# | 0.445 | 0.602 | 0.436 | 0.180 | 0.131 | 0.042 | 0.257 | 0.987 | 0.972 | 0.809 | 0.136 |
| WeatherGFM^† | 0.436 | 0.619 | 0.447 | 0.208 | 0.157 | 0.053 | 0.310 | 0.993 | 0.968 | 0.808 | 0.222 |
| Task name | GOES-IR2GOES-Visible | GOES2POES-$NO_{2}$ |  |  |  |  |  |  |  |  |  |
| Metrics | RMSE | CSI/2000 | CSI/3200 | CSI/4400 | CSI/5600 | CSI/6800 | RMSE | CSI/1 | CSI/5 | CSI/10 | CSI/15 |
| UNet^# | 0.915 | 0.422 | 0.285 | 0.179 | 0.100 | 0.040 | 0.866 | 0.799 | 0.360 | 0.274 | 0.202 |
| ViT^# | 0.448 | 0.574 | 0.437 | 0.303 | 0.184 | 0.071 | 0.549 | 0.841 | 0.432 | 0.328 | 0.253 |
| WeatherGFM^† | 0.439 | 0.580 | 0.439 | 0.298 | 0.166 | 0.068 | 0.302 | 0.682 | 0.562 | 0.382 | 0.197 |

### 4.2 Implementation and Evaluation

Training details. During training, we resize the weather images of different resolutions to a resolution of 256$\times$256 and input them into the model in accordance with the combination mode of $P_{in},P_{out},X_{in},X_{out}$ in the task-specific prompt format, resulting in a N × 256 × 256 total input resolution. The L1 loss is employed as the loss function. For optimization, the AdamW optimizer with a cosine learning rate scheduler is utilized. The base learning rate is 1e4. The batch size is 20.

Evaluation metrics.
Besides RMSE, we also include the Critical Success Index (CSI), which is commonly used in weather understanding tasks (e.g., precipitation nowcasting) and is defined as

$$ $\text{CSI}=\frac{\text{Hits}}{\text{Hits}+\text{Misses}+\text{F.Alarms}}$ $$

To count the Hits (truth=1, pred=1), Misses (truth=1, pred=0) and F.Alarms (truth=0, pred=1), the prediction and the ground-truth are normalized using mean-variance normalization and binarized at different thresholds. For radar output tasks, we have established thresholds at [16, 74, 133, 160, 181, 219]. GEOS-visible output tasks are assigned thresholds of [2000, 3200, 4400, 5600, 6800]. The GEOS-IR107 output tasks operate with thresholds set to [-6000, -4000, 0, 2000]. Lastly, the GEOS-IR069 output task employs thresholds of [-4000, -5000, -6000, -7000].

### 4.3 Experimental results

Currently, there is no general weather foundation model that can comprehensively handle all the discussed weather understanding tasks simultaneously.
Although many machine learning methods have been investigated for single tasks, they generally adopt different backbone networks and design strategies tailored to them.
For a fair comparison, we have trained a series of baselines (i.e., single-task model) for each weather understanding task under a consistent training setup, including commonly used UNet (Trebing et al., ) and ViT (Nguyen et al., ) networks.
Notably, the purpose of this paper is not to achieve state-of-the-art performance on every task.
We focus on examining whether a generalist foundation model can handle multiple complex weather understanding tasks and weather data modalities. Beyond quantitative performance results, we are more concerned with the prompt learning capabilities of the generalist foundation model and the generalization ability it brings.

**Table 3: Standard deviation of the performance computed based on 20 different prompts. Avg. CSI denotes the mean of the CSI score across thresholds [16, 74, 133, 160, 181, 219].**
|  | GOES2Radar | Radar extrapolation | GOES-IR2GOES-IR | Radar Spatial SR | Radar Temporal SR | Deblur |
| --- | --- | --- | --- | --- | --- | --- |
| Avg. RMSE | 0.0087 | 0.0012 | 0.0481 | 0.0001 | 0.0002 | 0.0016 |
| Avg. CSI | 0.0187 | 0.0201 | 0.0284 | 0.0006 | 0.0010 | 0.0047 |

Figure: Figure 4: Case studies of our WeatherGFM with different prompts in the radar extrapolation task.
Refer to caption: x4.png

Weather generalist foundation model can achieve strong universal capabilities. As seen in Table 2, our WeatherGFM, equipped with a straightforward ViT backbone, shows impressive performance and adaptability in ten weather understanding tasks. It is not only capable of conducting weather forecasting and super-resolution tasks but is also proficient in dealing with weather image translation and post-processing tasks. Overall, our WeatherGFM achieves promising performance on a diversity of weather understanding tasks.

Weather generalist foundation model outperforms the performance of the single-task model. In Table 2, we notice that our WeatherGFM achieves results that outperform the baseline in weather forecasting, weather super-resolution, and image translation tasks. For instance, in radar extrapolation tasks, our WeatherGFM with universal ViT-based model outperforms the single-task ViT model. This indicates that a unified approach to weather understanding tasks can potentially break the performance upperbound of single-task models.

In-context learning can generate correct outputs across a variety of data modalities and tasks. As depicted in Figure 5, our WeatherGFM effectively carries out a wide array of weather understanding tasks on multi-modal weather data. In practical scenarios, weather forecasting and weather image transformation represent two substantially different tasks due to differences in temporal modalities. Despite their intricacies, our WeatherGFM with in-context learning can successfully recognize distinct task types, highlighting its significant generalization capacity.

Figure: Figure 5: Visual results of the weather understanding tasks by our WeatherGFM.
Refer to caption: x5.png

### 4.4 Ablation Studies and Explorations

Exploration of different task prompts.
To investigate the impact of various visual prompts on quantitative performance, we randomly select 20 meteorological prompts for each task and calculate their quantitative metrics on the test set. Table 3 presents the standard deviation of performance for each task across the 20 distinct meteorological prompts. We note that weather super-resolution tasks are minimally affected by the randomness of weather prompts, whereas weather forecasting tasks and image transformation tasks exhibit more significant variability, reaching approximately 0.02 in CSI. Figure 4 illustrates that for certain weather events, employing different prompts yields more precise outputs.
This indicates that our method can comprehend specific weather cases based on weather prompts rather than being a black box model incapable of interactive operations.

Exploration on out-of-distribution tasks.
To evaluate the generalization ability of our WeatherGFM, we have devised a variety of out-of-distribution (OOD) tasks that were not encountered during the training phase, including GEOS-IR107 extrapolation, weather image translation GEOS-IR107 to GEOS-IR069, weather temporal SR at 15 minutes and GEOS-visible satellite extrapolation. As shown in Figure 6, our WeatherGFM generates correct outputs for the first three tasks, which are similar to the training distribution.
However, the model encounters difficulties with the more challenging task of multiple-modal satellite spatial SR, where its outputs fail to provide effective meteorological information.
These OOD tests demonstrate the model’s ability to identify tasks outside the training distribution from new prompts, showcasing a degree of generalization.

Figure: Figure 6: Visual results of our our WeatherGFM on OOD tasks.
Refer to caption: x6.png

Scaling law of weather foundation model.
To evaluate the impact of data and model scale on performance, we compared single-task models, the base version of our WeatherGFM, and its large version. We established a baseline using a 30M parameter ViT under a single-task with 0.5 million samples. Subsequently, in a multi-task setting with 4 million samples, our model was configured with a base version of 110M and a large version of 330M parameters. Figure 7 illustrates that improvements in performance on various tasks are achieved with the increase of model and data scale. In SR task, we observe that scaling up both the data and the model is essential for performance gains.

Figure: Figure 7: The effect of dataset size and model sizes. ViT-ST: single-task ViT trained on 0.5 million samples. Base: our WeatherGFM with 100 M parameters trained on 4 million samples. Large: our WeatherGFM with 330 M parameters trained on 4 million samples.
Refer to caption: x7.png

## 5 Conclusion

We introduce the first weather generalist foundation model, WeatherGFM. By employing a unified representation for multiple weather understanding tasks and a multi-modal prompt design, our WeatherGFM skillfully addresses various tasks, such as weather forecasting, super-resolution, image translation, and post-processing through in-context learning. We conduct comprehensive explorations of the model’s adaptability for various tasks and its generalization capabilities to unseen tasks, and its scaling law at the data and model size. This study will facilitate the development of future large-scale generalist weather and climate foundation models.

## Appendix A Details of Weather Understanding Tasks

Weather forecasting. Radar echo extrapolation aims to forecast data for the subsequent 1-2 hours utilizing observations from past moments (Gao et al., ). This task, similar to precipitation nowcasting, plays a significant role in predicting local weather conditions. It can directly impact traffic plans, disaster warnings, and energy management. Likewise, meteorological satellite image extrapolation is crucial for monitoring and analyzing meteorological conditions.
Based on the SEVIR dataset, we consider two weather forecasting tasks: radar echo extrapolation Gong et al. () and satellite image extrapolation (Shukla et al., ).
Our weather prediction tasks incorporate observations from the hour before (0, 30, 60, and 90 minutes past) and the hour ahead (120 and 180 minutes into the future) for both radar and satellite IR-069 extrapolation. Consequently, for this task, the SEVIR data was extracted and processed to generate 135,696 sequences for training, along with an independent set of 1,200 sequences to validate/test the fitted model.

Weather super-resolution (SR).
Weather spatial super-resolution task (Veillette et al., ; Hu et al., ; ; Li et al., ) generates a high-resolution image from a low-resolution(LR) image, while temporal super-resolution predicts a high-resolution(HR) image based on two consecutive observed input images.
We take into consideration three weather super-resolution tasks: spatial SR for satellite IR-069, spatial SR for radar VIL, and temporal SR for radar VIL with a one-hour interval.
We utilize the SEVIR dataset as the source of the HR image. To obtain the LR image, we employ the “Bicubic” interpolation approach, which is commonly used in vision image SR (Zhang et al., ; ). In the context of meteorology, this is analogous to statistical downscaling, as described in statistical downscaling (Vandal et al., ).
Specifically, for the VIL image, given that its original resolution is 384×384, we resize it to 256×256 to serve as the HR image and resize it to 64×64 to function as the LR image, thereby implementing a 4x super-resolution task. For the IR-069 image, since its original image has a resolution of 196×196, we resize it to 256×256 to be the HR image and resize it to 64×64 to be the LR image, thus carrying out a 3x super-resolution task.
For each spatial SR for satellite tasks, the SEVIR data was extracted and processed to yield 542,784 images for training, along with an independent set of 4,800 images for validating/testing.
For the weather temporal SR, we use the radar VIL image at 1 hour (0 and 60 minutes) as the input to predict the radar VIL image at 30 minutes. For the temporal SR task, the SEVIR data was further extracted and processed to generate 407,088 sequences for training, along with an independent set of 3,600 sequences to validate/test the fitted model.

Weather image translation.
Weather image translation involves converting observation data (e.g., satellite data) to a desired weather image (Veillette et al., ).
For example, depictions of storms obtained from weather radar are extremely important. However, most areas of the world do not have access to ground-based radar.
It is useful for generating weather radar images of storm depictions from satellite observation (Veillette et al., ).
We consider three weather image translation tasks based on SEVIR dataset: translate geostationary IR-069 to geostationary IR-107 data (GEOS-IR2GEOS-IR), geostationary IR-069 to geostationary Visible data (GEOS-IR2GEOS-Vis), translate geostationary IR-069 and IR-107 to radar VIL data (GEOS-IR2Radar). In addition, we add a weather image translation task for environment monitoring: Translate geostationary $NO_{2}$ data to polar-orbiting satellites $NO_{2}$ data (GEOS2POES-$NO_{2}$) based on POMINO-TROPOMI (Liu et al., ) and GEOS-CF (Keller et al., ) product.
For the image translation tasks based on SEVIR dataset, we split SEVIR into 542,784 training samples, 4,800 validation samples and 4,800 test samples. For translating geostationary $NO_{2}$ data, the input images are sourced from GEMS as well as the GEOS-CF datasets, while the output images are obtained from the TROPOMI dataset. Each modality has 20,000 images with a resolution of 256×256. Among them, we allocate 18,000 images as the training set, 1,000 images as the validation set, and 1,000 images as the test set.

Weather post-processing: Post-processing (e.g., bias correction) aims to minimize or eliminate systematic biases in model outputs and observational data, which emerge due to uncertainties in weather models and measurement errors. Various methods, including statistical, machine learning, and deep learning techniques, can be employed for post-processing, tailoring the approach based on the specific application and data characteristics. By minimizing or eliminating systematic biases, post-processing improves the quality and reliability of weather and climate data. In our experiment, we consider a classic post-processing task: Debluring for radar VIL nowcasting. We employ the output of Earthformer and the corresponding high-quality image as a training sample. Deblurring aims to learn how to map from the output of Earthformer to the corresponding high-quality image.

## Appendix B Effects of Visual Prompts

Figure: Figure 8: Case studies of our WeatherGFM with different prompts in GEOS-IR2Radar task.
Refer to caption: x8.png

To assess the effectiveness of visual prompts within the WeatherGFM framework, we conducted a series of tests across various scenarios, including GOES2Radar, Radar Extrapolation, GOES-IR2GOES-IR, Radar Spatial SR, Radar Temporal SR, and Deblur tasks. In our main paper, we detail a thorough process of prompt selection, curating 20 unique prompts for each task and subsequently reporting the most favorable quantitative outcomes. The comprehensive results of these prompt variations on several representative tasks are presented in the tables.

By examining Table 5, we observed that prompts have a significant impact on the GOES2Radar and Radar Extrapolation tasks. Consequently, we conducted further experiments on these two tasks. The results are shown in Table 4. In these experiments, “random prompt” refers to prompts selected randomly, while “high prompt” refers to prompts selected from a high-quality prompt base derived from radar data. Specifically, we grouped 100 events and selected samples that contained values exceeding a threshold of 50 within each group, totaling 113 samples. Prompts were then randomly selected from this high-quality base for the experiments. “Searched prompt” refers to a method where prompts are selected based on the RMSE metric calculated from the input images to find similar prompts.

**Table 4: Comparison of the models with random prompts, high-quality prompts, and searched prompts according to RMSE.**
| Tasks | GOES2Radar | Radar Extrapolation |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | CSI/74 | CSI/133 | CSI/160 | CSI/181 | CSI/219 | CSI/74 | CSI/133 | CSI/160 | CSI/181 | CSI/219 |
| random prompts | 0.389 | 0.238 | 0.194 | 0.15 | 0.048 | 0.423 | 0.187 | 0.106 | 0.069 | 0.017 |
| high prompts | 0.399 | 0.249 | 0.208 | 0.164 | 0.057 | 0.426 | 0.191 | 0.11 | 0.072 | 0.019 |
| searched prompts | 0.401 | 0.24 | 0.199 | 0.155 | 0.049 | 0.425 | 0.188 | 0.108 | 0.071 | 0.018 |

## Appendix C More details of scaling law

In Figure 9, we present the results of the RMSE metric that compares single-task models, the base version of our WeatherGFM, and its large version. It can be observed that increasing the capacity of the model generally leads to better performance when the data size remains constant. In contrast, for smaller models, an increase in training data may result in poorer performance. We hypothesize that this could be due to the specificity of different tasks within the training data, which makes it more challenging for the model to fit effectively.

Figure: Figure 9: RMSE performance comparison across different model configurations.
Refer to caption: x9.png

## Appendix D More Visual Results

To comprehensively assess the performance of WeatherGFM, we present a range of qualitative visual results across various tasks, including weather forecasting, weather super-resolution, and weather image translation. Additionally, we conduct a comparative evaluation against the unified ViT-large model, as well as single-task ViT and UNet models. The visual outputs and comparisons are thoughtfully illustrated in Figure 10.

WeatherGFM’s proficiency in generating visually appealing outputs is readily evident in the presented results. Notably, the visual quality surpasses that of the baseline models. However, the significance of WeatherGFM’s capability extends beyond visual quality. Its distinctive strength lies in its ability to effectively handle a wide array of image enhancement tasks and image detection. This marks a noteworthy distinction from traditional models, which often struggle to concurrently address such a diverse spectrum of tasks.

Figure: Figure 10: Visual results of the weather understanding tasks by our WeatherGFM.
Refer to caption: x10.png

**Table 5: Influence of employing different visual prompts on different tasks. The color red is used for the poorest-performing prompts, and green is used for the best-performing prompts.**
| Prompts |  | Idx0 | Idx1 | Idx2 | Idx3 | Idx4 | Idx5 | Idx6 | Idx7 | Idx8 | Idx9 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GOES2Radar | RMSE | 0.4759 | 0.471 | 0.5117 | 0.4748 | 0.4691 | 0.4725 | 0.4726 | 0.4692 | 0.4722 | 0.473 |
| CSI | 0.3401 | 0.3335 | 0.3233 | 0.3300 | 0.3452 | 0.3298 | 0.3272 | 0.2467 | 0.3302 | 0.3317 |  |
| Radar<br>Extrapolation | RMSE | 0.4912 | 0.4908 | 0.4891 | 0.4925 | 0.4933 | 0.4937 | 0.4933 | 0.493 | 0.4933 | 0.4951 |
| CSI | 0.3401 | 0.2497 | 0.2534 | 0.2479 | 0.2484 | 0.2467 | 0.2462 | 0.2467 | 0.2453 | 0.2467 |  |
| GOES-IR2GOES-IR | RMSE | 0.283 | 0.2947 | 0.2593 | 0.3791 | 0.2424 | 0.2989 | 0.3213 | 0.2811 | 0.4372 | 0.387 |
| CSI | 0.6851 | 0.6907 | 0.6963 | 0.6559 | 0.6958 | 0.7525 | 0.6835 | 0.7314 | 0.6240 | 0.6529 |  |
| Radar Spatial SR | RMSE | 0.1331 | 0.1331 | 0.1334 | 0.1331 | 0.1332 | 0.1335 | 0.1334 | 0.1333 | 0.1333 | 0.1335 |
| CSI | 0.7208 | 0.7206 | 0.7209 | 0.7208 | 0.7228 | 0.7222 | 0.7221 | 0.7227 | 0.7217 | 0.7225 |  |
| Radar Temporal SR | RMSE | 0.3166 | 0.3166 | 0.3164 | 0.3164 | 0.3168 | 0.3172 | 0.3168 | 0.3166 | 0.317 | 0.3172 |
| CSI | 0.3387 | 0.3395 | 0.3389 | 0.3359 | 0.3398 | 0.3366 | 0.3376 | 0.3376 | 0.3372 | 0.3388 |  |
| Deblur | RMSE | 0.2272 | 0.2236 | 0.2236 | 0.2255 | 0.2253 | 0.2274 | 0.2273 | 0.2286 | 0.2241 | 0.2233 |
| CSI | 0.3412 | 0.3405 | 0.3420 | 0.3413 | 0.3407 | 0.3406 | 0.3408 | 0.3408 | 0.3410 | 0.3419 |  |
| Prompts |  | Idx10 | Idx11 | Idx12 | Idx13 | Idx14 | Idx15 | Idx16 | Idx17 | Idx18 | Idx19 |
| GOES2Radar | RMSE | 0.4711 | 0.4762 | 0.4731 | 0.4706 | 0.4743 | 0.4752 | 0.4747 | 0.4764 | 0.4735 | 0.4725 |
| CSI | 0.3260 | 0.3252 | 0.3277 | 0.3266 | 0.3264 | 0.3297 | 0.3248 | 0.3247 | 0.3276 | 0.3265 |  |
| Radar<br>Extrapolation | RMSE | 0.4927 | 0.4931 | 0.4934 | 0.493 | 0.4938 | 0.4929 | 0.4932 | 0.4934 | 0.4937 | 0.4934 |
| CSI | 0.2478 | 0.2474 | 0.2483 | 0.2483 | 0.2493 | 0.2488 | 0.2474 | 0.2477 | 0.2479 | 0.2474 |  |
| GOES-IR2GOES-IR | RMSE | 0.4052 | 0.3017 | 0.2917 | 0.3136 | 0.3027 | 0.2828 | 0.2886 | 0.3189 | 0.3147 | 0.3102 |
| CSI | 0.6451 | 0.7057 | 0.7034 | 0.6930 | 0.7064 | 0.7044 | 0.7076 | 0.6899 | 0.6982 | 0.7040 |  |
| Radar Spatial SR | RMSE | 0.1332 | 0.1333 | 0.1332 | 0.1332 | 0.1332 | 0.1332 | 0.1332 | 0.1333 | 0.1332 | 0.1332 |
| CSI | 0.7222 | 0.7221 | 0.7215 | 0.7216 | 0.7217 | 0.7219 | 0.7218 | 0.7215 | 0.7224 | 0.7213 |  |
| Radar Temporal SR | RMSE | 0.3166 | 0.3168 | 0.3167 | 0.3168 | 0.3166 | 0.3167 | 0.3166 | 0.3169 | 0.317 | 0.3166 |
| CSI | 0.3359 | 0.3374 | 0.3370 | 0.3371 | 0.3381 | 0.3367 | 0.3375 | 0.3374 | 0.3371 | 0.3376 |  |
| Deblur | RMSE | 0.2252 | 0.2233 | 0.2272 | 0.2257 | 0.2259 | 0.2257 | 0.2264 | 0.2267 | 0.2235 | 0.2263 |
| CSI | 0.3405 | 0.3405 | 0.3413 | 0.3417 | 0.3416 | 0.3418 | 0.3416 | 0.3417 | 0.3414 | 0.3415 |  |

## Appendix E Additional Quantitative Results

In this section, we present more quantitative results on various weather understanding tasks. The results include the Probability of Detection (POD) and False Alarm Rate (FAR) metrics for different models across multiple tasks. Table 6 provides a detailed comparison of these metrics for the UNet, ViT, and WeatherGFM models. AVG.POD and AVG.FAR represent the mean scores across different thresholds for various tasks. For radar output tasks, the thresholds are [16, 74, 133, 160, 181, 219]; for GEOS-visible output tasks, the thresholds are [2000, 3200, 4400, 5600, 6800]; and for GEOS-IR069 output tasks, the thresholds are [-4000, -5000, -6000, -7000].

**Table 6: More quantitative results on weather understanding tasks. We also report the results of the POD and FAR metrics.**
|  | Downscaling | Forecasting |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Task name | Satellite Spatial SR | Radar Temporal SR | Radar Spatial SR | Radar Extrapolation |  |  |  |  |
| Metrics | POD | FAR | POD | FAR | POD | FAR | POD | FAR |
| UNet | 1.0000 | 0.7466 | 0.4380 | 0.1738 | 0.6568 | 0.1702 | 0.3207 | 0.1751 |
| ViT | 0.9941 | 0.0560 | 0.4520 | 0.0139 | 0.7219 | 0.0047 | 0.2749 | 0.0228 |
| WeatherGFM | 0.9973 | 0.0540 | 0.4362 | 0.0111 | 0.7320 | 0.0039 | 0.3028 | 0.0162 |
|  | Inversion | Forecasting |  |  |  |  |  |  |
| Task name | GOES2Radar | GOES-IR2GOES-Visible | GOES2POES-NO2 | Satellite Extrapolation |  |  |  |  |
| Metrics | POD | FAR | POD | FAR | POD | FAR | POD | FAR |
| UNet | 0.4691 | 0.1821 | 0.4260 | 0.1358 | 0.5993 | 0.2548 | 1.0000 | 0.7841 |
| ViT | 0.3537 | 0.0229 | 0.4626 | 0.0829 | 0.5421 | 0.1686 | 0.9599 | 0.3514 |
| WeatherGFM | 0.3524 | 0.0147 | 0.3630 | 0.1388 | 0.5023 | 0.0024 | 0.9578 | 0.2379 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - Why does WeatherGFM show weaker performance in weather image translation compared to other tasks?
- In Table 1, there is a misuse of double lines for "GOES2Radar, CSI/219."
soundness: 3
strengths: - The proposed weather prompt design effectively handles a wide range of weather-related tasks.
- By utilizing an in-context learning approach, WeatherGFM can generalize to unseen tasks without requiring fine-tuning.
- WeatherGFM demonstrates its scalability across different model sizes.
- The model successfully handles ten various tasks on the SEVIR dataset using a single, unified framework.
summary: This paper introduces WeatherGFM, a weather foundation model inspired by in-context learning approaches used in visual and language models. WeatherGFM aims to address multiple weather-related tasks within a unified framework by converting diverse tasks with various data types into general weather prompts. The model employs a masked modeling approach for training, while fully masking the targets during inference. Experimental results on SEVIR dataset show that WeatherGFM can handle up to ten different weather tasks, including forecasting, super-resolution, image translation, and post-processing. The model also demonstrates its generalization capabilities on unseen tasks.
weaknesses: - In Section 3.2, it appears that WeatherGFM uses task-specific patch embedding layers, which may limit the framework's generalization ability. For instance, with the current design, WeatherGFM may struggle to generalize to new, unseen tasks that require novel sensor channels.
- The impact of increasing dataset size is unclear in Figure 7. Here, the authors compare ViT-ST with WeatherGFM-Base to illustrate the scalability of WeatherGFM. However, since ViT-ST is trained on a single task while WeatherGFM is trained on multiple tasks with more data and parameters, it’s unclear whether the performance gains are due to dataset scalability or multi-task training or larger number of parameters. Additionally, on radar spatial super-resolution (SR), scaling the dataset size or using multi-task training appears to degrade performance. A more detailed analysis and discussion on the effects of dataset scalability would be valuable.
- It would be beneficial to compare WeatherGFM’s performance against other climate foundation models (e.g., Aurora, ClimaX) using the ERA5 dataset. One of WeatherGFM’s key strengths is its ability to unify multi-task, multi-modal datasets. Thus, similar to Aurora, pretraining WeatherGFM on a combination of ERA5 and other heterogeneous datasets could potentially enhance its performance. It would be impressive if WeatherGFM could achieve comparable results to ClimaX or Aurora without fine-tuning, especially by leveraging a larger dataset and multi-task training.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: What is the motivation for designing Mixed-modal mask modeling? It seems like there should be some references that motivate this design.
soundness: 3
strengths: 1. A good contribution of this work is to apply in-context learning to weather-understanding tasks for building a unified model and confirming its effectiveness.
2. The proposed model demonstrates comparable results with specialist models on ten weather understanding tasks,
3. Mixed-modal mask modeling is interesting.
summary: This paper presented a weather generalist foundation model based on in-context learning, which unified a wide range of weather understanding tasks, like weather forecasting, wealth image translation, wealth super-resolution, deblur, etc.
It first defines the visual prompt template and then does in-context learning to achieve such a unified model.
The experiments demonstrate that this unified model achieved comparable results with specialist models, e.g., ViT and Unet.
weaknesses: 1. The motivation of the training objective in Figure 3 is unclear. The eq.5 is mismatched with Figure 3. There should be more details about the training objective. X_T in eq.5 lacks a definition for your universal representation.
2. Details missing. For example, the patch embedding layer is task-specific. Does this mean that there is a hard code to select which patch embedding layer is used to handle different tasks or data modalities? If so, can this model be claimed as a unified model?
3. The motivation of such a unified model is actually unclear to real-world applications. There is no obvious improvement over the conventional specialist model. In addition, there are not many combinations of tasks here. What is the essential advance of the proposed model compared with per-task specialists?

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: See Weaknesses. 

Are there ablations on the length of the input prompt sequence and how that affects performance?
Keeping text bold in the introduction paragraphs hampers readability.
Line 159-180: Define what t is and what its range is
soundness: 1
strengths: **(S1)**: The problem of creating a more general weather foundation model is important, and it is valuable that the authors are investigating this area. 

**(S2)**: Demonstrating in-context learning is also useful for weather foundation models, although I admit I haven’t looked too closely into whether there exists prior work that tackles this.
summary: The authors propose a generalist foundation model for weather tasks that includes in-context learning that can handle multiple weather tasks.
weaknesses: **(W1)**: Experimental comparisons are missing. The authors don’t seem to compare WeatherGFM on the same tasks used in the Climax or Aurora papers, and it is unclear why this comparison isn’t made. The main claim is that WeatherGFM is a more general foundation model, and yet the only comparison is made with simple task-specific baselines such as a ViT or UNet. There needs to be compelling evidence that WeatherGFM can outperform/compete with Climax or Aurora. Moreover, only 3 qualitative examples on out-of-distribution (OOD) examples are given in Figure 6. Quantitative evaluations on OOD tasks are quite important for a foundation model.

**(W2)**: Experimental results are weak. For the image translation tasks in Table 2, it is unclear that WeatherGFM can outperform baselines on these datasets. The authors do mention that their goal is not to achieve SoTA performance on each task— this is fine, but since comparison with other SoTA methods is missing, the results on the tasks described are not compelling enough. Moreover, in weather forecasting and super-resolution, the performance of the ViT baseline is nearly the same as WeatherGFM, which suggests only a minute (if any) improvement with the author’s designed modifications and pre-training strategy. Moreover, details on how these baseline ViT or UNets were trained are lacking, which are very crucial to determine if the experiments are fair.

**(W3)**: Lack of significant novelty. While the authors do propose changes to the ViT pre-training setup used in other Weather foundation model setups, these are not significant. The authors’ claims of supporting multi-modality isn’t well supported considering other foundation models already do support inputs from different weather sensors and earth observation (EO) systems. The only difference here seems to be satellite imagery which I don’t think is too different from the setup of the other foundation models, and the authors don’t provide any evidence that the techniques of Climax or Aurora wouldn’t also extend to this additional input. Masked image modelling for EO data and designing patch embeddings for different sensory inputs is an established technique, eg: [1].

**(W4)**: Lack of presentation clarity. I think this paper is tackling an important problem, but does not clearly motivate its solution. 
* For example, figure 3 is quite unclear and it is very hard to understand what exactly is going on (in terms of the pre-training strategy or the architectural modifications). The masked/predicted/ground-truth patch disambiguation is not clear to me. It seems that this figure contains multiple subfigures that should clearly be disambiguated and explained.
* The actual pre-training method is not described extensively. Specifically, lines 236-253 need to be clearly explained and expanded, since this seems to be the crux of the paper’s modifications. Instead, too much space is given to explaining the architecture of a ViT (lines 254-280), which is not required in the main text of the paper.
* Qualitative outputs from the model are not compared with qualitative outputs from other baselines. The authors demonstrate the outputs of WeatherGFM but don’t compare these on tasks with other weather foundation models. This is related to (W1).
* Critical details on training and inference are missing. What is the patch size? What kind of compute was used for training? How much compute? Was the dataset combined? Any dataset augmentation? etc. etc. etc. these details are important for reproducibility but are missing from the paper and appendix.

Overall, given the weaknesses describe above, I don't think this paper is ready for acceptance in this conference. 

References:  
[1] SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery, NeurIPS 2022.

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 4
questions: 1. Can you explain in detail which part of the module WeatherGFM shows its advantages in various tasks?
2. Can you simply explain the significance of CSI threshold setting?
soundness: 2
strengths: 1.	The proposed WeatherGFM can handle different weather data modalities and task objectives in a unified manner
2.	In the unseen understanding task, the WeatherGFM still shows a certain generalization.
3.	The introduction of the model input format is very clear, and the classification and identification of various weather image tasks are fully explained
summary: The paper presents WeatherGFM, a pioneering general-purpose weather foundation model that unifies diverse weather-related tasks through contextual learning. By integrating different weather data formats, WeatherGFM is trained with visual prompts to handle a wide range of tasks, including forecasting, super-resolution, image translation, and post-processing. The model is designed to overcome the limitations of task-specific models by leveraging a single architecture capable of understanding and adapting to varying weather data patterns. Extensive experiments show that WeatherGFM achieves state-of-the-art performance across multiple tasks and exhibits impressive generalization on unseen tasks, demonstrating its potential as a robust, adaptable solution for complex weather modeling challenges.
weaknesses: 1.	A mixed-modal masked image modeling (MMIM) pipeline, which is the core innovation part of the proposed framework of WeatherGFM, has not been introduced in this paper. Other parts of  WeatherGFM architecture are actually closer to the comparative ClimaX model framework. It will lack the innovative elaboration of the model framework.
2.	CSI is the main evaluation index in this paper, but ACC is also a very important index to evaluate the correlation of weather. Should this index be taken into account? Moreover, the setting of CSI threshold value needs some explanation in this paper to explain its important reference for this kind of weather situation classification.
3.	According to the comparative results of Table 2 experiments in this paper, in fact, the basic Vision Transformer can also realize a variety of understanding tasks of various data sets in the form of input format proposed by the author, and the effect of some tasks is better than that of WeatherGFM, so the advantages of this model in multi-task are specifically highlighted.
4.	There are two clerical errors: (a) "the" should be capitalized. (b) The basic learning rate in line 366 should be 1e-4.

Meta-review: This paper introduces a generalist model for various weather understanding tasks. The key contribution is demonstrating that the widely used principles for vision generalist models—framing diverse image-to-image translation tasks as a unified in-context learning problem—can be effectively extended to multi-modal weather data. The authors validate their approach across a range of weather tasks involving diverse modalities, showcasing its potential versatility.

The paper received mixed reviews, including one strong positive, two borderline positive, and one strong negative assessment. The primary concerns raised by the reviewers were as follows:

1. Missing comparisons to other generalist models(**vbFm**, **CC3i**),
2. Limited technical novelty (**CC3i**, **qe7B**),
3. Constrained generalization to other modalities (**vbFm**, **CC3i**), and
4. Choice of weak baselines (**vbFm**, **CC3i**).

In response, the authors provided significant clarifications and new results, including comparisons to foundation models (e.g., ClimaX), analysis of scale-performance correlation, and quantitative evaluations on out-of-distribution (OOD) tasks.

The AC thoroughly reviewed the paper, reviews, and rebuttal discussions. The AC concludes that the paper is on the borderline. On the critical side, the paper naively adapts the framework of PromptGIP (Liu et al., 2024) to weather data, with the primary extension being modality-specific input embedding layers. While a defining characteristic of foundation models is their ability to generalize to downstream tasks beyond the pre-training data (e.g., as seen in ClimaX and Aurora), the presented experiments mostly emphasize multi-task learning within the pre-training data, failing to stretch further from PromptGIP. However, the preliminary results on the WeatherBench dataset, provided during the rebuttal, are promising and partially demonstrate the potential of the proposed method for OOD generalization, even though the evidence remains limited.

Considering that the paper addresses an important problem and the rebuttal results on WeatherBench show encouraging signs of OOD generalization, the AC recommends **acceptance**. To strengthen the work, the authors are encouraged to:

1. Supplement the preliminary comparison results with additional analyses against other generalist models.
2. Clearly present experimental details to improve the credibility of the findings (e.g., clarifying fine-tuning datasets for ClimaX and WeatherGFM, and including more results).
3. Release the code to facilitate reproducibility and further exploration by the research community.

Decision: Accept (Poster)