Title: Identifying Spatio-Temporal Drivers of Extreme Events
Venue: NeurIPS 2024
Accepted: yes
Keywords: anomaly detection

weakly supervised learning

Earth science

climate science

remote sensing

deep learning
Abstract: The spatio-temporal relations of impacts of extreme events and their drivers in climate data are not fully understood and there is a need of machine learning approaches to identify such spatio-temporal relations from data. The task, however, is very challenging since there are time delays between extremes and their drivers, and the spatial response of such drivers is inhomogeneous. In this work, we propose a first approach and benchmarks to tackle this challenge. Our approach is trained end-to-end to predict spatio-temporally extremes and spatio-temporally drivers in the physical input variables jointly. By enforcing the network to predict extremes from spatio-temporal binary masks of identified drivers, the network successfully identifies drivers that are correlated with extremes. We evaluate our approach on three newly created synthetic benchmarks, where two of them are based on remote sensing or reanalysis climate data, and on two real-world reanalysis datasets. The source code and datasets are publicly available at the project page https://hakamshams.github.io/IDE.

Identifying Spatio-Temporal Drivers of Extreme Events

Abstract

The spatio-temporal relations of impacts of extreme events and their drivers in climate data are not fully understood and there is a need of machine learning approaches to identify such spatio-temporal relations from data. The task, however, is very challenging since there are time delays between extremes and their drivers, and the spatial response of such drivers is inhomogeneous. In this work, we propose a first approach and benchmarks to tackle this challenge. Our approach is trained end-to-end to predict spatio-temporally extremes and spatio-temporally drivers in the physical input variables jointly. By enforcing the network to predict extremes from spatio-temporal binary masks of identified drivers, the network successfully identifies drivers that are correlated with extremes. We evaluate our approach on three newly created synthetic benchmarks, where two of them are based on remote sensing or reanalysis climate data, and on two real-world reanalysis datasets. The source code and datasets are publicly available at the project page https://hakamshams.github.io/IDE .

1 Introduction

A frontier research challenge is to understand the affects of global change on the magnitude and probability of extreme weather events [1]. Overall, the evolution of extreme events such as agricultural droughts results from stochastic processes [2], conditions at ecosystem scales [3, 4], and the interaction between the Earth land and atmospheric variables as a part of a complex system of feedbacks [5]. However, the relative impacts of these factors differ depending on the event [2]. The time delays between extremes and their drivers vary seasonally [6-9], and the spatial response of these drivers is inhomogeneous [4]. A major challenge is therefore to model the spatio-temporal relations between extremes and their drivers during the development of these events [4]. The overarching goal of this modelling is to improve our understanding of the patterns and impacts of such events. This would improve our ability to project duration and intensity of extreme events and hence assisting in adaptation planning [10, 11].

In this work, we propose an approach that identifies spatio-temporal drivers in multivariate climate data that are correlated with the impact of extreme events. For the extreme events, we focus on agricultural droughts as an example, which can be measured by extremely low values of the vegetation health index (VHI). As drivers for such measurable extreme events, we consider anomalies in atmospheric and hydrological state variables like temperature or soil moisture, as well as landatmosphere fluxes like evaporation. The task of identifying end-to-end spatio-temporal drivers for measurable impacts of extremes has not been addressed before, and it is very challenging since the drivers can occur earlier in time and at a different location than the measured extreme event as illustrated in Fig. 1.

To address this challenging task, we propose a network that is trained to predict spatio-temporally extremes. Instead of simply predicting the extremes, the network quantizes the spatio-temporal input variables into binary states and predicts the extremes only from the spatio-temporal binary maps for each time series of input variables. In this way, the network is enforced to identify only drivers in the input variables that are spatio-temporally correlated with extreme events. While the network is trained using annotations of impacts of extreme events, which can be derived from remote sensing or reported data, we do not have any annotations of drivers or anomalies in the input variables.

Since drivers of extreme events are not fully understood, we cannot quantitatively measure the accuracy of the identified drivers on real-world data. We therefore propose a framework for generating synthetic data that can be used to assess the performance of our model as well as other baselines quantitatively. We evaluate our approach on three synthetic datasets where two of them are based on remote sensing or reanalysis climate data. Our evaluation shows that our approach outperforms approaches for interpretable forecasting, spatio-temporal anomaly detection, out-of-distribution detection, and multiple instance learning. Furthermore, we conduct empirical studies on two realworld reanalysis climate data. Our contributions can be summarized as follow:

We introduce the new task of identifying spatio-temporal drivers of extreme events and three benchmarks for evaluating this highly important task.

We propose a novel approach for identifying spatio-temporal drivers in climate data that are spatio-temporally correlated with the impacts of extreme events.

We further verify our approach on two long-term real-world reanalysis datasets including various physical variables from five biogeographical diverse regions.

2 Related works

Anomalies and extremes detection in climate data. The identification of climatic changes and extreme weather has been a subject of many studies [12, 13]. Typical algorithms for extreme events detection are built upon domain knowledge in setting usually empirical thresholds for the physical variables through sensitivity experiments [6, 14]. Many works applied multivariate and statistical methods to detect extreme events such as droughts [6, 9, 15-17]. However, individual events are difficult to generalized across multiple events [2] and predefined indicators become less effective with changing climate [18]. Thus, machine and deep learning methods have been proposed as an alternative to classical methods, i.e., for supervised anomaly detection [19, 20] and for the detection of extremes in climate data [14, 21-24].

While methods for forecasting vegetation indices [25-28] do not focus on extremes, future impacts of extremes like agricultural droughts can be derived from forecast vegetation indices like the vegetation health index (VHI). For instance, the work [28] uses a climate simulation as input and forecasts the vegetation health index. Since predicting VHI directly is difficult, the approach predicts the normalized difference vegetation index and the brightness temperature instead. Both indices are then normalized and used to estimate VHI. Although we obtain the impacts of extreme events in our study from vegetation indices, our approach is not limited to such extremes. Since we use a binary representation of extremes, our approach can also be applied to other extremes that cannot be derived from satellite products, but that are stored in a binary format in databases.

While we aim to learn the relations between the impacts of extreme events and their relevant drivers from a data-driven perspective, spatio-temporal relations within the Earth system can also be inferred by causal inference and causal representation learning [29-36]. In contrast to statistical methods [37], data-driven methods do not require a prior hypothesis about drivers for extremes. Instead, they generate hypotheses that can be verified by statistical methods in a second step. We believe that this is an important direction since climate reanalysis provides huge amount of data and it is infeasible to test all combinations. This is also known as a curse of dimensionality in causal discovery problems [38] and data-driven approaches are therefore needed to generate potential candidates.

Anomaly detection algorithms. Since we focus on anomalies in land and meteorological data as drivers, we give an overview of approaches for anomaly detection and discuss their applicability to our task, which has not been previously addressed. One-class: The main stream in one-class anomaly detection is to model the distribution of the normal data during training and consider the deviation from the learned features as anomalies. This includes distance-based [39-41], patch-based

Statevariablel

StatevariableIV

Time

Time

Impactofextremes

Statevariablell

StatevariableV

Time

Time

Time

StatevariableIll

StatevariableVI

[41-43], student-teacher [44-47], and embedding-based approaches [43, 48-51]. In general, the alignment between the anomaly type and the assumptions of the methods is the critical factor for their performance [52]. One of the limiting factors to apply these methods to climate data is that they assume priori knowledge about what is considered as normal. Furthermore, not all detected anomalies are drivers of an extreme event. Reconstruction-based: These methods assume that a trained model to reconstruct normal data will be unsuccessful in reconstructing anomalies, while it will reconstruct normal data well. Despite being widely applied for anomaly detection problems [53-64], these methods face the same problem as the one-class methods. In addition, many studies showed that anomalies can still be reconstructed by the trained model [65]. Self-supervised learning: These methods rely on the hypothesis that a model trained for a pretext task on normal data will be successful only on similar normal data during inference [66-69]. In addition to the above discussed limitations, finding a suitable pretext task for anomaly detection is challenging. For instance, common tasks such as solving a jigsaw puzzle [69] will fail in homogeneous regions. Pseudo-anomaly: The intuition in pseudo-based anomaly detection is to convert the problem of unsupervised learning into a supervised one by synthesizing abnormal data during training [70-77]. Since these methods depend partially on the degree to which the proxy anomalies correspond to the unknown true anomalies [44], applying these approaches to our task would require some knowledge about the coupling between the variables and extremes. Multivariate anomaly detection: Multivariate approaches detect anomalies simultaneously in multiple data streams [78-82]. The main difference to our task is that we aim to detect drivers across multiple data streams that do not necessary occur simultaneously. Multiple instance learning: Multiple instance learning (MIL) has been proposed for weakly supervised anomaly detection [83-91]. In MIL-based algorithms, the model is provided with labeled positive and negative bags where each bag includes a set of instances. The model is then trained to classify the instances inside the bags giving only the high level supervision, i.e., label of the bag [92]. A weakly supervised approach has been also applied for hyperspectral anomaly detection [93]. Most algorithms such as [94-97] choose the top-k potentially anomalous snippets within each video. This makes it challenging to apply them since the abnormality ratio varies in real-world applications [98]. Furthermore, the MIL detector can be biased toward a specific class depending on the context [99].

3 Method

Our aim is to design a model that is capable of identifying spatio-temporal drivers of extremes in multivariate climate data, i.e., Earth observations or climate reanalysis. In particular, we want to identify anomalies that are spatio-temporally related to extreme events like agricultural droughts. This is different to standard anomaly detection since we are not interested in all anomalies, but in spatio-temporal configurations of variables that potentially cause an extreme event with some time

Feature embeddings

Quantization

△to

-t4

Volumetric soilmoisture

Volumetric soil moisture

Temporal-Attention

-△t

-ts

ExtremeAgriculturalDroughts

mean

2x Video Swin-

Transformer

1x Transformer

-△t

-△t2

-△t6

△to

Anomalies/ Drivers of extremes

std

-△t3

-△t7

-△t

△to

Temporal-Attention

△to

-△t4

3DCNN

mean

2mTemperature

2x Video Swin-

Transformer

1x Transformer

LFQ

-t

-△ts

2mTemperature

△to

std

-△t2

-t6

-△t

△to

-t3

-t7

mean

umedity

ntion

△to

-△t4

-t

delay and at a potentially different location as illustrated in Fig. 1. To achieve this, we propose a network that is trained end-to-end on observed extremes where we focus on agricultural droughts. The network classifies the input variables before an extreme event occurs into spatio-temporal drivers without additional annotations besides the annotated extremes. The network then aims to predict future extremes based on the identified drivers. It needs to be noted that we are interested in input variables that do not define an extreme event, but we aim to find anomalies in input variables that are correlated with the occurrence of an extreme event. We will thus denote potential drivers as anomalies.

An overview of our approach is shown in Fig. 2. The input are weekly climate variables at sequential time steps ( ∆ t -7 , . . . , ∆ t 0 ) and the model is composed of three main parts. First, a feature extractor extracts relevant features from each input variable independently. The second component is a quantization layer that takes the extracted features as input and classifies the input variables into drivers. The role of the quantization layer is to transform the input variable into a binary representation ( 1 = drivers and 0 otherwise). This ensures that no additional information is encoded as an input to the subsequent classifier except that if the input variable at a specific location and time is a driver or not. The third component is a classifier that takes as input the variables, location, and time where drivers have been identified and predicts where extreme droughts occur at the time step ∆ t 0 . All model components are trained jointly.

We denote the input data as X ∈ R V × C × T × Lat × Lon , where V is the input variables, C is the channel dimension for each variable (i.e., mean and standard deviation of the week), T is the temporal resolution ( ∆ t -T +1 , . . . , ∆ t 0 ), and Lat and Lon are the spatial extensions. The model has two outputs, Q ∈ Z V × T × Lat × Lon 2 representing the binary classification of the input variables into potential drivers of the extreme events, and probabilities E ∈ R Lat × Lon to predict extreme events at the time step ∆ t 0 . In the following, we describe the model components:

Feature extraction. First, embedded features f θ : X → Z ∈ R V × K × T × Lat × Lon are extracted independently for each input variable v ∈ V with K embedding dimensions and parameters θ . The rationale behind this independence is to prevent that drivers leak into other variables. We use the Video Swin Transformer model [100] as backbone to capture long-range interrelations across time and space. The input X is projected into a higher feature dimension K and followed by two Video Swin Transformer layers. The first layer has two consecutive 3D shifted window blocks for a spatio-temporal feature extraction. The second layer consists of one block for a temporal feature

extraction. The later is useful to focus only on the temporal evolution of the variables. An ablation study regarding the backbone is provided in Sec. C.4.

Quantization layer. The role of this layer is to map Z from an embedded space into a compact binary representation Q suitable for detecting drivers. Using vector quantization (VQ) [101], each embedded feature vector z ∈ Z is assigned into a learnable codebook feature vector z q ∈ Z q based on the Euclidean distance:

where Q is the size of the codebook. Recently, lookup-free quantization (LFQ) [102] substitutes the learnable codebook with a set of integers Q with | Q | = Q and represents the embedding space as a Cartesian product of binary numbers. This omits the need for a distance metric to do the nearest vector assignment and simplifies the quantization. Based on experimental results, we built the vector quantizer on LFQ with two integers Q = 2 ( q = 1 for drivers and q = 0 otherwise). Given a feature vector z , LFQ first maps z into a scalar value z l ∈ Z l ⊂ R V × 1 × T × Lat × Lon . For multi-modality, we use two sequential 3D CNNs on each input variable followed by a shared linear layer that maps z to z l and reduces the dimensions from K to 1 . The quantization is then given by the sign of z l :

where q ∈ Q represents the class ( q = 1 or q = 0 ), and Linear is a linear layer that converts sign ( z l ) back to the dimension K of the input after the quantization. Note that Z q has only two unique vectors z q =1 for a driver and z q =0 otherwise.

Prediction of extreme events. We use a classifier that predicts the probably of extreme events E at the time step ∆ t 0 from the identified drivers Z q . We use a 3D CNN classifier instead of a transformer to reduce the computations. For training, we only know the ground truth of extremes at time step ∆ t 0 denoted by ˆ E ∈ Z Lat × Lon 2 . While we could compute the cross-entropy between E and ˆ E , we found that a single 3D CNN is insufficient to detect all drivers that are correlated with an extreme event. Instead, we use V +1 3D CNNs where each predicts E v . While the first V 3D CNNs take the identified drivers for a single variable v as input, the last one takes the identified drivers of all variables as input. The multiple CNNs are only used for training. During inference, E is only predicted by the multivariate CNN where all variables are jointly used. The loss is thus given by

where S ∈ Z T × Lat × Lon 2 is a mask for the valid regions. We actually utilize a weighted version of L ( extreme ) to mitigate the class imbalance issue (Sec. C.3). While the loss L ( extreme ) ensures that extreme events can be predicted from the identified drivers, we need to add standard loss terms for the quantization to ensure that the learned codes and thus drivers are compact:

The commitment loss ∥ Z l -sg (sign( Z l )) ∥ 2 2 prevents the outputs of the encoder from growing and encourages Z l to commit to the codes [101], where sg stands for the stopgradient operator with zero partial derivative. The term E [ H ( sign( Z l ) ) ] encourages that the entropy per quantized code is low [102, 103], meaning that it provides more confident assignments. Whereas the term H [ E ( sign( Z l ) ) ] increases the entropy inside the batch to encourage the utilization of all codes [102, 103]. The last important ingredient is a loss that ensures that only spatio-temporal regions are identified that correlate with an extreme event. To this end, we look at regions and intervals where no extreme event occurred and use these examples without drivers. Formally, we use ˆ E t ∈ Z Lat × Lon 2 as the union of extreme ground truth at all time steps ( ∆ t -T +1 , . . . , ∆ t 0 ) and compute the loss by

where z q =0 is the quantization code for normal data without drivers. The model is trained end-to-end with the joint optimization of the loss function:

where θ, ϕ, ψ are the learnable parameters, and λ ( commit ) , λ ( ent ) , λ ( div ) , and λ ( driver ) are weighting parameters. Ablation studies are provided in Sec. 5.1 and in Appendix Sec. C.

4 Dataset

4.1 Defining extreme agricultural droughts from remote sensing

We are interested in a specific impact of extreme events namely extreme agricultural drought. To define such extreme event, we rely on the observational satellite-based vegetation health index (VHI) obtained from NOAA [104]. This remote sensing product cannot be directly derived from the input reanalysis, which makes the task very challenging. VHI approximates the vegetation state based on a combination of the brightness temperature and normalized difference vegetation index (VHI = 0 for unfavorable condition and VHI = 100 for favorable condition). Extreme agricultural droughts are usually defined as VHI < 26 [104]. The dataset has a temporal coverage of 1981-onward and is provided globally on a weekly basis. We mapped this dataset into the same domains of the reanalysis data as described in Sec. 4.2 and used this dataset as ground truth for extreme events. Note that VHI is a general vegetation index and should be interpreted carefully. Details about this index, the dataset and pre-processing are provided in the Appendix Sec. I.

4.2 Climate reanalysis

Reanalysis data aim to provide a coherent and complete reconstruction of the historical Earth system state as close to reality as possible. During reanalysis, short-term forecasts from numerical climate models are refined with observations within the so called data assimilation framework [105]. We conducted the experiments on two real-world reanalysis datasets; CERRA reanalysis [106] and ERA5-Land [107]. ERA5-Land is widely used for global climate research and it is provided hourly at 0 . 1 ◦ × 0 . 1 ◦ on the regular latitude longitude grid. CERRA is a regional reanalysis for Europe and is provided originally at 5 . 5 km × 5 . 5 km on its Lambert conformal conical grid with a 3 -hourly temporal resolution. We aggregated these two datasets on a weekly basis and selected the years within the period overlapping with the remote sensing data. In addition, we mapped ERA5-Land into 6 CORDEX domains [108] over the globe and conduct experiments on each region separately. We do the experiments with 6 common variables from ERA5-Land and CERRA based on their connections to agricultural droughts. For each variable and week, we computed the mean and standard deviation separately. More details regarding the variables and the domains along with the training/validation/test splits are provided in the Appendix Sec. H and Tables 20 and 21.

4.3 Synthetic dataset

Although ground truth for extreme droughts can be obtained from remote sensing, an important methodological question remains as how to reliably have a meaningful quantitative evaluation of the identified drivers and their relations to the extreme events. To solve this critical issue, we introduce a new synthetic dataset that mimics the properties of Earth observations including drivers and anomalies. We are aware that the dynamic of the synthetic data are simplified compared to real Earth observations. However, we rely on this generated dataset to perform the quantitative evaluation of the proposed approach. In a first step, we generate the normal data. For instance to generate synthetic data of 2m temperature from CERRA reanalysis, the normal signal at a specific time and location is generated based on the typical value of 2m temperature at that time and location (i.e., the mean or median value from a long-term climatology). The second step is to generate anomalies conditioned on the occurrence of extremes. To achieve this, we assign binary spatio-temporally connected flags as extreme events randomly within the datacube and track their precise spatio-temporal locations. Then based on a predefined coupling matrix between the variables and the extreme event, we generate anomalous events only for the variables that are defined to be correlated with the extremes. We consider these anomalies as the drivers for the extreme events. Finally, we add additional random anomalous events for all variables. We synthesize overall 46 years of data; 34 years for training, 6 subsequent years for validation and the last 6 years for testing. The challenge is to identify the drivers, i.e., the anomalous events that are correlated with extreme events. Examples of the synthetic data are shown in Fig. 3 and in Appendix in Figs. 7-12. Technical details are explained in Appendix Sec. A.

5 Experimental results

First, we conducted experiments and ablation studies on the synthetic datasets (Sec. 4.3). We also empirically verified the effectiveness of the proposed design compared to baselines on this synthetic dataset. Then, we validate the model on two real-world datasets over five continents in Sec. 5.2.

Setup and implementation details. We set the hidden dimension K to 16 by default. The temporal resolution is T = 6 for the synthetic data and T = 8 for real-world data. Since, seasonal cycles are typical in climate data, we deseasonalize locally by subtracting the median seasonal cycle and normalizing by the seasonal variance for each pixel. Details regarding the model and implementation setup are given in Appendix Sec. G. For evaluation, we use the F1-score, intersection over union (IoU), and overall accuracy on both classes (OA).

5.1 Experiments on the synthetic datasets

We show the results on the Synthetic CERRA described in Sec. 4.3 and in Appendix Table 3. The generated dataset mimics a set of variables ( V = 6 ) using statistics from the real-world CERRA reanalysis [106]. We artificially correlated four variables with extremes (2m temperature, total cloud cover, total precipitation, and volumetric soil moisture) and kept two variables uncorrelated (albedo and relative humidity).

Comparison to the baselines. We compare the new approach to interpretable forecasting approaches using integrated gradients [109] and to 8 baselines from 3 different categories of anomaly detection approaches; one-class unsupervised [39, 110, 51], reconstruction-based [59, 65], and multiple instance learning [94-96]. We also compare to a naive baseline which labels all variables as drivers for pixels where extreme events occur. The implementation details of these baselines are given in Appendix Sec. E.

The quantitative results are shown in Table 1. The naive baseline is impacted by two main issues; first by the time delay between drivers and extreme events, and second not all variables are correlated with the extremes. The second issue affects the one-class and reconstruction-based baselines where

Synthetic

Ground

IntegratedIntegrated

Variable

Truth

GradientsIGradientsII1

IF

OCSVM

SimpleNet

STEALNet

UniAD

RTFM

DeepMIL

ARNet

Ours

albedo

cloudcover temperature

total

precipitation

total

relative

humidity

volumetric

soilmoisture

they suffer mostly from false positives. In fact, both integrated gradients models achieve high F1scores for detecting extremes (93.32 for Integrated Gradients I and 93.80 for Integrated Gradients II), but they have worse performance on identifying the drivers. SimpleNet is trained with our model as a feature extractor which explains its good performance. However, SimpleNet showed a drop of performance when it is tested on other datasets (see Appendix Sec. B for results on two more synthetic datasets). Among the reconstruction-based approach, STEALNet outperforms UniAD. This is probability because STEALNet exploits more weakly supervision information during training by maximizing the reconstruction loss for locations with extreme flags. MIL-based baselines are more suitable for the task. Finally, our model consistently outperforms the baselines on all metrics.

Qualitative samples in comparison with baselines are presented in Fig. 3. The qualitative examples indicate that our model and the MILbased baselines except RTFM are capable of learning which variables are correlated with the extremes. The main weakness of RTFM is the reliance on feature magnitudes and the cross attention module (see Appendix Sec. E and Table 2), which make it more prone to produce false positives. Other baselines predict incorrect relations between the variables and extremes. Regarding the explainable AI methods, when we add more interactions between the variables (Integrated Gradients II), the gradients tend to omit some variables (soil moisture). Both integrated gradients models have also difficulties with the synthetic t2m, which includes red noise

1/6

2/6

3/6

4/6

5/6

6/6

Number of variables correlated with extreme events

0

10

20

30

40

50

60

70

80

90

100

F1-score [%]

IF

OCSVM

SimpleNet

STEALNet

UniAD

DeepMIL

ARNet

RTFM

Ours

by design. These results demonstrate that networks that predict the extremes directly from the input variables utilize much more information even when it is not correlated with an extreme. It is thus beneficial to introduce a bottleneck into the network that enforces the network to explicitly identify drivers of extremes.

Performance on easy-to-hard correlation settings. We conduct an additional experiment to assess the model performance in relation to the correlation setup between the variables and extremes. We generate a synthetic CERRA dataset starting with only one correlated variable with the target extreme. We then generate different versions of the dataset by increasing the number of correlated variables with the extremes up to 100% . This analysis allows us to point out the strengths and weaknesses of

the comparative models for different scenarios and where our model becomes more effective, as well as where it could struggle most. The results are shown in Fig. 4. One-class, reconstruction-based and RTFM baselines benefit with increasing the number of correlated variables. In case of 6/6, the task reduces to an anomaly detection task. The performance of our model and MIL-based baselines generally decreases when the number of correlated variables increases as the task of finding all correlated anomalies becomes harder. Nevertheless, our approach performs best in all settings.

Ablation study. We conducted a set of ablation studies. This includes three main experiments:

Loss functions. As shown in Table 2 (a), L ( quantize ) and L ( driver ) are essential for training. As other quantization models, ours can not be trained without L ( quanitze ) , which ensures that the outputs of f θ do not grow and commit to the binary embedding. L ( driver ) unifies the representation of drivers for all variable as q = 1 , which boosts the extreme detection. Moreover, the results demonstrate the impact of using V +1 3D CNNs (multi-head) instead of one for L ( extreme ) . If a single 3D CNN is used, drivers are only identified in a small subset of variables. We discuss this more in detail in Sec. C.7.

Feature extractor. In Table 2 (b), we show the benefit of having independent feature extractors for driver detection. In a first experiment, we share the feature extractor f θ among the variables. The performance is worst. Second, we replaced the temporal attention by a cross attention between the variables similar to [112] and [113] where each variable performs a cross attention with the other variables. We can see a drop of performance for the second experiment. We noticed that anomalies propagate between variables when adding connections in the feature extraction stage. This also explains the poor performance of RTFM compared to other MIL baselines. The best performance is shown for the proposed setup (last row), which also shows the benefit for the temporal attention.

Temporal resolution T . Table 2 (c) evaluates the impact of the temporal resolution on driver and extreme detection. T =6 provides a good balance between driver and extreme detection. More ablation studies on other aspects of the model design can be found in Appendix Sec. C.

5.2 Experiments on real-world datasets

We evaluate our model on two reanalysis data with diverse geographical and climate regions (Sec. 4.2). The input for the experiment is the normalized mean and standard deviation of each week. We exclude pixels over water surfaces, desert, and snow. In addition to the quantitative evaluation on the synthetic data, we aim to verify our method considering the following aspects:

Quantitative results. Weexpect that the developed model can identify drivers in real-world scenarios. We demonstrate this by measuring how well the model can predict extreme agricultural droughts from the identified drivers. The results verify that the model can predict the droughts across different regions and datasets (see Appendix Sec. D and Table 14). Note that compared to the synthetic dataset, the real-world drought prediction is much more difficult.

Extreme detection without anomaly detection. We trained the model without the quantization step, meaning without driver detection. This can be considered as an upper bound on the extreme detection accuracy since there is no information reduction by the quantization. We found that when we trained on the synthetic and real-world EUR-11 data, the F1-score for detecting extreme events increased only by ∼ 0 . 96% and ∼ 1 . 93% , respectively, compared to the model with quantization (see Appendix Table 12). This verifies that the detected drivers are highly correlated with the extremes.

Qualitative results and spatial distribution. In Fig. 5 (a), we show the spatial distribution of the identified drivers at a specific time over EUR-11. Shown are the identified drivers up to 7 weeks

(a)year=2023,week=16

(b)EUR-Summer,weeks(25-37),2018-2023

Observed

Predicted

△to

-△t

-△t2

extreme droughts

extreme droughts

fal

st1

fal

st/1

e

fal

stl1

% of anomalous pixels

total evaporation (e)

05

t2m

tp

t2m

tp

t2m

tp

aue3s-t

-t3

-t4

-△t5

-t6

-t7

volumetric soill water (swvlL)

fal

Stl1

e

fal

Stl1

fal

stl1

fal

stl1

e

fal

stl1

C

05

t2m

tp

t2m

tp

t2m

tp

t2m

tp

t2m

tp

total evaporation (e)

1 volumetric soil water (swvl1) — normal condition [z-score = 0]

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

( ∆ t -7 ) before the extreme agricultural droughts at time ∆ t 0 . We can see that the prediction of drivers and extremes are spatially correlated with the ground truth.

Physical consistency. In Fig. 5 (b), we show the relation between the input reanalysis data, extreme droughts, and identified drivers. For this experiment, we selected pixels with extreme events during summer (weeks 25 -38 ) and visualize the average distribution of drivers with time. The red line at ∆ t 0 indicates the beginning of the extreme droughts. Z score in the underneath curve represents the deviation from the mean computed from the climatology. It is expected that evaporation reduces soil moisture, which dries out the soil and vegetation [4]. Our model indicates that over Europe, the evaporation and soil moisture are the most informative variables to detect drivers related to extreme droughts. All pixels experienced a pronounced decline in soil moisture and an increase in evaporation as the events evolved. Please see Sec. D.3 for more discussion on the scientific validity.

6 Discussions and conclusions

We introduced a model that can identify the spatio-temporal relations between impacts of extreme events and their drivers. For this, we assumed that there exist precursor drivers, primarily as anomalies in assimilated land surface and atmospheric data, for every observable impact of extremes. We demonstrated the effectiveness of our approach by measuring to which degree the identified drivers can be used to predict extreme agricultural droughts. Apart from experiments on two realworld datasets, we also presented a new framework to generate synthetic datasets that can be used for spatio-temporal anomaly detection and climate research. The results on the synthetic datasets show that the approach is not limited to droughts and can be applied to other extremes. While we have shown that our approach outperforms other approaches, the study has some limitations. First, evaluating ability to handle a very large number of climate variables in a unified model needs further examination. Similarly, performing an additional pre-processing of specific variables like accumulating precipitation over many weeks might also improve the results. Second, modelling the temporal relations is limited by the time window T . Moreover, teleconnections of climatic anomalies can occur in distant regions on Earth, e.g., affects of El Niño and La Niña variability on drought and flood [12]. Modelling and disentangle such large spatio-temporal relations across the globe is an open research problem. Third, it would be appealing to provide scores for drivers instead of a binary classification. This could be achieved by measuring the distance to the nearest code in the LFQ. Forth, the prediction of the model depends on the capacity of reanalysis data to accurately represent the local environmental factors and land-atmospheric feedbacks. Most importantly, drawing conclusions on drivers from weak predictive models may lead to unreliable interpretations. Finally, our model does not identify causal relationships.

Despite these limitations, our approach demonstrates a clear capability in identifying drivers and anomalies in climate data which would allow a more timely event attributions during and right after extreme events. The identified spatio-temporal relations between extreme events and their drivers could support the understanding and forecasting of extremes.

7 Acknowledgments and Disclosure of Funding

We thank Petra Friederichs, Till Fohrmann, Sebastian Buschow, and Svenja Szemkus for the insightful discussions on detection and attribution of weather and climate extremes. We would also like to thank Olga Zatsarynna and Emad Bahrami for the technical discussion related to feature extraction. Finally, we thank the four anonymous reviewers for their comments and suggestions which improved the quality of this paper.

This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - SFB 1502/1-2022 - project no. 450058266 within the Collaborative Research Center (CRC) for the project Regional Climate Change: Disentangling the Role of Land Use and Water Management (DETECT) and by the Federal Ministry of Education and Research (BMBF) under grant no. 01IS24075C RAINA.

We acknowledge EuroHPC Joint Undertaking for awarding us access to Leonardo at CINECA, Italy, through EuroHPC Regular Access Call - proposal No. EHPC-REG-2024R01-076. The authors also gratefully acknowledge the granted access to the Marvin cluster hosted by the University of Bonn.

References

Stephanie C. Herring, Nikolaos Christidis, Andrew Hoell, and Peter A. Stott. Explaining extreme events of 2020 from a climate perspective. Bulletin of the American Meteorological Society , 103(3):S1 - S129, 2022.

Jana Sillmann, Thordis Thorarinsdottir, Noel Keenlyside, Nathalie Schaller, Lisa V. Alexander, Gabriele Hegerl, Sonia I. Seneviratne, Robert Vautard, Xuebin Zhang, and Francis W. Zwiers. Understanding, modeling and predicting weather and climate extremes: Challenges and opportunities. Weather and Climate Extremes , 18:65-74, 2017.

M. B. Ek and A. A. M. Holtslag. Influence of soil moisture on boundary layer cloud development. Journal of Hydrometeorology , 5(1):86 - 99, 2004.

Diego G. Miralles, Pierre Gentine, Sonia I. Seneviratne, and Adriaan J. Teuling. Land-atmospheric feedbacks during droughts and heatwaves: state of the science and current challenges. Annals of the New York Academy of Sciences , 1436(1):19-35, 2019.

JL Geirinhas, AC Russo, R Libonati, DG Miralles, AM Ramos, L Gimeno, and RM Trigo. Combined large-scale tropical and subtropical forcing on the severe 2019-2022 drought in south america. npj Climate and Atmospheric Science , 6(1):185, 2023.

Shengzhi Huang, Qiang Huang, Jianxia Chang, Guoyong Leng, and Li Xing. The response of agricultural drought to meteorological drought and the influencing factors: A case study in the wei river basin, china. Agricultural Water Management , 159:45-54, 2015.

Fei Wang, Hexin Lai, Yanbin Li, Kai Feng, Zezhong Zhang, Qingqing Tian, Xiaomeng Zhu, and Haibo Yang. Dynamic variation of meteorological drought and its relationships with agricultural drought across china. Agricultural Water Management , 261:107301, 2022.

Meng Dai, Shengzhi Huang, Qiang Huang, Xudong Zheng, Xiaoling Su, Guoyong Leng, Ziyan Li, Yi Guo, Wei Fang, and Yongjia Liu. Propagation characteristics and mechanism from meteorological to agricultural drought in various seasons. Journal of Hydrology , 610:127897, 2022.

Shengpeng Cao, Lifeng Zhang, Yi He, Yali Zhang, Yi Chen, Sheng Yao, Wang Yang, and Qiang Sun. Effects and contributions of meteorological drought on agricultural drought under different climatic zones and vegetation types in northwest china. Science of The Total Environment , 821:153270, 2022.

Peter A. Stott, Nikolaos Christidis, Friederike E. L. Otto, Ying Sun, Jean-Paul Vanderlinden, Geert Jan van Oldenborgh, Robert Vautard, Hans von Storch, Peter Walton, Pascal Yiou, and Francis W. Zwiers. Attribution of extreme weather and climate-related events. WIREs Climate Change , 7(1):23-41, 2016.

Zengchao Hao, Vijay P. Singh, and Youlong Xia. Seasonal drought prediction: Advances, challenges, and future prospects. Reviews of Geophysics , 56(1):108-141, 2018.

Mahashweta Das and Srinivasan Parthasarathy. Anomaly detection and spatio-temporal analysis of global climate system. In Proceedings of the Third International Workshop on Knowledge Discovery from Sensor Data , SensorKDD '09, page 142-150, New York, NY, USA, 2009. Association for Computing Machinery.

Sancho Salcedo-Sanz, Jorge Pérez-Aracil, Guido Ascenso, Javier Del Ser, David Casillas-Pérez, Christopher Kadow, Dušan Fister, David Barriopedro, Ricardo García-Herrera, Matteo Giuliani, et al. Analysis, characterization, prediction, and attribution of extreme atmospheric events with machine learning and deep learning techniques: a review. Theoretical and Applied Climatology , 155(1):1-44, 2024.

Yunjie Liu, Evan Racah, Joaquin Correa, Amir Khosrowshahi, David Lavers, Kenneth Kunkel, Michael Wehner, William Collins, et al. Application of deep convolutional neural networks for detecting extreme weather in climate datasets. arXiv preprint arXiv:1605.01156 , 2016.

G. Sepulcre-Canto, S. Horion, A. Singleton, H. Carrao, and J. Vogt. Development of a combined drought indicator to detect agricultural drought in europe. Natural Hazards and Earth System Sciences , 12(11):3519-3531, 2012.

Yu Zhang, Zengchao Hao, Sifang Feng, Xuan Zhang, Yang Xu, and Fanghua Hao. Agricultural drought prediction in china based on drought propagation and large-scale drivers. Agricultural Water Management , 255:107028, 2021.

Xiang Zhang, Nengcheng Chen, Jizhen Li, Zhihong Chen, and Dev Niyogi. Multi-sensor integrated framework and index for agricultural drought monitoring. Remote Sensing of Environment , 188:141-163, 2017.

Yohannes Yihdego, Babak Vaheddoost, and Radwan A Al-Weshah. Drought indices and indicators revisited. Arabian Journal of Geosciences , 12:1-12, 2019.

Mohamed Dahoui. Use of machine learning for the detection and classification of observation anomalies, 01/2023 2023.

Peter Düben, Umberto Modigliani, Alan Geer, Stephan Siemen, Florian Pappenberger, Peter Bauer, Andy Brown, Martin Palkovic, Baudouin Raoult, Nils Wedi, and Vasileios Baousis. Machine learning at ecmwf: A roadmap for the next 10 years, 01/2021 2021.

M. Flach, F. Gans, A. Brenning, J. Denzler, M. Reichstein, E. Rodner, S. Bathiany, P. Bodesheim, Y. Guanche, S. Sippel, and M. D. Mahecha. Multivariate anomaly detection for earth observations: a comparison of algorithms and feature extraction techniques. Earth System Dynamics , 8(3):677-696, 2017.

Evan Racah, Christopher Beckham, Tegan Maharaj, Samira Ebrahimi Kahou, Mr Prabhat, and Chris Pal. Extremeweather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events. Advances in neural information processing systems , 30, 2017.

Prabhat, K. Kashinath, M. Mudigonda, S. Kim, L. Kapp-Schwoerer, A. Graubner, E. Karaismailoglu, L. von Kleist, T. Kurth, A. Greiner, A. Mahesh, K. Yang, C. Lewis, J. Chen, A. Lou, S. Chandran, B. Toms, W. Chapman, K. Dagon, C. A. Shields, T. O'Brien, M. Wehner, and W. Collins. Climatenet: an expert-labeled open dataset and deep learning architecture for enabling high-precision analyses of extreme weather. Geoscientific Model Development , 14(1):107-124, 2021.

Elizabeth Weirich-Benet, Maria Pyrina, Bernat Jiménez-Esteve, Ernest Fraenkel, Judah Cohen, and Daniela I. V. Domeisen. Subseasonal prediction of central european summer heatwaves with linear and random forest machine learning models. Artificial Intelligence for the Earth Systems , 2(2):e220038, 2023.

Adam B. Barrett, Steven Duivenvoorden, Edward E. Salakpi, James M. Muthoka, John Mwangi, Seb Oliver, and Pedram Rowhani. Forecasting vegetation condition for drought early warning systems in pastoral communities in kenya. Remote Sensing of Environment , 248:111886, 2020.

Christian Requena-Mesa, Vitus Benson, Markus Reichstein, Jakob Runge, and Joachim Denzler. Earthnet2021: A large-scale dataset and challenge for earth surface forecasting as a guided video prediction task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pages 1132-1142, June 2021.

Vitus Benson, Claire Robin, Christian Requena-Mesa, Lazaro Alonso, Nuno Carvalhais, José Cortés, Zhihan Gao, Nora Linscheid, Mélanie Weynants, and Markus Reichstein. Multi-modal learning for geospatial vegetation forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 27788-27799, June 2024.

M. H. Shams Eddin and J. Gall. Focal-tsmp: deep learning for vegetation health prediction and agricultural drought assessment from a regional climate simulation. Geoscientific Model Development , 17(7):29873023, 2024.

Jakob Runge, Sebastian Bathiany, Erik Bollt, Gustau Camps-Valls, Dim Coumou, Ethan Deyle, Clark Glymour, Marlene Kretschmer, Miguel D Mahecha, Jordi Muñoz-Marí, et al. Inferring causation from time series in earth system sciences. Nature communications , 10(1):2553, 2019.

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612634, 2021.

Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems , 35:38319-38331, 2022.

Xavier-Andoni Tibau, Christian Reimers, Andreas Gerhardus, Joachim Denzler, Veronika Eyring, and Jakob Runge. A spatiotemporal stochastic climate model for benchmarking causal discovery methods for teleconnections. Environmental Data Science , 1:e12, 2022.

Julien Boussard, Chandni Nagda, Julia Kaltenborn, Charlotte Emilie Elektra Lange, Philippe Brouillard, Yaniv Gurwicz, Peer Nowack, and David Rolnick. Towards causal representations of climate model data. arXiv preprint arXiv:2312.02858 , 2023.

Jakob Runge, Andreas Gerhardus, Gherardo Varando, Veronika Eyring, and Gustau Camps-Valls. Causal inference for time series. Nature Reviews Earth & Environment , 4(7):487-505, 2023.

Raanan Yehezkel Rohekar, Shami Nisimov, Yaniv Gurwicz, and Gal Novik. From temporal to contemporaneous iterative causal discovery in the presence of latent confounders. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 39939-39950. PMLR, 23-29 Jul 2023.

Jose María Tárraga, Eva Sevillano-Marco, Jordi Muñoz-Marí, María Piles, Vasileios Sitokonstantinou, Michele Ronco, María Teresa Miranda, Jordi Cerdà, and Gustau Camps-Valls. Causal discovery reveals complex patterns of drought-induced displacement. iScience , 27(9):110628, 2024.

J. Vogel, P. Rivoire, C. Deidda, L. Rahimi, C. A. Sauter, E. Tschumi, K. van der Wiel, T. Zhang, and J. Zscheischler. Identifying meteorological drivers of extreme impacts: an application to simulated crop yields. Earth System Dynamics , 12(1):151-172, 2021.

Jakob Runge, Peer Nowack, Marlene Kretschmer, Seth Flaxman, and Dino Sejdinovic. Detecting and quantifying causal associations in large nonlinear time series datasets. Science Advances , 5(11):eaau4996, 2019.

Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. Advances in neural information processing systems , 12, 1999.

Philipp Liznerski, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and KlausRobert Müller. Explainable deep one-class classification. In International Conference on Learning Representations , 2021.

Jihun Yi and Sungroh Yoon. Patch svdd: Patch-level svdd for anomaly detection and segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV) , November 2020.

Shenzhi Wang, Liwei Wu, Lei Cui, and Yujun Shen. Glancing at the patch: Anomaly localization with global and local feature comparison. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 254-263, 2021.

Thomas Defard, Aleksandr Setkov, Angelique Loesch, and Romaric Audigier. Padim: A patch distribution modeling framework for anomaly detection and localization. In Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani, editors, Pattern Recognition. ICPR International Workshops and Challenges , pages 475-489, Cham, 2021. Springer International Publishing.

Hanqiu Deng and Xingyu Li. Anomaly detection via reverse distillation from one-class embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9737-9746, June 2022.

Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H Rohban, and Hamid R Rabiee. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14902-14912, 2021.

Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Studentteacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4183-4192, 2020.

Guodong Wang, Shumin Han, Errui Ding, and Di Huang. Student-teacher feature pyramid matching for anomaly detection. arXiv preprint arXiv:2103.04257 , 2021.

Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2806-2814, June 2021.

Oliver Rippel, Patrick Mertens, Eike König, and Dorit Merhof. Gaussian anomaly detection by modeling the distribution of normal data in pretrained deep features. IEEE Transactions on Instrumentation and Measurement , 70:1-13, 2021.

Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14298-14308, 2022.

Zhikang Liu, Yiming Zhou, Yuansheng Xu, and Zilei Wang. Simplenet: A simple network for image anomaly detection and localization. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20402-20411, 2023.

Songqiao Han, Xiyang Hu, Hailiang Huang, Minqi Jiang, and Yue Zhao. ADBench: Anomaly detection benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Laurenz Strothmann, Uwe Rascher, and Ribana Roscher. Detection of anomalous grapevine berries using all-convolutional autoencoders. In IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium , pages 3701-3704, 2019.

Pankaj Mishra, Riccardo Verk, Daniele Fornasier, Claudio Piciarelli, and Gian Luca Foresti. VT-ADL: A vision transformer network for image anomaly detection and localization. In 30th IEEE/IES International Symposium on Industrial Electronics (ISIE) , June 2021.

Yunseung Lee and Pilsung Kang. Anovit: Unsupervised anomaly detection and localization with vision transformer-based encoder-decoder. IEEE Access , 10:46717-46724, 2022.

Miro Miranda, Laura Zabawa, Anna Kicherer, Laurenz Strothmann, Uwe Rascher, and Ribana Roscher. Detection of anomalous grapevine berries using variational autoencoders. Frontiers in Plant Science , 13, 2022.

Mohammad Sabokrou, Masoud Pourreza, Mohsen Fayyaz, Rahim Entezari, Mahmood Fathy, Jürgen Gall, and Ehsan Adeli. Avid: Adversarial visual irregularity detection. In C.V. Jawahar, Hongdong Li, Greg Mori, and Konrad Schindler, editors, Computer Vision - ACCV 2018 , pages 488-505, Cham, 2019. Springer International Publishing.

Shashanka Venkataramanan, Kuan-Chuan Peng, Rajat Vikram Singh, and Abhijit Mahalanobis. Attention guided anomaly localization in images. In European Conference on Computer Vision , pages 485-503. Springer, 2020.

Marcella Astrid, Muhammad Zaigham Zaheer, and Seung-Ik Lee. Synthetic temporal anomaly guided end-to-end video anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 207-214, 2021.

Wenrui Liu, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Diversity-measurable anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12147-12156, 2023.

Zhiwei Yang, Jing Liu, Zhaoyang Wu, Peng Wu, and Xiaotao Liu. Video event restoration based on keyframes for video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14592-14601, 2023.

Neelu Madan, Nicolae-C˘ at˘ alin Ristea, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B. Moeslund, and Mubarak Shah. Self-supervised masked convolutional transformer block for anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(1):525-542, 2024.

Julian Wyatt, Adam Leach, Sebastian M. Schmon, and Chris G. Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 649-655, 2022.

Haoyang He, Yuhu Bai, Jiangning Zhang, Qingdong He, Hongxu Chen, Zhenye Gan, Chengjie Wang, Xiangtai Li, Guanzhong Tian, and Lei Xie. Mambaad: Exploring state space models for multi-class unsupervised anomaly detection. arXiv preprint arXiv:2404.06564 , 2024.

Zhiyuan You, Lei Cui, Yujun Shen, Kai Yang, Xin Lu, Yu Zheng, and Xinyi Le. A unified model for multi-class anomaly detection. Advances in Neural Information Processing Systems , 35:4571-4584, 2022.

Chaoqin Huang, Qinwei Xu, Yanfeng Wang, Yu Wang, and Ya Zhang. Self-supervised masking for unsupervised anomaly detection and localization. IEEE Transactions on Multimedia , 25:4426-4438, 2023.

Dezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang, Can Ma, Qixiang Ye, and Weiping Wang. Video cloze procedure for self-supervised spatio-temporal learning. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11701-11708, 2020.

Hannah M. Schlüter, Jeremy Tan, Benjamin Hou, and Bernhard Kainz. Natural synthetic anomalies for self-supervised anomaly detection and localization. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , pages 474-489, Cham, 2022. Springer Nature Switzerland.

Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. In European Conference on Computer Vision , pages 494-511. Springer, 2022.

Vitjan Zavrtanik, Matej Kristan, and Danijel Skoˇ caj. Draem-a discriminatively trained reconstruction embedding for surface anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8330-8339, 2021.

Anne-Sophie Collin and Christophe De Vleeschouwer. Improved anomaly detection by training an autoencoder with skip connections on images corrupted with stain-shaped noise. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 7915-7922. IEEE, 2021.

Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9659-9669, 2021.

Masoud Pourreza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou. G2d: Generate to detect anomaly. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2003-2012, 2021.

Muhammad Zaigham Zaheer, Jin-ha Lee, Marcella Astrid, and Seung-Ik Lee. Old is gold: Redefining the adversarially learned one-class classifier training paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14183-14193, 2020.

Kamalakar Vijay Thakare, Yash Raghuwanshi, Debi Prosad Dogra, Heeseung Choi, and Ig-Jae Kim. Dyannet: A scene dynamicity guided self-trained video anomaly detection network. In Proceedings of the IEEE/CVF Winter conference on applications of computer vision , pages 5541-5550, 2023.

Jingtao Li, Xinyu Wang, Hengwei Zhao, Liangpei Zhang, and Yanfei Zhong. A unified remote sensing anomaly detector across modalities and scenes via deviation relationship learning. arXiv preprint arXiv:2310.07511 , 2023.

Ximiao Zhang, Min Xu, and Xiuzhuang Zhou. Realnet: A feature selection network with realistic synthetic anomaly for anomaly detection. arXiv preprint arXiv:2403.05897 , 2024.

Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In International Conference on Learning Representations , 2022.

Dongha Lee, Sehun Yu, Hyunjun Ju, and Hwanjo Yu. Weakly supervised temporal anomaly segmentation with dynamic time warping. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7355-7364, 2021.

Zhijie Zhong, Zhiwen Yu, Yiyuan Yang, Weizheng Wang, and Kaixiang Yang. Patchad: Patch-based mlp-mixer for time series anomaly detection. arXiv preprint arXiv:2401.09793 , 2024.

Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data. Proceedings of VLDB , 15(6):1201-1214, 2022.

Yu Zheng, Huan Yee Koh, Ming Jin, Lianhua Chi, Khoa T. Phan, Shirui Pan, Yi-Ping Phoebe Chen, and Wei Xiang. Correlation-aware spatial-temporal graph learning for multivariate time-series anomaly detection. IEEE Transactions on Neural Networks and Learning Systems , 35(9):11802-11816, 2024.

Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H. Li, and Ge Li. Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Muhammad Zaigham Zaheer, Arif Mahmood, Marcella Astrid, and Seung-Ik Lee. Claws: Clustering assisted weakly supervised learning with normalcy suppression for anomalous event detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16 , pages 358-376. Springer, 2020.

Hui Lv, Chuanwei Zhou, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Localizing anomalies from weakly-labeled videos. IEEE Transactions on Image Processing , 30:4505-4515, 2021.

Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng. Mist: Multiple instance self-training framework for video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14009-14018, 2021.

Seongheon Park, Hanjae Kim, Minsu Kim, Dahye Kim, and Kwanghoon Sohn. Normality guided multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2665-2674, January 2023.

Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 3769-3777, 2023.

Zhenting Qi, Ruike Zhu, Zheyu Fu, Wenhao Chai, and Volodymyr Kindratenko. Weakly supervised twostage training scheme for deep video fight detection model. In 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI) , pages 677-685. IEEE, 2022.

Jhih-Ciang Wu, He-Yen Hsieh, Ding-Jie Chen, Chiou-Shann Fuh, and Tyng-Luh Liu. Self-supervised sparse representation for video anomaly detection. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 , pages 729-745, Cham, 2022. Springer Nature Switzerland.

Hitesh Sapkota and Qi Yu. Bayesian nonparametric submodular video partition for robust anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3212-3221, 2022.

Minqi Jiang, Chaochuan Hou, Ao Zheng, Xiyang Hu, Songqiao Han, Hailiang Huang, Xiangnan He, Philip S Yu, and Yue Zhao. Weakly supervised anomaly detection: A survey. arXiv preprint arXiv:2302.04549 , 2023.

Tao Jiang, Weiying Xie, Yunsong Li, Jie Lei, and Qian Du. Weakly supervised discriminative learning with spectral constrained generative adversarial network for hyperspectral anomaly detection. IEEE Transactions on Neural Networks and Learning Systems , 33(11):6504-6517, 2022.

Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6479-6488, 2018.

Boyang Wan, Yuming Fang, Xue Xia, and Jiajie Mei. Weakly supervised video anomaly detection via center-guided discriminative learning. In 2020 IEEE International Conference on Multimedia and Expo (ICME) , pages 1-6, 2020.

Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W. Verjans, and Gustavo Carneiro. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4955-4966, 2021.

Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-Chung Wu. Mgfn: Magnitude-contrastive glance-and-focus network for weakly-supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 387-395, 2023.

Yixuan Zhou, Yi Qu, Xing Xu, Fumin Shen, Jingkuan Song, and Hengtao Shen. Batchnorm-based weakly supervised video anomaly detection. arXiv preprint arXiv:2311.15367 , 2023.

Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8022-8031, 2023.

Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 32023211, 2022.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation, 2023.

Aren Jansen, Daniel PW Ellis, Shawn Hershey, R Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 121-125. IEEE, 2020.

Wenze Yang, Felix Kogan, and Wei Guo. An ongoing blended long-term vegetation health product for monitoring global food security. Agronomy , 10(12), 2020.

Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Elías Hólm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean-Noël Thépaut. The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society , 146(730):1999-2049, 2020.

S. Schimanke, M. Ridal, P. Le Moigne, L. Berggren, P. Undén, R. Randriamampianina, U. Andrea, E. Bazile, A. Bertelsen, P. Brousseau, P. Dahlgren, L. Edvinsson, A. El Said, M. Glinton, S. Hopsch, L. Isaksson, R. Mladek, E. Olsson, A. Verrelle, and Z.Q. Wang. Cerra sub-daily regional reanalysis data for europe on single levels from 1984 to present, 2021.

J. Muñoz Sabater, E. Dutra, A. Agustí-Panareda, C. Albergel, G. Arduini, G. Balsamo, S. Boussetta, M. Choulga, S. Harrigan, H. Hersbach, B. Martens, D. G. Miralles, M. Piles, N. J. Rodríguez-Fernández, E. Zsoter, C. Buontempo, and J.-N. Thépaut. Era5-land: a state-of-the-art global reanalysis dataset for land applications. Earth System Science Data , 13(9):4349-4383, 2021.

F. Giorgi, C. Jones, and G. R. Asrar. Addressing climate information needs at the regional level: the cordex framework. Bulletin - World Meteorological Organization , 58(3):175-183, 2009.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 3319-3328. JMLR.org, 2017.

Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data (TKDD) , 6(1):1-39, 2012.

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 , 2023.

Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343 , 2023.

Christian Lessig, Ilaria Luise, Bing Gong, Michael Langguth, Scarlet Stadler, and Martin Schultz. Atmorep: A stochastic model of atmosphere dynamics using large scale representation learning. arXiv preprint arXiv:2308.13280 , 2023.

Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning , pages 3915-3924. PMLR, 2022.

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. In International Conference on Learning Representations , 2022.

Woncheol Shin, Gyubok Lee, Jiyoung Lee, Eunyi Lyou, Joonseok Lee, and Edward Choi. Exploration into translation-equivariant image quantization. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. In The Twelfth International Conference on Learning Representations , 2024.

Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417 , 2024.

Xiaohuan Pei, Tao Huang, and Chang Xu. Efficientvmamba: Atrous selective scan for light weight visual mamba. arXiv preprint arXiv:2403.09977 , 2024.

Guo Chen, Yifei Huang, Jilan Xu, Baoqi Pei, Zhe Chen, Zhiqi Li, Jiahao Wang, Kunchang Li, Tong Lu, and Limin Wang. Video mamba suite: State space model as a versatile alternative for video understanding. arXiv preprint arXiv:2403.09626 , 2024.

Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977 , 2024.

Tao Huang, Xiaohuan Pei, Shan You, Fei Wang, Chen Qian, and Chang Xu. Localmamba: Visual state space model with windowed selective scan. arXiv preprint arXiv:2403.09338 , 2024.

Jinyoung Park, Hee-Seon Kim, Kangwook Ko, Minbeom Kim, and Changick Kim. Videomamba: Spatio-temporal selective state space model. arXiv preprint arXiv:2407.08476 , 2024.

Vincent Tao Hu, Stefan Andreas Baumann, Ming Gui, Olga Grebenkova, Pingchuan Ma, Johannes Fischer, and Björn Ommer. Zigma: A dit-style zigzag mamba diffusion model. In Arxiv , 2024.

Almudena García-García, Francisco José Cuesta-Valero, Diego G Miralles, Miguel D Mahecha, Johannes Quaas, Markus Reichstein, Jakob Zscheischler, and Jian Peng. Soil heat extremes can outpace air temperature extremes. Nature Climate Change , 13(11):1237-1241, 2023.

Preet Lal, Gurjeet Singh, Narendra N. Das, Andreas Colliander, and Dara Entekhabi. Assessment of era5-land volumetric soil water layer product using in situ and smap soil moisture observations. IEEE Geoscience and Remote Sensing Letters , 19:1-5, 2022.

Dara Entekhabi, Eni G. Njoku, Peggy E. O'Neill, Kent H. Kellogg, Wade T. Crow, Wendy N. Edelstein, Jared K. Entin, Shawn D. Goodman, Thomas J. Jackson, Joel Johnson, John Kimball, Jeffrey R. Piepmeier, Randal D. Koster, Neil Martin, Kyle C. McDonald, Mahta Moghaddam, Susan Moran, Rolf Reichle, J. C. Shi, Michael W. Spencer, Samuel W. Thurman, Leung Tsang, and Jakob Van Zyl. The soil moisture active passive (smap) mission. Proceedings of the IEEE , 98(5):704-716, 2010.

Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Xianfeng Liu, Xiufang Zhu, Yaozhong Pan, Shuangshuang Li, Yanxu Liu, and Yuqi Ma. Agricultural drought monitoring: Progress, challenges, and prospects. Journal of Geographical Sciences , 26:750-767, 2016.

I. Meza, S. Siebert, P. Döll, J. Kusche, C. Herbert, E. Eyshi Rezaei, H. Nouri, H. Gerdener, E. Popat, J. Frischen, G. Naumann, J. V. Vogt, Y. Walz, Z. Sebesvari, and M. Hagenlocher. Global-scale drought risk assessment for agricultural systems. Natural Hazards and Earth System Sciences , 20(2):695-712, 2020.

María Pedro-Monzonís, Abel Solera, Javier Ferrer, Teodoro Estrela, and Javier Paredes-Arquiola. A review of water scarcity and drought indexes in water resources planning and management. Journal of Hydrology , 527:482-493, 2015.

Compton J. Tucker. Red and photographic infrared linear combinations for monitoring vegetation. Remote Sensing of Environment , 8(2):127-150, 1979.

F.N. Kogan. Application of vegetation index and brightness temperature for drought detection. Advances in Space Research , 15(11):91-100, 1995. Natural Hazards: Monitoring and Assessment Using Remote Sensing Technique.

Felix Kogan, Wei Guo, and Aleksandar Jelenak. Global vegetation health: Long-term data records. In Felix Kogan, Alfred Powell, and Oleg Fedorov, editors, Use of Satellite and In-Situ Data to Improve Sustainability , pages 247-255, Dordrecht, 2011. Springer Netherlands.

F. N. Kogan. Remote sensing of weather impacts on vegetation in non-homogeneous areas. International Journal of Remote Sensing , 11(8):1405-1419, 1990.

Anne F Van Loon, Tom Gleeson, Julian Clark, Albert IJM Van Dijk, Kerstin Stahl, Jamie Hannaford, Giuliano Di Baldassarre, Adriaan J Teuling, Lena M Tallaksen, Remko Uijlenhoet, et al. Drought in the anthropocene. Nature Geoscience , 9(2):89-91, 2016.

Jiawei Zhuang, raphael dussin, André Jüling, and Stephan Rasp. JiaweiZhuang/xESMF: v0.3.0 Adding ESMF.LocStream capabilities, March 2020.

Mohamad Hakam Shams Eddin and Juergen Gall. Identifying spatio-temporal drivers of extreme events [data set], 2024.

A Synthetic data

To the best of our knowledge, ground truth labels of drivers or anomalies which are correlated with extreme events impacts barely exist. This is especially the case when it comes to drivers or anomalous events which are related to a specific definition of an extreme event within the Earth system (i.e., drought or flood). For this reason, we designed a framework to generate artificial datasets which can be adopted to the task. Our framework is inspired by and related to Flach, et al. [21]. However, we differ in the following two aspects:

(1) First, we aim to generate multivariate anomalous that are correlated with a specific extreme event, while their aim is to generate anomalous that can occur simultaneously in multivariate data streams similar to [78-81]. Think of an increasing/decreasing of temperature, existing approaches are interested in temperature anomalies regardless of the subsequent extreme events they might cause, while we focus on temperature anomalies that can cause a particular extreme events in the near future. (2) Second, we generate the synthetic data based on real-world data stream signals, while they use trigonometric functions (i.e., sine function) to mimic Earth observations across spacetime.

In the following, we explain our overall framework in more details:

(1) First, we generate the normal base signals B ∈ R V × T × Lat × Lon for a set of different variables V , where T is the temporal extension, and Lat and Lon are the spatial extensions. For instance, to synthesize CERRA [106] B signals, we take the mean values from the CERRA climatologoy pixel-wise. This represents the typical value of B at specific time (week) and location (lat, lon). By definition, B inherits the intrinsic properties of the simulated variables including the existence of seasonality and correlations among variables.

(2) In the next step, we induce binary extreme events E ex ∈ Z T × Lat × Lon 2 within the datacube and track the precise spatio-temporal location of these events. Similar to Flach, et. al [21], the type and duration of extreme events vary within the datacube. For instance, we alter the duration of the events between long and short extreme events. The spatial distribution of the extreme event vary also between a local event at one pixel (LocalEvent), a rectangular event (CubeEvent), a Gaussian shape (GaussianEvent), an onset event that starts at specific time and lasts until the end of the series (OnsetEvent), and a random walk event that starts at a specific pixel and affecting neighboring pixels with time (RandomWalkEvent).

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

Time

albedo

2m temperature

total cloud cover

total precipitation

relative humidity

volumetric soil moisture

Simulated variables

normal event

anomalous event

extreme event

We randomly define a coupling matrix M between the variables and the extreme event. An example is shown in Fig. 6, i.e., the anomalous events for 2m temperature start 5 time steps ( -∆ T a ) before the extreme and last for 4 time steps afterwards ( +∆ T a ). While albedo and relative humidity are not coupled with the extremes. Based on M and E ex , we generate the binary anomalous events matrix E a ∈ Z T × Lat × Lon 2 .

.

(4) Similar to steps (2)-(3), we generate binary random anomalous events E r ∈ Z T × Lat × Lon 2 However, these events are uncorrelated with the extremes and generated randomlies for all variables. (5) We sample noise signals N ∈ R V × T × Lat × Lon for each variable. The noise signals could be sampled from a normal Gaussian distribution (GaussianNoise), a standard Cauchy distribution

(CauchyNoise), a double exponential distribution (LaplaceNoise), or from a spatiotemporal correlated noise across the datacube (RedNoise). (6) Using the indices v ∈ { 1 , . . . , V } , t ∈ { 1 , . . . , T } , lat ∈ { 1 , . . . , Lat } , and lon ∈ { 1 , . . . , Lon } , the synthetic signal Φ ( v,t,lat,lon ) is generated with the following formula:

where Θ ( v,t,lat,lon ) is the induced anomaly, kb, kn , and ks are control parameters for the events magnitudes, σ N is the standard deviation of the noise signal, δ ∈ {-1 , 1 } is the predefined coupling sign with extremes from M , and Λ ( v,t,lat,lon ) controls the sign of the induced anomaly for extremes (i.e., a deficiency in soil moisture ( δ = -1 ) and an increased in temperature ( δ = +1 ) during an extreme drought event).

Using this framework, we generated 3 types of datasets; (1) synthetic CERRA reanalysis (Tables 3), (2) synthetic NOAA remote sensing (Tables 4), and (3) synthetic artificial data (Tables 5).

We generate the NOAA base signals from [104]. For the NOAA and artificial datasets, we also considered artificial linear (LinearCoupling) and non-linear (QuadraticCoupling) dependencies among the variables as additional data properties. For instance to generate a new dependent base signal from independent bases:

where w ( v ) ∈ R V are weighted coefficients sampled either from a normal (NormWeight) or a Lablace (LaplaceWeight) distribution. In addition, we add an option to generate disturbed weights where weights vary spatio-temporally based on the locations of anomalies. Anomalous events that occur in one of the independent base signals propagate to the new generated dependent signals.

The third dataset (Artificial) does not depend on real-world data but rather consists of basis signals with trigonometric functions where we also add a linear latitudinal gradient (latGrad). Finally, we mask out some regions to exhibit no anomalies i.e., pixels over water surface. Each generated dataset consists of 52 × 46 time steps corresponding to 46 years of simulated data in 7 -day intervals. Examples of the synthesized dataset can be found in Figs. 7-12. The results on the synthetic CERRA reanalysis are reported in Table 1 and the results on synthetic NOAA and the artificial data are shown Sec. B in Tables 6 and 7.

Synthetic artificial

Ind.

Dep.

Coupled variables Dimension

Extreme events

-△

T

a

+

△

T

a

%Extreme % Correlated

variables

variables

with extreme

anomalous

3

3

4

{lon=200,

{CubeEvent(n=200, sx=35, sy=35, sz=25),

9

4

1.24

1.81

lat=200,

RandomWalkEvent(n=1100, s=125),

time=

52

×

46

} LocalEvent(n=2600, sz=17),

%Random

GaussianEvent(n=340, sx=35, sy=35, sz=25),

anomalous

OnsetEvent(n=25, sx=17, sy=17, os=0.98)}

2.93

Base

Dependency Weights

Noise

Random events

δ

kb

kn

ks

SineBase(

-

-

RedNoise(

{CubeEvent(n=320, sx=35, sy=35, sz=25),

-

0.35

0.35

0.35

shift=0,

meu=0,

RandomWalkEvent(n=3000, s=125),

amp=3,

sigma=0.20)

LocalEvent(n=4000, sz=17),

nOsc=46,

GaussianEvent(n=300, sx=35, sy=35, sz=25)}

CosineBase(

-

-

LaplaceNoise(

{OnsetEvent(n=18, sx=17, sy=17, os=0.98),

+1

0.35

0.35

0.35

shift=0,

meu=0,

RandomWalkEvent(n=1800, s=125),

amp=3,

sigma=0.08,

nOsc=46,

lambda=1)

GaussianEvent(n=350, sx=35, sy=35, sz=25)}

ConstantBase( -

-

WhiteNoise(

{CubeEvent(n=300, sx=35, sy=35, sz=25),

-1

0.90

0.90

0.90

const=0,

meu=0,

RandomWalkEvent(n=2000, s=125),

sigma=0.07)

LocalEvent(n=2800, sz=17),

GaussianEvent(n=290, sx=35, sy=35, sz=25)}

-

Quadratic

Norm

WhiteNoise(

-

-1

0.35

0.35

0.35

Coupling()

Weight()

meu=0,

sigma=0.65)

-

Linear

Laplace

WhiteNoise(

-

-

0.35

0.35

0.35

Coupling()

Weight()

meu=0,

sigma=.065)

-

Linear

Laplace

WhiteNoise(

-

-1

0.35

0.35

0.35

Coupling()

Weight()

meu=0.,

sigma=.065)

albedo

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

2m temperature

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

total cloud cover

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

total precipitation

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

relative humidity

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

volumetric soil moisture

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

Time

NDVI

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

BT

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

variable 03

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

variable 04

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

variable 05

t0

-

t1

-

t2

-

t3

-

-

t5

-

t6

-

t7

Time

71

61

△to

-△t1

-△t2

-△t3

-△t4

-△t5

-△t6

-△t7

variable 01

△to

-△t1

△t2

△t3

△t5

-△t6

-△t7

variable 02

△to

-△t1

-△t2

-△t3

-△t4

-△t5

-△t6

-△t7

variable 03

△to

-△t1

-△t2

-△t3

-△t4

-△t5

-△t6

-△t7

variable 04

△to

-△t1

-△t2

-△t3

-△t4

-△t5

-△t6

-△t7

variable 05

△to

-△t1

-△t2

-△t3

-△t4

-△t5

-△t6

-△t7

variable06

Time

0.25

0.20

0.15

0

500

1000

1500

2000

timestep

2mtemperature

290

280

500

1000

1500

2000

timestep

total cloud cover

80

70

60

500

1000

1500

2000

timestep

total precipitation

500

1000

1500

2000

timestep

relative humidity

06

80

70

500

1000

1500

2000

timestep

volumetricsoil moisture

0.3

0.2

0.1

0

500

1000

timestep

1500

2000

simulated signal

anomalous event correlated with extremes

random anomalousevent

NDVI

500

1000

1500

2000

timestep

BT

290 -

280

500

1000

1500

2000

timestep

variable03

20000

19000

500

1000

1500

2000

timestep

variable 04

77.5

80.0

82.5

85.0 -

500

1000

1500

2000

timestep

variable 05

155

150

145

500

1000

1500

2000

timestep

simulated signal

anomalous event correlated with extremes

random anomalous event

variable01

2.5

0.0

2.5

0

500

1000

1500

2000

timestep

variable 02

2.5

0.0

2.5

500

1000

1500

2000

timestep

variable03

0.0

0.2

0.4 -

0.6

500

1000

1500

2000

timestep

variable 04

500

1000

1500

2000

timestep

variable05

2.5

0.0

2.5

500

1000

1500

2000

timestep

variable 06

500

1000

1500

2000

timestep

simulated signal

anomalous event correlated with extremes

random anomalous event

B Additional results on the synthetic data

In Tables 6 and 7, we report the results on the synthetic NOAA remote sensing and artificial data described in Sec. A. We noticed a drop of performance for all models for the synthetic artificial data (Table 7). This explained as the ratio of anomalies is higher than the other two datasets. Second, we used for this artificial dataset red noise and quadratic coupling to generate dependent base signals. This makes this dataset harder for training. SimpleNet exhibits a dramatic dropped out of performance when it is tested on these two synthetic datasets. This illustrates that the model performance is highly dependent on the dataset and the backbone the model was trained on. Our model still outperforms all baselines on these datasets.

†

Mamba backbone [111]

C Ablation studies

In this section, we do ablation analyses on different aspects of the proposed model. All experiments are done on the validation set of synthetic CERRA reanalysis and evaluated with the F1-scores for drivers anomalies and extreme events detection.

C.1 Quantization layer

In Table 8, we study the performance of our model with different vector quantization algorithms. The first row Threshold (Tanh) represents a simple straight through estimator. For this quantization, we first map the input into a scalar value followed by a Tanh activation. Then we set positive values to be anomalies ( q = 1 ). In Random Quantization (RQ) [114] the input is projected with a randomly initialized weights and then compared with a randomly initialized codes. Vector Quantization (VQ) is the standard quantizer which uses an Euclidean distance [101] or a cosine similarity [115]. We further add an orthogonality loss for VQ similar to [116]. Finite Scalar Quantization (FSQ) maps the input into a bounded scalar value. The code is then assigned based on the rounded value in the discrete space. As seen, Lockup free quantization (LFQ) [102] performs the best. We speculate that LFQ does not need to learn the code vectors which simplifies the task of quantization.

C.2 Objective function for quantization

This part studies the loss function L ( quantize ) used in Eq. (4). We denote the term ∥ Z l -sg (sign( Z l )) ∥ 2 2 as L ( commit ) , H [ E ( sign( Z l ) ) ] as L ( div ) , and E [ H ( sign( Z l ) ) ] as L ( ent ) . As seen from Table 9, L ( commit ) is essential for training to prevent the input for the quantization layer from growing. While L ( ent ) and L ( div ) (rows 4, 5, and 7) improve the results compared to the model with L ( commit ) (second row).

C.3 Objective function for extreme events prediction.

Due to class imbalanced, we add class weighting applied to the binary cross entropy loss L ( extreme ) to predict extremes. The weighting is based on the logarithm of the inverse square roots of the relative

class frequencies in the batch. The weighted loss function achieves better results as shown in Table 10.

C.4 Feature extractor ( f θ )

We evaluate three types of feature extractors, 3D CNN, Video Swin Transformer [100], and SSM Vision Mamba [111] with local scans. All models are trained from scratch. To build the 3D CNN model, we replaces the attention blocks in Swin Transformer with 3D convolutions and keep the overall architecture. For Mamba, we use local scans. Vision Mamba backbones replace the attention module with a linear selective state space model which provides an efficient alternative for Video Swin Transformer [118-124]. From Table 11 we notice that Swin Transformer achieves better results compared to 3D CNN. The parameters are also less compared to 3D CNN. The results also indicate that using Mamba instead of Swin Transformer achieves similar or better results when less parameters are used. In contrast to Swin Transformer, Mamba can be scaled to the global scale. However, we have used Swin Transformer in all experiments to have a fair comparison with the baselines due to its commonality as a backbone.

Table 11 shows that increasing the model parameters still does not show a sign of overfitting.

C.5 Lossy vs lossless driver detection

We trained the model without the quantization layer. In other words, we remove the driver detection step and trained the model to predict extreme droughts directly from the extracted features with one classification head. The reason behind this experiment is to check the information loss through the quantization/driver detection step. Table 12 shows the improvement in performance on both the synthetic and real-world datasets. Adding the driver detection step identifies the related drivers to the events. However, at the cost of a slight decrease in accuracy on extreme events prediction.

C.6 Weighting parameters in the main objective function

We study the role of the weighting parameters in the main loss function in Eq. (6). We can observe from Table 13 that λ ( driver ) plays a crucial role in anomaly detection. Having λ ( driver ) small makes the model less constrained and reduces supervision on where extremes were reported in the training data. Bigger values of λ ( driver ) make the model more constrained to identify drivers near or in overlap with the extremes and thus reduce the overall performance. We can also observe that the model is less sensitive to λ ( ent ) compared to other parameters. We selected the final default weighting parameters in the experiments based on the average performance on both driver and extreme events prediction.

C.7 Objective functions

Var02

Var03

Var02

Var03

Var02

Var03

Prediction

Prediction

Prediction

Ground

Ground

Ground

Truth

Truth

L(driver)

×

L(extreme)

L(driver)

L(extreme) x

L(driver)

√

L(extreme)

multi-heads

multi-heads

multi-heads

Without the loss L ( driver ) , the detection of drivers/anomalies is not reliable since pixels at regions and intervals where no extreme event occurred can be assigned to z q =1 (driver) as well. In case of L ( extreme ) multi-heads, we observe that anomalies are identified in a small subset of variables because the network omits some variables if there is a correlation with other variables. Please see Fig. 13. In case of a single head such flips occur less often, but they can occur. If the loss L ( driver ) is used, such flips cannot occur and the multi-head improves both F1-scores by a large margin. When comparing rows 2 and 4 in Table 2 (a), there is a slight decrease in extreme prediction but a large improvement in anomaly detection. Note that there is always a trade-off between extreme and anomaly detection. The anomalies generate a bottleneck of information. When more information goes through the bottleneck the better the extreme prediction gets. Without any anomaly detection, the extreme prediction is best as shown in Table 12, but the increase in F1 score is only moderate. This is also visible in rows 2 and 3 in Table 2 (b). Cross-attention improves extreme prediction, but it hurts the detection of anomalies since the information is propagated between the variables.

D Results on the real-world reanalysis data

D.1 Quantitative results

It is difficult to quantify the quality of the predicted drivers and anomalies on real-world data. Therefore, we hypothesize that the model can predict extreme agricultural droughts from the identified

drivers and anomalies, only if those drivers and anomalies are causally correlated with the extreme events. To verify this, we report in Table 14 the prediction accuracy on extreme agricultural droughts. The results show that using the identify drivers as input, the model can predict extreme agricultural droughts across different regions and datasets.

The performance depends on the type and ratio of extremes, spatio-temporal resolution, the quality and consistency between the remote sensing and the reanalysis data. F1 scores substantially increase when the threshold on VHI is increased (see Table. 16 in Sec. D.3). Note that it is not required to predict all extremes in order to learn specific relations from the predicted events.

D.2 Robustness

To further check the model robustness, we train the model on the same EUR-11 region with 6 different combinations of physical variables. We anticipate that if a variable is relevant to extremes over specific region i.e., Europe it should appear in all identified sets of variables i.e., we expect soil moisture (swvl1) and evaporation (e) to be always presented as explanatory variables for extremes over Europe. Our experimental results in Table 15 confirms this assumption.

D.3 Scientific validity

Surely, soil temperature is a key factor in the drought processing and soil moisture-temperature feedback [125]. State variable of the land surface such as albedo (fal/al) and soil temperature (stl1/sot) should be very related to reflectance on the ground and consequently to VHI from remote sensing. However, our approach indicates that these variables are not informative enough for the model to identify drivers and predict extremes. To investigate this issue, we conducted 4 more experiments on both CERRA and ERA5-Land where we trained models that take only one variable al/fal or stl as input and predict the extreme events directly without the driver/anomaly detection step (similar to Sec. C.5). In all of these experiments, the F1-score was very low. In the next experiment, we increased the threshold for VHI and trained new models to predict extremes directly. The results for the validation set are shown below in Table 16:

The first potential reason to consider is that some land surface variables might deviate from the reality. ERA5-Land does not use data assimilation directly. The evolution and the simulated land fields

are controlled by the ERA5 atmospheric forcing. Another reason might be that when training only on extremes (VHI < 26), there are not enough samples to learn the relations. Note that VHI is a combination of both TCI and VCI. Most extremes (VHI < 26) might result from a deficiency in both stl/t2m and vsw. This might also explain why stl and albedo cannot be that informative to predict very extreme events. Last row in Table 15 shows the result when we discard albedo and soil temperature as input variables from ERA5-Land.

Moreover, state variables of the hydrological cycle in ERA5-land like volumetric soil water variable (swvl1/vsw) has biases [126]. One solution to improve the validity of investigation is to use satellite observations for the top layer [127]. However, the experiments showed that the model relates soil moisture anomalies with the extremes in VHI and provides reasonable predictions.

D.4 Spatial Resolution

E Baselines

We evaluate our approach with an interpretable forecasting approach using integrated gradients [109] and 8 baselines from 3 main related categories in anomaly detection; one-class unsupervised [39, 110, 51], reconstruction-based [59, 65], and multiple instance learning [94-96]. Note that these baselines are not directly applicable to the task we addressed in this paper. We modified and trained all baselines from scratch. For this, we relied on the officially released codes and started from the default hyperparameters.

Integrated Gradients [109] is an axiomatic attribution method which can be used as a post-hoc method to explain the model prediction. We trained two models that predict extreme events directly from the input variables and then we applied a post-hoc integrated gradients from Pytorch Captum [128]. Both models use the same Swin Transformer backbone as our model but without the anomaly detection step. For Integrated Gradients II, we added a cross attention. For these baselines, we compute the gradient only with respect to predicted extremes and computed a different threshold for each variable separately.

Isolated Forest (IF) [110] is an ensemble of random trees that isolate input instances by randomly selecting abounded splits for features. A shorter averaged path for a recursive partitioning implies an anomalous input. We empirically set the number of estimators to 100 .

OCSVM [39] is a one-class support vector machine solved using stochastic gradient descent. We use a radial basis kernel with 100 components.

For IF and OCSVM, we extracted multivariate features as the distances between the input variables and then train a model on each input variable separately. We sampled 400k normal data points (locations without extreme flags) for training and defined the expected ratio of anomalous to be roughly equivalent to the ratio of extreme events in the data.

SimpleNet [51] is categorized as an embedding-based one-class algorithm for anomaly detection. In SimpleNet, a feature extractor is first used to extract local features from normal data followed by a feature adaptor. Then an anomaly feature generator induces anomalous features by adding a Gaussian noise in the feature space. Finally, a discriminator is trained to distinguish between the normal and anomalous generated features. During inference, the feature generator is discarded and the trained discriminator is expected to separate anomalous from normal input features. During inference, we compute the median anomaly score for both normal pixels and pixels with extreme flags. Then, we set a threshold for anomalous pixels based on the average of the two former median anomaly scores. We trained SimpleNet with our pretrained model as a feature extractor and set the feature dimension for feature adaptor to 512 . The extracted features are scaled by 10 -2 and the anomalous feature generator uses a Gaussian noise ϵ ∼ N (0 , σ = 1 . 5) . The discriminator composes of two linear layers with a hidden dimension of 96 . We set th + and th -to 1 . 0 and trained with AdamW optimizer for 48 epochs with a batch size of 2 and a learning rate of 3 × 10 -4 with 1 × 10 -5 weight decay.

STEALNet [59] or synthetic temporal anomaly guided end-to-end video anomaly detection. This is a reconstruction-based algorithm. STEALNet was trained to maximize the reconstruction loss for locations with extreme flags and to minimize the reconstruction loss otherwise. We set the dimensions for the auto-encoder as 96 , 128 , and 256 . We trained with a batch size of 8 for 100 epochs using Adam optimizer with 1 × 10 -3 weight decay and a learning rate of 3 × 10 -4 .

For STEALNet and UniAD, we compute during inference the mean reconstruction loss for both normal pixels and pixels with extreme flags. Then, we set a threshold for anomalous pixels based on the average of these two values.

DeepMIL [94] is a multiple instance learning for anomaly detection in surveillance videos. In MIL, each video is represented as a bag of snippets (instances). We define positive instances as pixels with extreme event flags where the exact information (which variable is anomalous) within the positive instances is unknown. In the original implementation, the ranking is enforced only on the top instance with the highest anomaly score in each bag. We modified the ranking loss to top-k and set K = 100 .

ARNet [95] or anomaly regression net is another MIL-based approach. The ranking loss is based on a top-k binary cross entropy. In addition, there is a center loss to reduce the intra-class discriminative features of normal instance. We set α = 400 and λ center = 20 .

RTFM [96] or robust temporal feature magnitude learning is a MIL-based algorithm which learns to distinguish between the normal and anomalous scores by selecting the top-k snippets with the highest feature magnitudes. We modified the multi-scale temporal network (MTN) to capture the local and global spatial dependencies between pixels. We set the dimension for MTN to 32 , K = 100 , the margin to separate features to × 10 2 and α = 1 × 10 -4 .

We use the same Video Swin Transformer backbone as our model to train DeepMIL, ARNet, and RTFM. During training, we noticed that the ranking loss becomes biases toward one variable when the ranking is computed among all variables, so we computed a loss for each variable and then average the losses. RTFM only worked when we added a cross attention between the variables. We suppose this to be related to the feature magnitude learning. We trained with a batch size of 2 using Adam optimizer and a learning rate of 6 × 10 -4 for 100 epochs with a weight decay of 1 × 10 -3 . We also added an instance dropout of 0 . 5 .

F Computational efficiency

In Table 18, we report the inference time with a fixed input of 6 variables, 8 days and 200 × 200 spatial resolution. STEALNet is based on a 3D CNN auto-encoder without a self-attentions which explains its efficiency but on the cost of accuracy. In the last row, we show the estimated time when we discard the classification head to detect extreme events and only use the model for anomaly detection.

The training on the real-world data for EUR-11 took about ∼ 21 hours with a Swin Transformer model, K = 16 , and 4 NVIDIA A 100 GPUs. Table 19 gives a rough estimation for training on the synthetic CERRA for 1 epoch. SimpleNet was trained with a pretrained backbone. The training time includes some postprocessing to compute metrics on the training set. The time might also differ depending on the I/O during training and the number of available workers.

G Implementation details

The training was done mainly on clusters with NVIDIA A100 80GB and NVIDIA A40 48GB GPUs. In the following, we highlight the main technical implementations and hyperparameters for training:

Synthetic data For the synthetic data we set the embedding dimension K = 16 . We use one layer Video Swin Transformer with {depth=[2, 1], heads=[2, 2], window size=[[2, 4, 4], [6, 1, 1]]}.

For the quantization layer, we use two sequential 3D CNN layers on each variable with: {[kernel=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1)]} and followed by a shared linear layer.

The classifier g ψ consists of the following layers: {[3D CNN, kernel=(3, 3, 3), stride=(3, 1, 1), padding=(0, 1, 1)], [3D CNN, kernel=(2, 3, 3), stride=(2, 1, 1), padding=(0, 1, 1)]}. We set λ ( ent ) = λ ( div ) = 0 . 1 , λ ( anomaly ) = 100 , and λ ( commit ) = 3 . The models were trained with Adam optimizer [129] for 100 epochs with a batch size of 4 . We use a linear warm up of 2 epochs and a cosine decay with an initial learning rate of 2 × 10 -3 and a weight decay of 3 × 10 -3 .

Reanalysis data We set the embedding dimension K = 24 for CERRA and CAS-11 datasets and K = 16 for the rest of ERA5-Land datasets. We use one layer Video Swin Transformer with {depth=[2, 1], heads=[2, 2], window size=[[2, 4, 4], [8, 1, 1]]}. The quantization layer is similar to the one for the synthetic data. The classifier g ψ consists of the 3 layers each has {[D CNN, kernel=(2, 3, 3), stride=(2, 1, 1), padding=(0, 1, 1)}. We set λ ( ent ) = λ ( div ) = 0 . 1 , and λ ( anomaly ) = 100 by defaults and λ ( commit ) = 1 . 0 for CAS-11. For CERRA, we set λ ( ent ) = λ ( div ) = 0 . 01 . Due to the high resolution on the reanalysis data, we use gradient checkpoint during training. For CERRA reanalysis, we also cut the boundaries between low and high latitudes focusing on the central region. This results in a final grid with 512×832 cells.

To handle missing data and temporal gaps in the input reanalysis data, we first normalize the data using the pre-computed statistics and then replace the invalid pixels with zero values.

H Reanalysis data

The raw reanalysis data were provided by the Climate Data Store (CDS) [107, 106]. Technical details regarding the reanalysis datasets are provided in Tables 20 and 21. CORDEX regions [108] used in this study are shown in Fig. 14. For all regions on ERA5-Land, we used the following variables: {"t2m", "fal", "e", "tp", "swvl1", "stl1"}. For CERRA, we did the experiments with: {"t2m", "al", "tcc", "tp", "vsw", "r2"}.

NAM-11

EUR-11

CAS-11

CERRA

SAM-11

AFR-11

EAS-11

I Agricultural drought definition and remote sensing data

I.1 Satellite-derived agricultural drought

It is generally challenging to define what exactly constitutes an extreme. Extremes can be categorized from the perspective of their impacts. For instance, extreme drought can be categorized into 4 types based on their impacts [130, 11]; meteorological or climatological drought, agricultural drought [131], hydrological drought, and socioeconomic drought [132]. Meteorological drought is mainly related to the dryness and can be defined based on a deficiency in temperature or precipitation. Agricultural drought measures the impact of stress on vegetation and usually defined as soil water deficits. It is also widely conceived that drought originates and progresses from meteorological into agricultural drought [6, 17]. Hydrological drought is related to the water storage. While socioeconomic drought can be measured based on supply and demand related to weather and deficit in water supply. In this paper, we are interested in extreme agricultural droughts.

NDVI

VCI

仑

VHI

VHI<26

BT

TCI

Worldwide satellite observations allow for almost real-time monitoring of drought and vegetation conditions. In practice, vegetation states can be estimated from land surface reflectances acquired from satellites. As a result, the reflectances on the ground can be employed as agricultural drought indicators and as proxies for vegetation health. To define extreme agricultural drought events, we processed satellite-based vegetation health dataset [104] from the National Oceanic and Atmospheric Adminis-

tration (NOAA) ( https://www.star.nesdis.noaa.gov/smcd/emb/vci/VH/index.php [last access: 22 May 2024)]). This dataset consists of long-term remote sensing data acquired from a system of NOAA satellites: the Advanced Very-High-Resolution Radiometer (AVHRR) which starts from 1981 until 2012 and the new system the Visible Infrared Imaging Radiometer Suite (VIIRS) from 2013 onwards. The dataset has a global coverage with ∼ 0 . 05 ◦ ( ∼ 4km) spatial resolution. The normalized difference vegetation index (NDVI) [133] and brightness temperature (BT) [134] are the two key products of the dataset. The BT is an infrared (IR)-based calibrated spectrum radiation. While the NDVI is a combination of the near-infrared (NIR) and red (R) bands. To remove the effects of clouds, atmospheric disturbance, and other error sources, the data were aggregated temporally into a smoothed product on a weekly basis. The weekly temporal coverage is needed for outliers and discontinuities removal and is suitable to study the phenological phases of vegetation and consequently to define agricultural drought [104, 135]. Based on the long-term upper and lower bounds of the ecosystem (maximum and minimum values of the NDVI and BT), agricultural drought indicators such as vegetation condition index (VCI), thermal condition index (TCI), and vegetation health index (VHI) can be derived [134, 136]. VHI is a combination of VCI and TCI (Fig. 15) and it fluctuates between 0 (unfavourable condition) and 100 (favourable condition). Values outside the range are clipped. Based on this definition of vegetation health, extreme agricultural drought can be defined when VHI < 26. Please note that vegetation stress detected by VHI could not be necessarily caused by a drought event i.e., a change in the land cover can change the signal as well [15, 137]. Thus, VHI should be interpreted carefully.

I.2 Pre-processing of the remote sensing data

The remote sensing dataset is provided on the Plate Carrée projection (geographic latitude and longitude). The target agricultural drought data and reanalysis data have to be aligned in the same coordinate systems and over the same regions. To realize this, we mapped the remote sensing data onto the Lambert conformal conical grid for CERRA and onto the rotated coordinate systems over the different CORDEX domains for ERA5-Land. For the mapping, we use the first-order conservative mapping using the software from Zhuang et. al [138]. To calculate the spatial averaged, we excluded coastal lines, invalid, and water body pixels. Furthermore, we combined the dataset with masks obtained from the quality assurance metadata for pixels over no vegetation and very cold areas. As mentioned in Sec. I.1, a temporal decomposition was conducted to remove some discontinuity and aggregate the data into a weekly product. However, some pixels will still be empty. To tackle this issue, we first checked if the pixel was covered by another satellite and averaged the measurements of the satellites. If it was not the case, we flagged the pixel as invalid and discard it from the training and evaluation. This remote sensing dataset serves as a reference of extreme agricultural drought events to train and evaluate the performance of the model. Table 22 shows the ratio of extreme events in the datasets. Please note that there is no ground truth for drivers or anomalies in our real-world dataset. We only report the ratio of extreme agricultural drought events, which can be detected using remote sensing data.

J Additional results

100

total_evaporation

forecast albedo

soiltemperature

50

volumetricsoilwater

2m temperature

totalprecipitation

100

total_evaporation

forecast albedo

soil temperature

% of weeks with anomalous events

50

volumetricsoilwater

2mtemperature

totalprecipitation

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

EUR-11,   year=2020, week=035

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-2

0

+2

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

EUR-11 - Summer: weeks (25 - 37), 2018 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

AFR-11,   year=2023, week=048

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-2

0

+2

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-2

0

+2

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

AFR-11 - Summer: weeks (25 - 37), 2019 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

SAM-11,   year=2018, week=039

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

SAM-11 - Summer: weeks (25 - 37), 2018 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

NAM-11,   year=2017, week=024

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1 0 +1

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

NAM-11 - Summer: weeks (25 - 37), 2018 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

EAS-11,   year=2019, week=016

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

EAS-11 - Summer: weeks (25 - 37), 2018 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

e

t0

fal

stl1

t2m

tp

e

-

t1

fal

stl1

t2m

tp

e

-

t2

fal

stl1

t2m

tp

e

-

t3

fal

stl1

t2m

tp

e

-

fal

stl1

t2m

tp

e

-

t5

fal

stl1

t2m

tp

e

-

t6

fal

stl1

t2m

tp

e

-

t7

fal

stl1

t2m

tp

CAS-11,   year=2021, week=031

25

50

75

100

% of anomalous pixels

total evaporation (e)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

forecast albedo (fal)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

25

50

75

100

% of anomalous pixels

soil temperature (stl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

volumetric soil water (swvl1)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

CAS-11 - Summer: weeks (25 - 37), 2018 - 2023

total evaporation (e)

forecast albedo (fal)

soil temperature (stl1)

volumetric soil water (swvl1)

2m temperature (t2m)

total precipitation (tp)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

Observed

extreme droughts

Predicted

extreme droughts

r2

t0

sot

sp

t2m

tp

vsw

r2

-

t1

sot

sp

t2m

tp

vsw

r2

-

t2

sot

sp

t2m

tp

vsw

r2

-

t3

sot

sp

t2m

tp

vsw

r2

-

sot

sp

t2m

tp

vsw

r2

-

t5

sot

sp

t2m

tp

vsw

r2

-

t6

sot

sp

t2m

tp

vsw

r2

-

t7

sot

sp

t2m

tp

vsw

CERRA,   year=2020, week=029

25

50

75

100

% of anomalous pixels

albedo (al)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

relative humidity (r2)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

2m temperature (t2m)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0 +1

z-score

25

50

75

100

% of anomalous pixels

total cloud cover (tcc)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

25

50

75

100

% of anomalous pixels

total precipitation (tp)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

+0

z-score

25

50

75

100

% of anomalous pixels

volumetric soil moisture (vsw)

beginning of

extreme drought

-

t26

-

t25

-

t24

-

t23

-

t22

-

t21

-

t20

-

t19

-

t18

-

t17

-

t16

-

t15

-

t14

-

t13

-

t12

-

t11

-

t10

-

t09

-

t08

-

t07

-

-

-

t04

-

t03

-

t02

-

t01

t0

+ t01

+ t02

+ t03

+ t04

+ t05

+ t06

+ t07

-1

0

+1

z-score

CERRA - Summer: weeks (25 - 37), 2016 - 2020

albedo (al)

relative humidity (r2)

2m temperature (t2m)

total cloud cover (tcc)

total precipitation (tp)

volumetric soil moisture (vsw)

% of pixels with severe-to-exceptional agricultural drought [VHI < 26]

normal condition [z-score = 0]

K Code and data availability

The source code to reproduce the results is available on GitHub at https://github.com/ HakamShams/IDEE . The source code for the synthetic data generation is also available on GitHub at https://github.com/HakamShams/Synthetic_Multivariate_Anomalies . The pre-processed data used in this study are available at https://doi.org/10.60507/FK2/RD9E33 [139].

L Broader impacts

There are generally no direct negative social impacts for conducting climate science researches. However, anomaly detection algorithms in general could be adapted for video surveillance and might infringe privacy considerations. Although this is the risk of developing anomaly detection algorithms.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Sec. 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper dose not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See implementation details in Sec. 5 and Appendix Sec. G.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and datasets are publicly available. See Sec. K.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See implementation details in Sec. 5 and Appendix Sec. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean and standard deviation of the results for 3 different random seed runs. See Tables 1, 6, 7, 14, and 18.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix Sec. F and Sec. G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Sec. 6 and Appendix Sec. L.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We think the paper does not pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original publications for the raw data and provide URLs when applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See Sec. 4 and Appendix Sec. A, H, and I.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author investigates a novel, significant, and practical problem: how to efficiently identify extreme anomaly events from climate data. To address the temporal delays between anomalies and extremes and the spatially uneven response of anomaly events, the author first innovatively constructs three comprehensive datasets, including synthetic and real-world datasets. Next, the author proposes an end-to-end spatio-temporal anomaly detection network model, with key concepts including independent spatio-temporal encoders and a compact quantization layer adapted for anomaly detection. Finally, the author conducts detailed experiments on extreme anomaly event detection tasks using two types of datasets to demonstrate the effectiveness of the proposed datasets and methods.\n\nSTRENGTHS:\n+ 1. This paper addresses an important but overlooked issue in practical scenarios: identifying extreme anomaly events from climate data. The motivation for this research is intuitive and substantial, as the task of recognizing extreme anomaly events is crucial for understanding climate patterns and can be applied in significant areas such as agricultural production and social activities.\n\n---\n\n+ 2. Unlike traditional anomaly detection, anomaly events typically exhibit temporal delay characteristics and spatially uneven responses. Therefore, the author first constructs an anomaly event detection dataset approximating the historical Earth system based on reanalysis and remote sensing datasets. However, there are reliability issues with extreme drought events obtained from remote sensing data as ground truth. Consequently, the author further develops multiple synthetic datasets to facilitate more convenient and reasonable experimental analysis and guidance. I greatly admire and appreciate the immense effort behind this work.\n\n---\n\n+ 3. Furthermore, the author presents an end-to-end neural network solution. Despite the limited technical innovations, the performance achieved is still impressive. In particular, the thorough experimental comparisons and analyses are commendable.\n\nWEAKNESSES:\n+ 1. This paper involves extensive preprocessing and adaptation of datasets. Although the author has not released the related code and data details, I believe a thorough introduction to the datasets or code is necessary for future use, which is crucial for subsequent work. This raises another issue: I find the contributions from the problem and dataset introduction to be overwhelming compared to those from the method itself. Therefore, it seems more suitable for the datasets and benchmarks track rather than the research track.\n\n---\n\n+ 2. This paper appears to remain limited to regional-level anomaly event detection. However, we know that the entire Earth system is dynamically interconnected and evolving; changes in one corner of the Earth can significantly impact distant regions (e.g., the butterfly effect). To model the spatiotemporal dynamics at the global level, the current backbone network seems inadequate for such scalability.\n\n---\n\n+ 3. Although the paper is structurally clear, there is still much room for improvement in writing and presentation. Particularly for interdisciplinary papers, the author should not assume that readers have backgrounds in multiple disciplines. Necessary background knowledge and related work should be supplemented as thoroughly as possible.\n\n---\n\n+ 4. I have also listed several confusions and suggestions in the paper. If the author can address my concerns, I would be willing to support its publication here and recommend an increase in its score, even though it is more suitable for the DB track.\n\nQUESTIONS:\n+ 1. The second contribution point in Section 1 should be swapped with the first to reflect the paper's primary contribution more logically. The main contribution appears to be the introduction of a novel anomaly detection problem in Earth sciences, supported by detailed benchmark experiments, while the methodological innovation is limited and should be de-emphasized.\n\n---\n\n+ 2. The anomaly detection part in Section 2 is overly redundant. It should focus on the works most relevant to the new anomaly detection approach proposed in this paper, emphasizing the connections and main differences with existing work, as well as the motivations.\n\n---\n\n+ 3. In line 113, the phrase \"but in spatio-temporal configurations of variables that potentially cause an extreme event with some time delay and at a potentially different location\" lacks clarity about how these variables differ from standard anomaly variables. I suggest the author provide a simple illustrative example figure to help readers understand.\n\n---\n\n+ 4. Considering the large parameters of the video Swin-Transformer, is it suitable as a backbone network for global-level spatiotemporal anomaly detection? There are many lightweight alternative spatiotemporal prediction backbone networks. The author could consider adding experiments with different backbone networks. Although this is not mandatory, it would be commendable.\n\n---\n\n+ 5. Regarding the dataset, what is the spatial resolution? Specifically, what is the actual area of one pixel? How do different resolutions impact the final detection results? Are there any experiments addressing this? \n\n---\n\n+ 6. In Table 2(a), the comparison between the first and second rows shows that while the $\\mathcal{L}_{(extreme)}$ loss term improves extreme detection, it harms anomaly prediction. Ideally, it should benefit both. Why is this the case? Can you provide an explanation? Similarly, adding the $\\mathcal{L}(anomal)$ loss term (comparing the third and first rows) shows it helps both types of detection, but the comparison between the fourth and second rows indicates it harms extreme detection, which seems contradictory. I hope the author can provide reasonable supplementary experiments and explanations. \n\n---\n\n+ 7. The text in the figures is too small and not reader-friendly. It should be just one size smaller than the main text for better readability.\n\nLIMITATIONS:\nI have already listed them in the question section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to identify the atmospheric drivers of extreme droughts. For this, they assume that for every impact of extreme droughts measurable with remote sensing, there is a precursor signal in assimilated land surface and meteorological data. The work proposes to identify these precursor signals with inherently interpretable machine learning: a computer vision model is trained to map input data into a binary latent space. These binary encodings are subsequently used to predict the future occurance of drought impacts. In that way, the binary encodings are assumed to be interpretable as \"anomaly in atmospheric data\" and \"normal atmospheric data\". Models trained in this way achieve good prediction skill (F1 score ~0.9) of drought impacts on synthetic data. Also, the binary encodings match anomalies in the synthetic input data reasonably well (F1 score ~0.8). The prediction skill on real world data is rather low: F1 Score ~0.2 for drought impacts.\n\nSTRENGTHS:\n1. This work introduces a potentially novel variant of anomaly detection: Detecting only those Anomalies that are predictive for correlated impacts. This variant is relevant for studying the drivers of extreme drought impacts.\n2. The work compares a wide array of baselines and performs many ablation studies.\n3. Synthetic experiments are conducted to study the proposed method before shifting to real-world data\n4. The main text of the paper is reasonably concisely written, with many additional details supplied in the appendix.\n\nWEAKNESSES:\nMajor points:\n1. Confusing terminology: The authors speak about \"anomalies\" and \"extreme events\" without properly defining what is meant with which term. Furthermore I believe the used terminology is non-standard in the field, and propose the authors to instead use:\n    - Land surface Impacts of extreme events: These are what you call \"extreme events\", i.e. the VHI below a certain threshold. I would argue what you mean is the impact on state variables representing the land surface state (the ecosystem health) of extreme events. In your case droughts, but this could be any type of extreme events.\n    - Atmospheric drivers of extreme events: These are what you call \"anomalies\", but both the VHI below a certain threshold or the surface temperature above a certain level could be considered anomalous. Hence I recommend you rather focus on drivers here, these could be atmospheric or hydrological state variables (e.g. temperature or soil moisture) or land-atmosphere fluxes (e.g. evaporation).\n    - Luckily, you should be able to resolve this issue through simply rewriting your article.\n2. Scientific validity of experiment design: I find a few choices of the authors a bit odd in the experiment design:\n    - Albedo \"fal\" / \"al\" & Soil temperature \"stl1\" are state variables of the land surface that should be very related to reflectance. In fact, I believe it is not unlikely that remotely sensed brightness and brightness temperature has been assimilated to obtain these variables. This is not too different from VHI, which is created from similare remote sensing products. Thus I would say it should be if anything an Output of your approach. And even if no satellite products have been used to assimilate these variables, this would not solve the issue, but rather raise another issue: then the variables would entirely depend on prescribed schemes in the land surface model of IFS, which means your whole approach is limited by how well IFS reproduces these variables, which I assume is pretty poorly, so any \"anomalies\" you detect in these variables could be considered spurious.\n    -  Soil moisture \"swvl1\" / \"vsw\" is a state variable of the hydrological cycle, and thus should be highly correlated with VHI. However, as far as I know, its representation in ECMWFs land surface reanalysis is relatively poor (e.g. https://ieeexplore.ieee.org/document/9957057)\n    - The anomalies in precipitation that would drive drought are **no precipitation for many weeks**. So if you produce a binary encoding for every single time step, this should not be very predictive. In fact, precipitation is somewhat exponentially distributed: in most regions many days observe 0 precipitation, even under no drought conditions. One way you could potentially circumvent this issue is by using accumulated precipitation over many weeks, e.g. through an exponential moving average. Another way could be to actually implement a simple water balance model. Than again, if you use Soil moisture as inputs, it is essentially coming from such a water balance model...\n   - In other words: I think scientifically most interesting would be if you could connect anomalies in atmospheric variables like temperature, humidity and precipitation to the anomalies in land surface states (VHI). Then you could find the primary drivers for the impacts on vegetation and their time lag, and their spatio-temporal variability, which could be super interesting to study.\n3. Low predictive skill: The performance on the real-world data is pretty bad if I understand correctly (F1 of 0.2 - 0.3, Table 14). In addition the synthetic experiments revealed there seemingly is typically a lower skill on the latent binary variables compared to the outputs, so this makes we wonder if the predicted anomalies for extreme events mean anything at all?\n4. Many baselines for anomaly detection, but none for interpretable forecasting. You compare with a lot of baselines, which is generally great. But all of these perform some sort of anomaly detection on the inputs, which you then assume to be predictive features for your VHI labels. To me what would be a more interesting baseline is one that directly predicts the VHI label from the inputs, and then uses some post-hoc method to try to explain the predictions (e.g. shap, integrated gradients, ...). Because then you are comparing predictions of the drivers directly, and not just general anomalies. \n\nMinor Points:\n1. This work seems to focus a lot on the spatial aspect of things. However, arguably what matters most for drought at a particular pixel is the water balance at that pixel. And that is primarily driven by precipitation and evapotranspiration at that pixel, with only runoff introducing some sort of spatial component.\n2. VHI < 26 may be the result of not just drought. A heatwave could have a similar effect.  Also, VHI is a general vegetation condition index, not just for agricultural areas, but also for natural land cover. You may wish to rephrase your framing of this work studying the \"drivers of impacts of agricultural droughts\" into the \"drivers of impacts of extreme events on vegetation\".\n3. An alternative approach could be to not just predict a binary label, but rather the exact value of VHI. This would be similar to vegetation forecasting (e.g. https://www.sciencedirect.com/science/article/abs/pii/S003442572030256X , https://openaccess.thecvf.com/content/CVPR2024/html/Benson_Multi-modal_Learning_for_Geospatial_Vegetation_Forecasting_CVPR_2024_paper.html , https://gmd.copernicus.org/articles/17/2987/2024/ ). Probably it would be relevant to mention this related stream of literature also in the related works section. Also you may want to consider adding a comment on why you directly predict the (VHI < 26) label instead of the raw VHI, and afterwards apply detection. \n4. Quite some typos, e.g. l.22 \"very\" instead of \"vary\"\n\nQUESTIONS:\nClimate data has extremely high spatio-temporal autocorrelation. How do you ensure your models are not overfitting?\n\nLIMITATIONS:\nThe authors mention a variety of limitations in section 6. However, I believe as is the work has more fundamental flaws that I mention in the Weaknesses section, which, unless fixed, should definitely be named as limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper proposes a novel approach to identifying spatio-temporal anomalies correlated to extremes such as drought. Using neural network, to predict extreme events by learning spatio-temporal binary masks of anomalies identified in climate data. The network is trained end-to-end to predict both anomalies and extremes from physical input variables, focusing on the spatio-temporal relations between them.

STRENGTHS:
Introduces a new method for identifying spatio-temporal anomalies that are correlated with extreme events.

WEAKNESSES:
The model is dependent on temporal resolution, which might not be well documented in all parts of the world.
The method only shows results on droughts.
Binary masks tend to oversimplify real-world events.
The method seems to bluntly connect anomalies with extremes without specific theoretical reasoning.

QUESTIONS:
Have you tried the method on other extreme events other than droughts? And what if there is discontinuity in terms of temporal data?

LIMITATIONS:
If possible, please add the reasoning behind feature represetations and extremes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
This paper proposes an approach to learning the spatio-temporal relationships between events with spatial differences and temporal delays. Specifically, they propose a method that identifies spatial-temporal anomalies in multivariate climate data that are correlated with extremes. The authors conduct experiments on both synthetic data and climate reanalysis data.

STRENGTHS:
1. The problem of anomaly detection and learning their relations is crucial. 
2. The summary of the relevant literature is relatively complete.
3. The authors conduct experiments on both synthetic data and real-world data.

WEAKNESSES:
1. The motivation of model design is not clear. For example, why do you need to detect the anomaly and then classify the extreme events instead of detecting extreme events directly? Such a designed pipeline will lead to more accumulated errors. 
2. The writing can be improved. For example, there are typos, such as 'MIL Is a weakly ...'. And the difference between anomaly and extreme in this paper is not clear. 
3. The title is somehow misleading, actually what the paper does is more about extreme event prediction instead of learning spatial-temporal relations.

QUESTIONS:
1. Anamly detection is a classification problem with severe class imbalance problems, how did the authors tackle this problem? What is the ratio of extreme events to ordinary events?
2. What is the difference between anomaly and extreme in this paper? Could you provide some examples to illustrate this?
3. When I read spatio-temporal relations, I thought this paper would build spatio-temporal graphs to describe the relations. Have you considered using graphs to tackle this problem?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15