RuDar: Weather Radar Dataset for Precipitation Nowcasting with Geographical and Seasonal Variability
Authors: Petr Vytovtov
Eugenia Elistratova
Evgenii Tsymbalov
Alexander Ganshin
Yuri Pavlyukov
Venue: Submitted to ICLR 2023 (2023)
Keywords: precipitation nowcasting
weather forecasting
weather radar
benchmark

Abstract: Precipitation nowcasting, a short-term (up to six hours) rain prediction, is arguably one of the most demanding weather forecasting tasks.
To achieve accurate predictions, a forecasting model should consider miscellaneous meteorological and geographical data sources.
Currently available datasets provide information only about precipitation intensity, vertically integrated liquid (VIL), or maximum reflectivity on the vertical section.
Such single-level or aggregated data lacks description of the reflectivity change in vertical dimension, simplifying or distorting the corresponding models.

To fill this gap, we introduce an additional dimension of the precipitation measurements in the RuDar dataset that incorporates 3D radar echo observations.
Measurements are collected from 30 weather radars located mostly in the European part of Russia, covering multiple climate zones.
Radar product updates every 10 minutes with a 2 km spatial resolution.
The measurements include precipitation intensity (mm/h) at an altitude of 600 m, reflectivity (dBZ) and radial velocity (m/s) at 10 altitude levels from 1 km to 10 km with 1 km step.
We also add the orography information as it affects the intensity and distribution of precipitation.
The dataset includes over 50 000 timestamps over a two-year period from 2019 to 2021, totalling in roughly 100 GB of data.

We evaluate several baselines, including optical flow and neural network models, for precipitation nowcasting on the proposed data. We also evaluate the uncertainty quantification for the ensemble scenario and show that the corresponding estimates do correlate with the ensemble errors on different sections of data.
We believe that RuDar dataset will become a reliable benchmark for precipitation nowcasting models and also will be used in other machine learning tasks, e.g., in data shift studying, anomaly detection, or uncertainty estimation.
Both dataset and code for data processing and model preparation are publicly available.

Under review as a conference paper at ICLR 2023
RUDAR: WEATHER RADAR DATASET FOR PRECIPITA- TION NOWCASTING WITH GEOGRAPHICAL AND SEA- SONAL VARIABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Precipitation nowcasting, short-term (up six hours) rain prediction, guably one of the most demanding weather forecasting tasks. To achieve accurate predictions, a forecasting model should consider miscellaneous meteorological and geographical data sources. Currently available datasets provide information only about precipitation intensity, vertically integrated liquid (VIL), or maximum reflectivity on the vertical section. Such single-level or aggregated data lacks de- scription of the reflectivity change in vertical dimension, simplifying or distorting the corresponding models.
To fill this gap, introduce an additional dimension of the precipitation mea- surements in the RuDar dataset that incorporates 3D radar echo observations. Measurements are collected from 30 weather radars located mostly in the European part of Russia, covering multiple climate zones. Radar product updates every 10 minutes with 2 km spatial resolution. The measurements include precipitation intensity (mm/h) at an altitude of 600 m. reflectivity (dBZ) and radial velocity (m/s) at 10 altitude levels from km to 10 km with I km step. We also add the orog- raphy information as it affects the intensity and distribution of precipitation. The dataset includes over 50 000 timestamps over a two-year period from 2019 to 2021, totalling in roughly 100 GB of data.
We evaluate several baselines, including optical flow and neural network models, for precipitation nowcasting on the proposed data. We also evaluate the uncertainty quantification for the ensemble scenario and show that the corresponding estimates do correlate with the ensemble errors on different sections of data. We believe that RuDar dataset will become a reliable benchmark for precipitation nowcasting models and also will be used in other machine learning tasks, e.g., in data shift studying, anomaly detection, or uncertainty estimation. Both dataset and code for data processing and model preparation publicly available
INTRODUCTION
Precipitation nowcasting is the task of forecasting rainfall situation (precipitation location and strength) for a short period of time, usually up to six hours. Due to climate change the frequency and magnitude of extreme weather events, e.g. sudden downpours, increase, and the techniques for forecasting such events are needed. Precipitation nowcasting can provide information about such events with high spatiotemporal resolution. Such kind of weather forecasting plays essential role in resource planning in the agricultural industry, aviation, sailing, etc. as well as in daily life.
Incorrect precipitation forecasting could have negative impact human life activity, and data with diverse meteorological and geographical characteristics are needed for improving precipitation nowcasting models. The different benchmark dataset usage could improve the quality of precipitation nowcasting models to minimize the risk of forecasting error.
Previously published benchmarks Holleman (2077) ; Shi al. (2017) : Ansari et al. 2018); Ramsauer ct al. (2018) Franch et al. (2000) ; Veileette et al. (020) provide data collected with one or several
URR is hidden for the blind review.

---

Under review as a conference paper at ICLR 2023
An
The geographical area covered by the proposed weather radar dataset (light areas). The covered area has variety of geographic and climatic characteristics. The color indicates the height above sea level.
Figure I:
weather radars. Some of those datasets only contain information about precipitation intensity, others provide vertically integrated liquid value (VIL) or maximum reflectivity on the vertical section.
However, single measurement type is often not enough for extreme weather events forecasting. Thereby, we propose RuDar dataset that contains several measurement products: reflectivity (dBZ) and radial velocity (m/s) on ten altitude levels from km to 10 km with km step and intensity (mm/h) on a 600 m altitude level. Each measurement was carried out with a 2 km spatial resolution and a 10 minute temporal resolution. The dataset from 30 dual-pol Doppler weather radars were collected and processed at the Radar Center of the Central Acrological Observatory (CAO) of the Russian Federal Service for Hydrometeorology and Environmental Monitoring (ROSGIDROMET) and is used by our team within the conditions of commercial contract. For each radar, we additionally provide information about the surrounding orography |Bckker et al.) (2009). The radars are located mostly in the European part of Russia as shown in Figure [ therefore, wide range of geographical and climatic conditions considered. The proposed dataset includes more than 50 000 timestamps over two year period from 2019 to 2021, allowing to investigate the effect of seasonality rainfall forecast.
We illustrate the applicability of our dataset to the nowcasting task by benchmarking the current state-of-the-att optical flow approach Ayzel et al. (2019) and neural network models Shi et al. (2013); Veillette al. (2020); Ravuri et al. (2021) on it. Experiments show seasonal dependence having effect on the algorithm performance due to different precipitation intensity rates and differences between adjacent timestamps in different months.
The main paper contributions are (i) published weather radar dataset with different geographical and climatic conditions (provided under the CC BY NC SA 4.0 license) together with the accompanying exploratory data analysis, (ii) evaluations of common simple precipitation nowcasting models and its extenstion support additional data, (ii) uncertainty estimation and its connection the error for the nowcasting ensemble case, and (iv) accompanying source code for data processing and experiments.
The structure of the paper as follows: Section 2 precipitation nowcasting task, nowcasting benchmarks, Section exploree the uncertainty estimation scenario for the ensemble of models, and Section concludes the paper.
covers previously published datasets for the
the proposed dataset, Section fintroduues evaluated
RELATED WORK
Doppler weather radar the most effective tool for detecting precipitation. The radar measures reflectivity of radio waves from precipitation drops, which can then be converted into precipitation intensity using Z-R relation Marshall Palmer (1948). Standard ways of obtaining single reflectivity measure from the different heights is either taking measurements from only the lower level (base reflectivity), or aggregating these measurements by the maximum value (composite reflectivity). In addition, Doppler radar can detect movement towards away from itself, which allows measuring the speed of precipitation movement along or against the direction of the radar. The latter type of measurement is called radial velocity.
In the public domain, one can find quite a variety of weather radar datasets collected and maintained by international agencies. We summarized the information about some of them in Table T where
2

---

Under review as a conference paper at ICLR 2023
provided comparison by spatial, temporal and pixel resolution, time periods, geographic coverage and number of radars from which measurements were taken.
The first three data sources in Table ! are more large databases than ready-made ML benchmarks:
NEXRAD
Ansari et al. (2018) by US National Oceanic and Atmospherie Service (NOAA), which has been collected since 1994 and contains reflectivity data, radial velocity, and derivative products.
MRMS Zhang et al. (2016) by NOAA National Severe Storms Laboratory combines the previous source together with various sources to provide severe weather, transportation, and precipitation products with 2-minute updates. This dataset is used in a number of nowcasting studies Sønderby et (2020), Klocek al. (202T).
Radarnet Fairman et 2017 by UK Met Office, which has been collected since 1970 and contains composite precipitation data derived from reflectivity data.
The ready-to-ge go datasets, suitable for exploration by ML practitioners and researchers include:
HKO-7 Shi et al.) (2017) by Hong Kong Observatory (HKO) contains single-height
reflectivity level from single radar at the center of Hong Kong for a six years period.
KNMI Overeem & Imhofi (2020) by the Royal Netherlands Meteorological Insti-
tute (KNMI) contains ten-year single-height composite reflectivity data from two radars.
TAASRADI9 Franch al. (2020) by Meteotrentino contains nine-year aggregated reflec- tivity data from a single radar located in the Italian Alps. It is interesting due to geographical specifics and a large amount of extreme phenomena, such snowstorms, hails, downpours.
etc.
SEVIR Veillette et al. 2020) by NOAA and the Geostationary Environmental Satellite System (GOES) contains US radar and satellite data for two year-period, sampled either randomly or on an event basis. It is focused on the detection of storm events.
RADOLAN Ramsauer et al. (2018) by the German Weather Service contains three-year reflectivity and precipitation data collected by 18 radars in Germany.
Our dataset was collected by 30 radars of the Central Aerological Observatory (CAO) and contains
two years of observations (spring 2019 - spring 2021). Measurements were carried out mainly European part of Russia, but some areas of the Siberian and Far Eastern regions were also captured.
The main advantages of our dataset relative to the abovementioned are as follows:
1. A large area is covered with different climatic conditions from the extreme north (Arkhangelsk, 64.62" N 40.51 E' ) to the southern regions (Krasnodar, 45.04 N 39.15 E).
2. Dataset introduces continuous measurements account the influence of seasonal dependence on strength and distribution of precipitation.
year period, which allows taking into
3. The data presents several radar products at once: reflectivity and radial velocity at 10 altitude levels (1 10 km) and precipitation (600 m). We do not specifically aggregate the reflectivity and radial velocity data, as we believe that it can be useful for forecasting to know how moisture and its velocity are distributed depending on the height. There are situations when precipitation has not yet appeared at low levels, but at the same time, at high levels, there is information about future precipitation (see an example in Figure G Supp.Mat.).
4. In addition the radar data itself, provide information about the orography in the area around each radar. These data allow one to investigate the influence of the surrounding orography in predicting precipitation.
3 DATASET DESCRIPTION AND PROCESSING
The proposed RuDar dataset contains measurements from 30 weather radars mainly located in the European part of Russia (Figure T. Radars scan the area around with radius of 250 kilometers.
3

---

Under review as a conference paper at ICLR 2023
Table 1: Comparative table of various international open sources of radar data. The data were compared by spatio-temporal (km per pixel/ / minutes) and pixel resolution, as well as by time periods, geographical coverage and the number of radars involved. The main products contained in the datasets are base and composite reflectivity. as well as derived products: precipitation, maximum reflectivity and vertically integrated liquid (VIL). If there is more than one radar, all except NEXRAD and RuDar combine measurements from different radars into one frame.
(+) For the 2020.03-2021.1
period, we publish every fifth day to be used for validation and testing. ($*) Feature codes: (1) precipitation rate, (2) radial velocity, (3) reflectivity by altitude layers, (4) integrated or one layer reflectivity, (5) satellites
Dutuset
Time
numporltemporal resolution
Pixel resolution
Geugraphy
No. radars
Feutures
Reudy
periods
per frame
use
2015
min
400x460
States
3,2,3,
MRMSS Zrang of of
2016
2014-
km min
1500
United States
TAI
1,3,4
Rularnet
1970-
km 5 min
1536x1280
United Kingskom
15
1.3
HKO-7 (Sni
2009
1.0
480x450
Kong
Kveeee & Imsoll 2020
2008 2018
1 km min
100x100
Notactlands
4
+
IAASRADI Franch
2010-2019
0.5 km min
48004800
lialian Alps
4
SEVIR Vellialic di 2020
2017-2019
km min
384x3W4
United States
160
RADOLAN Kaamsnee al. 2018
2011,12-2017.1
1 km/ min 2m /
500+500
Germany
18
1,4
Kubbar (Otrs)
2019.01 2022
25252552
Kussia
30
1,2.
Measurements come from radars every 10 minutes and have a spatial resolution of 2 x 2 kilometers. The dataset covers two years from 2019 to 2021 and has over 50 000 unique timestamps.
Each data sample is a furee-dimensione tensor that contains the result of a 10 minute scan of the atmosphere with a single radar. The center of the frame corresponds to the location of the radar in the scanned area. Tensors have 21 channels with spatial resolution of 252 x 252 pixels. The first channel contains information about precipitation intensity (mm/h) on 600 m altitude level. The channels 2-11 contain reflectivity measurements (dBZ) performed at 10 altitude levels from km to 10 km with 1 km step, and the channels 12-21 contain radial velocity (m/s) measurements from the same 1-10 km altitudes. We also provide data orography [eekker et al. (209) and latitude-longitude coordinates of the territory surrounding the area of the radar measurements. Coordinates were initially set kilometer grid with step of 2 kilometers with zero at the radar location.
The data example is partially shown in Figure 2a. Please refer to the Section in Supplementary for the comprehensive documentation
3.1 PRECIPITATION INTENSITY
Precipitation intensity is measured at an altitude of 600 meters in mm/h and is represented by single channel in the data. It not a direct measurement: the precipitation rate is calculated from reflectivity values with Marshal-Palmet type of Z-R relationship Marshall & Palmer 1948). In addition to mm/h values, two special values are presented in data: e value marks areas where measurements are not available, 6 value marks areas where no precipitation events were detected. Both values can be reduced to mm/h, however, when feeding radar frame to the model, it is better to mask 6 values so that the model can distinguish the absence of precipitation from the blind zone of the radar.
Figure 2 shows the distribution of precipitation by seasons. The data for each season is taken from the entire period 2019-2021. As can be seen in Figure 21, the intensity has a seasonal dependence: the highest values of precipitation amount are reached in summer and the lowest in winter.
We investigated the distribution of precipitation intensity difference between two adjacent radar im- ages. As Figure Ze shows, in the winter season the precipitation rate difference between two adjacent radar images is much lower than in the summer season. This means that in winter precipitation changes much less in the short-term interval than in summer, and therefore it is more difficult to solve the problem of nowcasting in warmer seasons.
Peaks which are visible in Figure 2 are the result of specificity of the combination of radar-side filtration algorithms and Marshall- Palmer type of Z-R relationship.
Data- Data-featurin Colab Notebook is availahle at URI_ wt__hiddee_duu_tt_bll rev_reeeew.
4

---

Under review as a conference paper at ICLR 2023
(a)
(c)
Mineralnye Vody
6000
252
1000
(10) radial velocity
(10) radar reflectivity
(3) precpitation intensity
252
200
120
(d)
10
Mider
10
bsent beettcmm count
10"
sutum
summer
10a
summee
coure
10%
20
80
-40
-30
-20
10
20
40
Interstyy mmch
Precipitation intersity difference on consecubive timestemps. mimthe lbinerizzed (ieee)
Figure 2: Featuring RuDar dataset. (a) The sky above the Moscow Region, Vnukovo Radar; axes ticks are given in kilometers. Ten levels of the precipitation intensity are shown (for each height level from 1 km to km), together with the orography (magnified for visualization purposes) on the bottom. The lined cuts the left part of the plot demonstrate specific case of urban obstruction in the radar vision: skyscrapers are not shown in the orography map yet they affect the scan result. (b) Tensor representation for single radar the given timestamp: each tensor has shape of (252, 252, 21). Note that orography information is stored separately to avoid duplication. (c) Natural obstructions near Mineralnye Vody city: the mountain chain on the south blocks the radar, reducing the receptive field down to the area denoted by the white dotted line. (d) Seasonal dependence of the precipitation intensity distribution (mm/h per pixel). During the winter season events with high precipitation rates are much less represented than in the summer season. We clipped high (e) The distribution precipitation intensity difference between two adjacent radar images (mm/h per pixel) in RuDar dataset. In the winter season, precipitation events change slower than in the summer season. The periodical peaks are the result a peculiarity of the combination of rider-side filtration algorithms and Sarshall-Palmes type of Z-R relationship. Best viewed in color.
precipitation rates (> 140 mm/h).
3.2 RADAR REFLECTIVITY
Reflectivity values represent direct radar measurements in dBZ. units. The radar measures the amount of energy reflected from droplets distributed in the atmosphere, and this amount energy turns out to be proportional to the amount of moisture in the air. All reflectivity measurements were carried out at heights of 1-10 km with km step. We provide reflectivity data at several altitude levels, since we believe that not only information about precipitation changes in time, but also in space, is important for nowcasting, see Fig. 6 in Supp. Mat. for an example.
To demonstrate the geospatial diversity, Arkhangelsk (in the north) and Krasnodar (in the south) in Figure 3 It can be seen that both seasonal and altirudinal dependencies are different for these radars, which shows the wide range of weather conditions and data variability.
compare the mean reflectivity for two radars:
3.3 RADIAL VELOCITY
The radial velocity the wind velocity projected onto the ray starting in the radar. Negative values correspond to the movement towards the radar, and positive values correspond to movement away from the radar, Due to the peculiarities this type of measurement. the values are available only at points where the radar has registered moisture droplets. The data is measured in m/s and provided
5

---

Under review as a conference paper at ICLR 2023
Krasrodar
Arhhanglls
Krasnodar
Aikrangelsk
-20
20
-20
20
Neon roflectivky
Rean roflcciiiy
Roon radio wo bg
Mean rodiol octyy
Figure 3: Comparison two different regions: Krasnodar (on the south) and Arkhangelsk (on the north) in terms of mean reflectivity (left) and mean radial velocity (right) different height levels. Only non-zero values are taken into account. Regions demonstrate different patterns in the altitude intensity and radial velocity distribution as well the seasonal changes.
for ten altitude levels from 1 km 10 km with km step similar to reflectivity data. According to Figure 3 radial velocity values are seasonally dependent.
3.4 GEOGRAPHICAL INFORMATION
In addition to atmosphere scans, the RuDar dataset contains orography and geographical coordinates associated with each radar. Geographical coordinates (latitudes and longitudes) are provided for each point where measurements are presented. Orography is two-dimensional tensor which positive values correspond to elevations above sea level and negative values correspond to elevations below sea level. We resampled an available information Becker et al. (2099) to the used coordinate grid. An example of orography is shown in Figure 2 where the demonstrated. The radar's 's visibility area in the figure is greatly reduced by high mountains located in the immediate vicinity of Mineralnye Vody. Note that the orography measurements do not include the urban areas, and Figure 2h shows that skyscrapers in Moscow may also pose a problem.
of natural radar vision obstruction is
4 NOWCASTING BASELINES
4.1 PROBLEM STATEMENT
The nowcasting problem can be formulated in terms of sequence prediction task (precisely video prediction task) the goal is to predict sequence of future observations from input sequence of historical measurements. More formally, let us have an input sequence of measurements X = (X1-M+1,-... X2) of length M for time step t, and sequence of future measurements Y (Y44h..., Y++K) of length K. Then our task is to construct a model f that predicts Y by X:
Y4+1, Y.42. ......... YO-K =
Each element X; is tensor of shape of one frame, and Cimput is the number of measurements used for prediction. An element Y; of the output sequence is tensor of shape x x Coutput. where Coutput is the number predicted measurements. The numbers Cinput and Coutput may differ, since in the input sequence can possibly use all the available measurements (intensity, reflectivity, radial velocity, orography), and the output sequence will contain only the target values. In our case, the target is future precipitation intensity, which implies Coutput =
W Cinput, where H and W are the height and width
The quality of the model can be measured by how well it predicts the strength of precipitation directly, as well as how well it copes with detecting precipitation. In the first case, we use pixelwise mean squared error (MSE) measure quality, in the second F, Fcoce,,, details of calculation in

---

Under review as a conference paper at ICLR 2023
Section I of Supplementary Material. In meteorological articles articlesEspehoot et al. (2021): Ravuri et al. (202I); Bouget et al. (2021), metrics such as Probability of Detection (POD), Success Ratio (SUCR), Critical Success Index (CSI) sometimes used, but they actually the equivalents of recall, precision and intersection over union (IOU), standard metrics in machine learning. We chose Fi because it correlates with IOU and depends on both precision and recall.
4.2
EXPERIMENTAL SETUP
4.2.1
DATA SETUP
We are focusing on a year scale setup, which uses 2019 data for training, and every fifth day of 2020-2021 data for validation and testing.
In our experiments, we use input tensors with historical observations to predict 12 future intensity frames (M 4, K 12. Coutput 1). The spatial size of the input and output tensors is H W = 252. We clip precipitation intensity values to 50 mm/h, since high intensity rates are rare in the given geographical area. As the special value -2e6 6 to prevent penalizing our models for errors outside the radar visibility range.
MSE loss to train our models,
binary mask for
When only intensity is used an input observation, Cinput 1. Using reflectivity levels increases Cinput to 11. All values are clipped in the interval between 32 dBZ and 65 dBZ, and then reduced to the interval from 0 to 1 by subtracting -32 dBZ and dividing by 65 (-32) 87 dBZ. The intensity values for this setup are divided by 50 mm/h (for both input and target frames) to be also in the unit range (0, 1). Radial velocity also adds 10 input channels (Canput 11). All radial velocity values are clipped between -63 m/s and 63 m/s and then divided by 63 m/s to be in the interval (-1,1). Orography adds one additional channel (Cinput 2). Negative values of heights are reduced to zero, and positive values are divided by the maximum height equal to 5336 m.
For both radial velocity and orography, the intensity preprocessing is the same as in the absence of additional features.
4.2.2 MODELS AND SETUPS
We represented f(-) with a persistent model as weak baseline and with state-of-the-a optical flow approach /Ayzel et al. (2019) as strong baseline. Also we trained UNet-like Veillette al.) (2020) and CSnvI.TTM- M-TM-MMiike like Shi et al. (2015) neural network models (including Extended ConvLSTM architecture shown 4)g and tested a pretrained GAN-like model Ravuri et al. (202T). Each of the above models (excepting the pretrained GAN model) was trained on single Tesla A100 80GB for approximately 70 hours. Details about model architectures are provided in Section E of Supplementary Materials.
Also we evaluated Earthformer et al. (2022) on the proposed dataset using official implementation. Other strong neural network baselines Sonderby et al. (2020); Klocek et al. (2021): Espeholt et al. (202I) will be considered in future research as they require additional input data that is not presented the proposed dataset. Satellite information is used in (onnerby et al. (2020), numerical weather
forecast used in Klocek et al. (202T), and Espeholt et al. (2021) requires both.
4.3 RESULTS
We provide the results on F, and MSE for each of the above models, as well as the results of Extended ConvLSTM using each of the additional features. We evaluate the baselines on seasonal test splits of the RuDar dataset. The results are shown in Table
All basclines work better than Persistent, and among basclines without additional features, Extended ConvLSTM shows the best result in all seasons except summer: there is inferior to Optical Flow in Fi.
Adding reflectivity shows the best result for all metrics and seasons, except for F1 in summer (where it is inferior to Optical Flow) and in autumn (where it is inferior to Extended ConvLSTM with orography). Adding radial speed improves both MSE and F, and adding orography improves F, relative to Extended ConvL without additional features.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Metries of baseline models calculated separately for each season. The first value in the cell is MSE. the second value is Fj. Metries are average across prediction horizons. Red highlights the maximum in the entire column, blue highlights the maximum among models without additional features.
Spring 2020
Summer 2020
Autumn 2020
Winter 2020-2021
Persistent
1,269170.5177
0.628100.9972
0.0579/0.522
0.0260 / 0.6425
Optical Flow
0.1936 0.6297
0.4768 7.66435
0.0384 0..651
0.0190 0.6884
GAN
0.18857 0.6368
0.662070.6100
0.0352 0.6500
0.0218 / 0.6580
U-Net
0.142070.5619
0.3215
0.031270.5730
0.01607 0.6232
ConVLSTTM
0.1217 / 0.6222
0.3244
0.5580
0.0242 0.6650
0.0156 0.6799
Extended ConvLSTM
0.6436
0.3193 0.5877
0.0233 / 0.6814
0.0150 0.6947
Extended ConvLSTM+ + reflectivity
0.1133/ / 0.6597
0.3080 / 0.6170
0.0219 / 0. 0.69
0.0140 7.,110/0.7014 70744
Extended ConvLSTMV + radial velocity
0.1770/065116
0.3153/ 0.5970
0.0228 / 0.6885
0.0144/ 7038
Extended ConvLSTM + orography
0..19370.6526
0.3197
7,5968
0.023470.6870
0.0148/ 7076
Earthformer
0.0124/NA
0.0153/NA
0.0047 NA
0.0048 / NA
0.3
stee:
Yrur
0.2
0.1
0.4
50
100
50
100
Horizon, minuless
Horizon, minutes
0.6
DesuLSTMM Ee evelec saceradde
0.125
Cervoote
DenvLSMM
OOWLSTN
N55
0.100
Graaderr
0.6
0.015
0.050
Hortzon, minutes
Horizon, minutes
X1
Figure 4: (Left) Algorithm performance (left column: F1. right column: MSE) as a function of a prediction horizon on the spring season of the test set. Top row: baseline algorithms, bottom row: Extended ConvLSTM with the various additional data. While optical flow may outperform other models on the initial time steps, NN- and trained baselines easily outperform baselines. Usage of the radar reflectivity improves the performance of the ConvLSTM model. (Right) An extension of the architecture from Shi et (2015). The first ConvLSTM layer receives the output of convolutional encoder as input, and the output of the last ConvLSTM layer is upsampled to the final prediction with a convolutional decoder. To output the next frame, the predictor uses the outputs of the last ConvLSTM layer from the previous step.
We also show the relationship between algorithm performance and forecast horizon for the spring season in Figure 4a.
UNCERTAINTY ESTIMATION FOR THE ENSEMBLE CASE
Weather forecasting task, given its properties such as noisy data, probabilistic predictions, and general weather variability, may organically benefit from the uncertainty quantification. Uncertainty estimates (UEs) are helpful in a number of cases, including "failing to predict" (when the model understands that its prediction is not reliable enough, and gets the results from the basic robust model), active learning, data shift reaction, as well as the error analysis and model selection.
We focus on the last scenario and analyze the relationship between the error and UE for the ensemble case. Given an ensemble of five ConvLSTM models, trained on the same data but different seeds, we treat the mean and standard deviation of predictions as the output and UE, correspondingly. An ensemble error and UE on the test set are correlated with a large Spearman coefficient (> 0.8); this relationship holds even when both metrics averaged over the various dimensions: time, location, and forecast horizon, see Fig. 7 in Section H of Supp.Mat. We also report that we did not find a re relationship between the UE and orography (Pearson |r| < 0.1).
8

---

Under review as a conference paper at ICLR 2023
This analysis suggests that UE metrics can be used to indicate large errors, which, in turn, can be used to deliver better predictions or uncertainty the forecast, as well as assist the model selection and improvement.
DISCUSSION
6.1 SUMMARY
In this paper, we propose a weather radar dataset with wide variety of geographical and climatic conditions and show in contrast to previously published works that a precipitation nowcasting task can be a seasonal dependent problem. This encourages the usage of the seasonal models and separate models for precipitation rate forecasting and precipitation events existence.
We also evaluated ensemble of nowcasting models, analyzing the uncertainty estimates and showing its deep relation to the ensemble error on various scales.
We believe that our proposed dataset will become reliable benchmark for precipitation nowcasting models. The code for data preprocessing and model preparation is publicly available This dataset is provided under the CC BY NC SA 4.0 license. The data is maintained on the cloud service, and is maintained by authors team
6.2 LIMITATIONS
We would like to point out a few limitations of out work:
Dataset size. We included the two-year period only order for the dataset to be compre- hensible and processible by the community, as its unpacked size already exceeds 100 GB. This dataset not intended for study of long-term climatic changes.
Data shift. Like any other piece of real-worke data, our dataset may contain several pecu- liarities, In the text, we discuss the seasonal dependence and geographical shifts, however, it should be noted that the measurements within the sample are also simultaneous as they were obtained by combining the measurements from the narrow beam that scans the atmosphere.
Baselines. We provide the readers with simple baselines supporting the dataset. However, we plan to benchmark number of state-of-the-ard neural network-based baselines Sonderby et al. (202)); Espeholt et al. (202I); Klocek et al. (202I), in future studies.
6.3 POSSIBLE PROSPECTS
two-year dataset may be applicable not for the nowcasting task only but number of contemporary ML problems, including:
Various storms and rare weather conditions are of special interest both the researchers and end-users. Some works even emphasize this area of research Veillette et al. (2020). We expect various anomaly detection algorithms to of great use here.
Rare event detection.
Data shift and uncertainty estimation. The variety of both geographical and temporal conditions makes this dataset a good candidate for the modeling of distribution shift scenarios (see, e.g. Malinin et al. (202T), where test data may naturally vary from the training data dramatically. In this case, various uncertainty estimation approaches, like in Grönquist et al. (2019) may be helpful.
Active learning. One of the most important scenarios for the day-to-day forecasting systems is active learning since continuous data flow allows for the production model to learn from its own mistakes, correcting previous predictions. However, processing and retraining huge amounts of data poses challenge, and one may use smarter ways of data sampling (of both past archives and daily chunks of data) in order to reduce the data processing and model training times.
'The link to full dataset will available after paper acceptance, a sample is available at URI wiw.hidee dden due_rt Que_to_bl inn_rviieee_eeeeeee ew.
*Please contact corresponding authors if you have any questions or additions regarding the dataset.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Steve Ansari, Stephen Del Greco, Edward Kearns, Otis Brown, Scott Wilkins, Mohan Rama- murthy, Jeff Weber, Ryan May, Jed Sundwall, Jeff Layton, Ariel Gold, Adam Pasch, and Valliappa Lakshmanan. Unlocking the Potential of NEXRAD Data through NOAA's Big Data Partnership. Bulletin of the American Meteorological Sociery, 99(1):189 204, 2018. doi: 10.1 N0.177/BAMMSD-1160000 Journa 1 s /a/am//////// 99/1 /bbmms- -eeaams--111 16-166-00000 0021. xm
URL httpss///jorraat ametsoc.
G. Ayzel, M. Heistermann, and T. Winterrath. Optical flow models as an open benchmark for radar-hased precipitation noweasting (rainymotion v0.1). Geoscientific Model Development, 12 (4):1387-1142, 2019, doi: 1w.1199/mm--1--137-222 12-1387-2019, URL httttts::// /gmd//mdd..ooo/// copernicus
/0019/
Becker, D. T. Sandwell, W. H. F. Smith, J. Braud, B. Binder, J. Depner, D. Fabre, Factor, S. Ingalls, S-H. Kim, R. Ladner, K. Marks, S. Nelson, A. Pharaoh, R. Trimmer, J. Von Rosenberg, G. Wallace, and P. Weatherall. Global bathymetry and elevation data 30 are seconds resolution: SRTM30 Marine Geodesy, 32(4))355 2009. doi: 10.108000149041000022777777 URL /11 01100001404100000222
httpss//doi .org/orrg00.
Vincent Bouget, Dominique Béréziat, Julien Brajard, Anastase Charantonis, and Arthur Filoche. Fusion of rain radar images and wind forecasts in a deep learning model applied to rain nowcasting.
Remote Sensing, 13(2), 2021. ISSN 2072-4292. doi: 1.3390/s130202444 URL https
7 //www.mdpp com wwtpc///2772--99/1///222222
Lasse Espeholt, Shreya Agrawal, Casper Sønderby. Manoj Kumar, Jonathan Heek, Carla Bromberg, Cenk Gazen, Jason Hickey, Aaron Bell, and Nal Kalchbrenner. Skillful Twelve Hour Precipitation Forecasts using Large Context Neural Networks. arXiv preprint wrXiv:2111.07444 2021.
Jonathan G Fairman, David M Schultz. Daniel J Kirshbaum, Suzanne L Gray, and Andrew I Barrett. Climatology of Size, Shape, and Intensity of Precipitation Features over Great Britain and Ireland. Journal of Hydrometeorology, 18(6):195-1615, 2017. doi: 10.
1175/HHM-D 16-0222.1. URL /attps::/jouuneee ael.ss..mestoooccc wtt.://...w.///////.ooo/ooo/o/// www.tssc..orgrteee///oo....... /tdr/118//jin d 16 0222
xm_
Gabriele Franch, Valerio Maggio, Luca Coviello, Marta Pendesini, Giuseppe Jurman, and Cesare Furlanello. TAASRAD19, high-resolution weather radar reflectivity dataset for precipitation nowcasting, Scientific Data, 7(1):1 13,1111,,2200 2020.
Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Bernie Wang, Mu Li, and Dit- Yan Yeung. Farthformer: Exploring space-time transformers for earth system forecasting. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho Advances in Neural Information Processing Systems, 2022. URL/tttss://ppeeeviiiee net.://wwweeeeeeeeeeeeeeeeeeeeee http://eeee/ooooot/oeeeem
Peter Grönquist, Tal Ben-Nun, Nikoli Dryden, Peter Dueben, Luca Lavarini, Shigang Li, and Torsten Hoefler. Predicting weather uncertainty with deep convnets. arXiv preprint 2019.
Iwan Holleman. Bias adjustment and long-term verification of based precipitation estimates. Meteorological Applications, 142):195-2033. 2007. doi: http::///ooiorr11.1000000000 ntttps://mmt //tst/////mt...m/// on www.llnnwbrrrr.leee........ www.lll..ccco..oooooomoo.oo http:/doo//ab///ee 1002/me00mmmmmme
URL
Sylwester Klocek, Haiyu Dong, Matthew Dixon, Panashe Kanengoni, Najeeb Kazmi, Pete Luferenko, Zhongjian Lv, Shikhar Sharma, Jonathan Weyn, and Siqi Xiang. MS-nowcasting: Operational Precipitation Nowcasting with Convolutional LSTMs at Microsoft Weather. arXiv preprint arXiv:2111.09954, 2021.
Andrey Malinin, Neil Band, German Chesnokov, Yarin Gal, Mark JF Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, et al. Shifts: A dataset of real distributional shift across multiple large-scale tasks. arXiv preprint arXiv:2107.0744 2021.
10

---

Under review as a conference paper at ICLR 2023
J. S. Marshall and W. Mc K. Palmer. The distribution of raindrops with size. Journal of Armo- spheric Sciences, 5(4):165 166, 1948. doi: 10.17551220000000094950000000000 2.0.CO:2. URL https:/ journals www.::....o.cco.cc......... 1520-04694 94 8 0000000500000 01 w5__ddoww__aaee 2_________________1___
Aart Overeem and Ruben Imhoff. Archived 5-min rainfall accumulations from radar dataset for the Netherlands, Mar 2020. URL httts://daat. 4tu wtw.n//aariiieee//taateaaa Archived 5-mim min rainfall accumulations from a radar dataset for
the__ Nether /
Thomas Ramsauer, Thomas WeiB, and Philip Marzahn. Comparison of the GPM IMERG final precipitation product RADOLAN weather radar data over the topographically and climatically diverse germany, Remote Sensing, 10(12). 2018. ISSN 2072-4292. doi: 10.3390/rs10122000
URL https::///// //t:://wwwwww http://wwwwppp... 2ttp:m/222222222//112222222
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation 2021.
nowcasting using deep generative models of radar. Nature, 597(7878)
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image compuling and computer- assisted intervention, pp. 234-241. Springer. 2015.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, pp. 802-810, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit- Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model. In Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL
wwwp:iiinnnnnneunnnoeeee https//eeaaaap////EEELLLL Paper pdf
Mennatullah Siam, Sepehr Valipour, Martin Jagersand, and Nilanjan Ray. Convolutional gated recurrent networks for video segmentation. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 3090-3094, 2017. doi: 10. 1..110//CI2221188299655111
Casper Kaae Sønderby, Lasse Espeholt, Jonathan Heek, Mostafa Dehghani, Avital Oliver, Tim Salimans, Shreya Agrawal, Jason Hickey, and Nal Kalchbrenner. Metnet: A neural weather model for precipitation forecasting. urXiv preprint arXiv:2003. / /2/40, 2020.
Mark Veillette, Siddharth Samsi, and Chris Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite meteorology. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 220099222019 Curran Associates, Inc., 2020. URL, httphhttt://///rooo
neur ips ps
Yunbo Wang, Zhifeng Gao, Mingsheng Long, Jianmin Wang, and Philip S Yu. PredRNN++: Towards resolution of the deep-in-time dilemma spatiotemporal predictive learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2123-5132, PMLR, 10-15 Jul 2018. mwr.g.smmmr... mlr www.w....p....ee......e.e /v0 http/////wwwgggg/ wwngl8n...... html
URL httpt:ss/ppoooeedingggg
Jian Zhang, Kenneth Howard, Carrie Langston, Brian Kaney, Youcun Qi, Lin Tang, Heather Grams, Yadong Wang, Stephen Cocks, Steven Martinaitis, et al. Multi-Radar Multi-Sensor (MRMS) quantitative precipitation estimation: Initial operating capabilities. Bulletin of the American Meteorological Society, 97(4):621-638 2016.
11

---

Under review as a conference paper at ICLR 2023
7 SUPPLEMENTARY MATERIALS
A SUMMARY AND DISCUSSION OF PREVIOUS REVIEWS
We would like to thank anonymous reviewers for their valuable comments and suggestions. The main difference between the previous submission and the current one is that we shifted our focus from the ML models to the dataset, and thus provide basic and simple baselines to follow in future use. The main points raised by reviewers are listed below together with our comments:
There is no major difference between the proposed dataset and previous datasets like HKO-7 and SEVIR. The outstanding difference a large number of channels needs to be justified and is straightforward to incorporate into the existing ConyLSTM-lik models.
We emphasized the difference between our datasets and several others, including HKO-7 and SEVIR. We discuss the benefits of the additional channels the main text.
The importance of this specific type of data for the MI community, not only meteorologists. We added a subsection in the summary section that discusses this issue. We argue that this dataset may be of great interest to ML researchers for anomaly detection and domain shift tasks, as well as active learning and uncertainty estimation.
Choices data processing and data split should be explained in detail, as well the choice of the baselines.
We refined and reasoned the data processing and split scheme and updated the text. As we shifted the focus of the paper the dataset, we now provide the most straightforward and ready-to-g baselines for comparison and future research.
Paper lacks critical reflection on the limitations of the proposed dataset.
We added discussion of the limitations to the last section of the main text.
B THE CHOICE OF THE BINARIZATION THRESHOLD
Initially, it was chosen upon our internal experiments within the company, which involved user's feedback for a weather forecasting service, and is close to the boundary of the rain versus moisture discrimination, given the following reasoning. Table shows the number of events in the first two bins from the seasonal precipitation distribution shown in Figure 2c of the main text. One event here is one pixel in radar frame with intensity measurements, and season data is combined for all occurrences of given season between 2019 and 2021. For each season, the first two columns contain the number of events when there no precipitation at all and when the precipitation strength is in the range from 0 to 0.1 mm/hr. The last column shows the proportion of no precipitation events (the sum of the first two columns) relative all events in the season. For summer and autumn, the threshold of 0.1 mm/hr turns out to be the 95- -e-rercee of the distribution. Since the frame- frame-by-frame by-frame variability much higher in summer than in winter (see Figure 2d in the main text), the summer season is more challenging for the nowcasting problem. Therefore, when choosing threshold for binarization to obtain F1 results, we focused on the summer season.
Scason Winter
No- No-pricipitation events
Precipitation events in range (0, 0.1]
Total events
Ratio
72,898,929, 174,324.614,608
2,963,366,689
81,044,347,514
0.936
Spring Summer
3,781,612.615
191,773,945,
0.929
175,629.693.08
1,852,301,657
186,500,102,531
0.952
Autumn
170,587,211,782
4,187,861,066
9356,65
0.953
Table 3: Seasonal comparison of the number of events with the absence of precipitation or with the strength of precipitation from the interval (0, 0.1] mm/hr, and their cumulative proportion among all events in the season.
C RAINYMOTION LAUNCH CODE LISTING
We have added a Rainymotion Ayzel et al. (2019) launch code, which basically demonstrates that we the default setting in experiments.
12

---

Under review as a conference paper at ICLR 2023
radar_...
timestamp
radar-data
hdfs file
latitude
radar_...
gea-data
longitude
topography
Figure 5: The scheme of an hdf5-dataset with RuDar data. Upper level - Shdf--group with radar data and geo-data. Geo-data is arranged in the geo- a group by radar and contains the topography of the radar and the latitude- lonnudde-onggiude coordinates of its territory. Radar data within the radar ddata group is decomposed into timestamps the form of UTC-timestanps. Data for each radar has shape (252, 252, 21), where the first channel of the last dimension corresponds to intensity, 1-1 to reflectivity, 12-21 to radial velocity.
workaround OpenCV 4. import cv2 cv2. optflow. createOptFlow DIS cv2. cw..DisOpticalFlow.ccr import numpy as np
from rainymotion. models import Dense
assuming x batch contains the necessary tensor model = ()
model. input_
x.batch SEQ x H x W
model. input-date [np. where model. input-data 0)] = 0 pred = model. run ()
D DATASET STRUCTURE
The dataset is represented with of hdf5 files one hdf5 file per year. We also share subsets of RuDar dataset used in data analysis and experiments.
Each hdf5 file consists of two groups: radar-data and geo-data. radar-data group provides measurements (precipitation intensity, reflectivity, and radial velocity) indexed by pairs (timestamp, radar). geo-data group contains the topography Becker et al. (2009) of the radar and the latitude-longitude coordinates of its territory
Figure shows the schema of an hdf5-file.
radar-data group contains subgroups where each subgroup named after a certain timestamp in UTC time-zone. Timestamps have 10 minute temporal resolution. Each timestamp subgroup is divided into named hd 5-datasets with measurements from weather radars that were currently available (sometimes radars are turned off due problems). Each radar dataset tensor that contains the result of ten-minute scan of the atmosphere with certain radar. The measurements include precipitation intensity (mm/hr) on the 600 altitude (the 1st channel in dataset), reflectivity (dBZ) (channels 2-1 and radial velocity (m/s) (channels 12-21) on 10 altitude levels from km to 10 km with km step and 2 km spatial resolution.
diree-dimensional
geo-data group contains subgroups where each subgroup corresponds to certain weather radar. Each radar subgroup consists three two-dimensional datasets with latitudes, longitudes, and topography Becker et al. (2009) of the radar territory.
"The link will be available after review
13

---

Under review as a conference paper at ICLR 2023
E MODEL INFORMATION
Persistent. In the persistent model, we consider the latest radar image from an input sequence as a forecast for all images of an output sequence, so it is simply a constant prediction.
Optical Flow. We take a state-of-the-art optical flow approach from Rainymotion library Ayzel al. (2019). and use Dense Inverse Search model with constant-vector advection scheme. The advantage of this particular optical flow approach over the others was shown in previously published works (Sonderby al. (2020) and our experiments. See code listing for usage in Section of Supplementary.
GAN-like. We used a pretrained GAN-based model from Ravuri et al. 2021 as model for comparison, and we did not additionally fine-tune or train this model on our data. The model, as is GAN-like, consists of a generator and discriminator parts. The generator contains two parts: encoder and decoder. An encoder is colly-convolutional neural network that separately processes each input frame and concatenates the resulting representations. A decoder is a recurrent neural network with ConvGRU cells Siam et al. (2017) for predicting future frames. The model contains two discriminators: one for temporal consistency between predicted frames and another for spatial consistency inside certain frame.
UNet. We use the same architecture as proposed in Veillette et al. 2020). It is based on the original UNet architecture |Ronneberger et al. (2015) which takes . input frames concatenated along the channel axis and predicts K output frames also concatenated along the channel axis. The model contains four downsampling and four upsampling convolutional blocks (32, 64, 128, 256) and (256, 128, 64, 32) filters respectively. The last layer is convolution with one filter for producing the requested number of output channels. We trained the model for 10 epochs with L2-loss and Adam optimization algorithm with a learning rate equal to 4e 3 and batch size 64. For evaluation, we took a checkpoint from the best epoch according to the validation loss.
ConYLSTM. We used the ConvLSTM and an encoder- encoder-predictor architecture proposed in Shi et al. (2013), but with few changes. Firstly, we do not concatenate predictions from all layers of the predictor but use only the last layer. Secondly, we use the Adam optimizer instead of RMSProp. Third, we use L2-loss instead of cross-entropy. And finally, we use teacher-forcing with probability decreasing with the iteration number, similar to what authors of Wang al. (2018) did in their implementation.
For uncertainty estimation, we trained ensemble of five models with (128, 64, 64) filters for each layer, respectively. We trained these models with batch size 64 and learning rate 4e - for one epoch and then selected the best checkpoint for each according to the values of the validation loss.
Extended ConVLSTM. To analyze the impact of additional features, we expanded the original architecture with convolutional blocks at the input and output of ConvLSTM layers (see Figure Ap) We trained five models to provide an ablation study: a model without expansion by convolutional blocks, extended model that accepts only intensity as input, and three more models, each of which accepts intensity and one of the additional features as input: reflectivity, radial velocity and orography. Each model consists of three ConvLSTM layers with (64, 64, 64) filters for each layer, respectively. We trained models for one epoch with the choice of the best checkpoint based on the values of the validation loss. We used batch size 16 and learning rate
for these experiments.
Earthformer. We use transformer-Ie like architecture called Earthformer Gao et al. (2022). The idea of the model dividing the input tensors to cuboids and calculating lelf-attention inside them. After that cuboids are merged also with self-attention mechanism using additional global vectors, We have used an official implementation with hyperparameters from the original paper.
F FIGURES: FEATURING DATASET PECULARITIES
14

---

Under review as a conference paper at ICLR 2023
9:271
Figure 6: An example of sudden precipitation occurrence. The image shows radar measurements of precipitation rate (mm/h) at ground level (up to 600 meters, top row) and reflectivity (dBZ converted mm/h by Marshall-Palmer relation) at an elevation of 3 km (bottom row) for three consequent time moments with an interval of ten minutes. The data from 3 km height provide information about future precipitation before the actual rain starts. The example for June 19th, 2020, 9:30 AM UTC, Moscow, Russia. The color in the pictures corresponds to the levels of precipitation intensity, which may vary from (blue) to 50 mm/h (light green).
G UNCERTAINTY ESTIMATION DETAILS
Here we provide an additional details to the uncertainty estimation (UE) section in the main text.
For given pixel value at the position (I, y) for radar time horizon k. and time / we can define a ground truth V(r,k,,,,.)) Let us have five predictor models fI,..., Ss with the corresponding predictions ((1) .......... . ((5) (we omit indices here for simplicity). We define ensemble prediction ((rsk.t,..)) = $ - Dio) and the (biased) standard deviation (which we treat as an UE here) as
U (( = { R/ E ))))). We note that while we do not use an unbiased estimate, the same up to the scaling factor, and does not affect the calculated correlations. An ensemble error is defined as (E(r,,.,,z..)) (( ((r,A.t,z.)))
We use the straightforward SciPy implementations for the Pearson and Spearman correlations, which, actually approximate
in turn, correspond to two different usage scenarios. In the first scenario, an error with UE if we believe that the linear relationship (indicated by Pearson correlation) strong enough for this region, season, radar. The second scenario, which uses rank Spearman correlation, more suitable for the outlier detection and other sample ranking activities, like acquisition function for active learning.
While aggregation performed for the Fig. is pretty straightforward, aggregation for the Fig. 7 was performed follows. First, perform the averaging of ensemble error and UE grouped by horizon, radar, month, and hour over the remaining dimensions (see pseudo-SQU code listing below). After that, we calculate the correlation between the vectors of UE and ensemble error corresponding to the graph points for both left and right plots. All the correlation values we provide are statistically significant.
SELECT
AVG ensemble error) AS avg ae, AVG stdev) AS avg ue horizon, radar, month, hour
FROM
SELECT ensemble_mad stdev, horizon, radar, month, hour
FROM
GROUP BY
horizon, radar, month, hour
15

---

Under review as a conference paper at ICLR 2023
Avg. error
UE (st.dev.)
57
57
56
56
Latitude
55
55
54
0.06
54
0.12
53
0.24
53
0.30
30.0
32.5
35.0
30.0
32.5 Longitude
35.0
Longitude
Figure 7: Error and UE for the Smolensk radar averaged pixelwise over the summer season for the 10 min horizon. Large error regions are located same coordinates large UE values.
0.95
0.95
Holiza
0.90
0.90
20
40
100
120
Horizon, minites
Figure 8: UE for the ensemble case. Left: Pearson correlation coefficient between the error and UE, averaged along the radars and time, as a function the forecast horizon. Autumn and winter seasons are less predictable with the uncertainty quantification. Right: Heatmap of the correlation coefficient as a function of radars and horizon. Northern radars' errors correlation with UE is smaller.
H FIGURES: UNCERTAINTY ESTIMATION
16

---

Under review as a conference paper at ICLR 2023
I METRICS CALCULATION DETAILS
Lis1 We use raw mm/h values to calculate MSE = HxW (ys - /H x W) for a single frame, where Yi a pixel value from the ground truth sequence and the corresponding value from the predicted sequence. To calculate M MSE on the entire dataset, we iterate over the dataset and maintain a vector of length Coupput accumulating the sum of MSE values for each prediction horizon. To get the final result, we first divide the values obtained in the vector by the total number of sequences and then average them.
In order to calculate Fi-measure, we beforehand binarize ground truth and predicted sequences with some threshold. In our experiments, we took one of the standard (see (onderby et al. (2020) binarization thresholds equal to 0.1 mm/h. The metric is defined as Fi = (2 TP) TP FP FN), where TP - cases when precipitation was present both in the predicted sequences and the ground truth, FP cases where precipitation was present in the predicted sequences but was not present in the ground truth, FN cases where precipitation was not predicted but was present in the ground truth. To calculate F1. we iterate over the dataset and maintain a vector of length Coutput accumulating TP, FP and TN for each prediction horizon. After we have gone through the entire dataset, count Fj for each position in the vector, and then average the values obtained.
17


Reviews:
[{"review_id": "rjAYcq5ZETz", "reviewer": "Reviewer_TLZx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1) This paper propsed a different weather prediction model by inducing a new dataset [above russia] which is adopted to evaluate all current models such as ConvLSTM etc. \n\n2) The results looks promising on this new datasets. ", "summary_of_the_review": "Good dataset, while limited domain relativity and novelty. ", "strength_and_weaknesses": "Pro: \n1) The dataset is attractive, I wonder how the models performance across different datasets, are their performance rankings consistent or not ? \n\nCons: \n2) The methods used are relatively low-cost, I wonder how heavy cost models, such as transformers like architectures, perform comparing with these baselines. \n\nHow is the dataset usability ?  Is the model trained based on such dataset can be comparable with the most advanced weather forcast devices .\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gYDO9jrX_dk", "reviewer": "Reviewer_MvGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work builds a new weather radar dataset to explore the precipitation nowcasting task, which additionally considers a wide variety of geographical and climatic conditions.  Moreover, the authors carefully describe the acquisition process and distributions of their dataset. Then, they perform several baselines for evaluation on their proposed dataset. ", "summary_of_the_review": "I suggest the authors could submit to a more appropriate journal will have a higher impact in their community.", "strength_and_weaknesses": "Strength: \nTheir proposed weather radar dataset considers more dimensions in contrast to previously published works, which could be useful to achieve the precipitation nowcasting. This dataset covers multiple climate zones, large timestamps over a two-year period. \n\nWeaknesses:\nThe baseline methods seem too naïve and lacks more ablation study experiments.\n1.\tThe authors claim their dataset could be applied into other ML task, e.g. data studying, anomaly detection. Wish to conduct related experiments to verify this statement.\n2.\tI wonder whether there exist additional data from other zones, which could be implement the cross evaluation of their baseline models.\n3.\tThere seems lack more details related works about this topic.\n4.\tThe evaluation metric just uses MSE in Table 2, which cannot reflect the accuracy of nowcasting.\n5.\tThe results of the previous methods on the proposed dataset also should be listed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TYrS1H16w5h", "reviewer": "Reviewer_ae7S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a database of precipitation forecasting from 30 Russian meteorological stations, and a benchmark of state-of-the-art deep learning methods for the forecast estimation. ", "summary_of_the_review": "The paper is very clear and presents an interesting database and benchmark of different precipitation forecasting databases and deep learning forecast estimation models. But I don't think it is enough for an ICLR publication as there is no particular novelty on either the database nor the methods.", "strength_and_weaknesses": "Strengths:\n- The dataset is rich, containing information at several altitude levels, during 2 years, and with additional fields that are not always present in such datasets.\n- the comparison with other datasets is detailed, clear and very useful\n- a large number of state-of-the-art deep learning methods are compared on this dataset\n- a study on uncertainty estimation, performed by varying the seed of the learning phase, is performed\n\nWeaknesses:\n- from the table 1, it is not clear how this new dataset would make a difference, in particular, it looks to me that the fact that this dataset is coarse (the one with the smaller grid size yet on a large region) is probably very limiting for a task such as precipitation.\n- there are no methodological novelties\n- the uncertainty estimation is rather simple\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject