RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling
Authors: Xuanhong Chen
Kairui Feng
Naiyuan Liu
Bingbing Ni
Yifan Lu
Zhengyan Tong
Ziang Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Machine Learning for Sciences
Downscaling
Meteorological Problems

Abstract: AI-for-science approaches have been applied to solve scientific problems (e.g., nuclear fusion, ecology, genomics, meteorology) and have achieved highly promising results. Spatial precipitation downscaling is one of the most important meteorological problem and urgently requires the participation of AI. However, the lack of a well-organized and annotated large-scale dataset hinders the training and verification of more effective and advancing deep-learning models for precipitation downscaling. To alleviate these obstacles, we present the first large-scale spatial precipitation downscaling dataset named RainNet, which contains more than 62,400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep learning models in precipitation downscaling. Specifically, the precipitation maps carefully collected in RainNet cover various meteorological phenomena (e.g., hurricane, squall), which is of great help to improve the model generalization ability. In addition, the map pairs in RainNet are organized in the form of image sequences (720 maps per month or 1 map/hour), showing complex physical properties, e.g., temporal misalignment, temporal sparse, and fluid properties. Furthermore, two deep-learning-oriented metrics are specifically introduced to evaluate or verify the comprehensive performance of the trained model (e.g., prediction maps reconstruction accuracy). To illustrate the applications of RainNet, 14 state-of-the-art models, including deep models and traditional approaches, are evaluated. To fully explore potential downscaling solutions, we propose an implicit physical estimation benchmark framework to learn the above characteristics. Extensive experiments demonstrate the value of RainNet in training and evaluating downscaling models. Our dataset is available at https://neuralchen.github.io/RainNet/.

RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling
Xuanhong Chen
Kairui Feng**
Naiyuan Liu
Bingbing Ni Yifan Lu
Zhengyan Tong' Ziang Liu
Shanghai Jiao Tong University
Princeton University University of Technology Sydney {chen19910528, yifan_lu, j8804))sstu.eddee edu.cn
(kairuif)@princetoon edu (naiyuan. Liut@studet. uts. edu.au
(ziang_ lin)@oown edu
Abstract
Al-for-science approaches have been applied to solve scientific problems (e.g.. nu- clear fusion, ecology, genomies, meteorology) and have achieved highly promising results. Spatial precipitation downscaling is one of the most important meteoro- logical problem and urgently requires the participation of AI. However, the lack of well-organized and annotated large-scale dataset hinders the training and verification of more effective and advancing deep-learning models for precipita- tion downscaling. To alleviate these obstacles, we present the first large-scale spatial precipitation downscaling dataset named RainNer, which contains more than 62, 400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep learning models in precipitation downscaling. Specifically, the precipitation maps carefully collected in RainNet cover various meteorological phenomena (e.g.. hurricane, squall), which is of great help to improve the model generalization ability. In addition, the map pairs in RainNet are organized in the form of image sequences (720 maps per month or I map/hour), showing complex physical properties, e.g., temporal misalignment, tem- poral sparse, and fluid properties. Furthermore, two deep-learning learning-orientedd metrics are specifically introduced to evaluate or verify the comprehensive performance of the trained model (c.g., prediction maps reconstruction accuracy). To illustrate the applications of RainNet, 14 state-of-the-art models, including deep models and traditional approaches, are evaluated. To fully explore potential downscaling solutions, we propose an implicit physical estimation benchmark framework learn the above characteristics. Extensive experiments demonstrate the value of RainNet in training and evaluating downscaling models. Our dataset is available at htttps:/nneurllchee github. io/ T/RainNet/
1 Introduction
Deep learning has made enormous breakthroughs in the natural sciences (e.g., nuclear fusion [I] ecology [52], genomies (28), meteorology [3],, which extremely good at extracting valuable knowledge from numerous amounts of data. In recent years, with computer science development, a deluge of earth system data is continuously being obtained, coming from sensors all over the
"Equal contribution.
"Binhiing Ni is the corresponding author.
Work done during undergraduate student at Shanghai Jiao Tong University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

earth and even in space. These ever ierrinnceasing massive amounts of data with different sources and structures challenge the geoscience community, which lacks practical approaches to understand and further utilize the raw data [44]. Specifically, several preliminary works [6 59 21 43 1 55
try
to introduce deep-learning frameworks to solve meteorological problems, e.g., spatial precipitation downscaling.
In this paper, we focus on the spatial precipi- lation downscaling task [60]. Spatial precipita-
Physical O Observation
tion downscaling is a procedure to infer high- resolution meteorological information from low- resolution variables, which is one of the most important upstream components for meteorolog- ical task [3]. The precision of weather and cli- mate prediction is highly dependent on the reso- lution and reliability of the initial environmental input variables, and spatial precipitation down- scaling is the most promising solution. The im- provement of the weather/climate forecast and geo-data quality saves tremendous money and lives; with the fiscal year 2021 budget of over $1 billion, NSF funds thousands of colleges in the U.S. to research on these topics [38].
Redittio
Radiation Redilios
La 3PC
Minnoy Scn canbbtart- drop!
Conmedaa
Consuttion
Mixd
Lengrake Lubery
Buse
Retbor Made
Meteorological mechanism
LR 100kas
HR 4kmn
laput
Output
Unfortunately, there are looming issues that hin- der the research of spatial precipitation down- scaling in the deep learning community: 1) Lack of "deep-learning ready" datasets. The existing deep-learning based downscaling methods are only applied to ideal retrospective problems and verified on simulated datasets (e.g., mapping bicubic of precipitation generated by weather forecast model to original data [4), which sig- nificantly weakens the credibility of the feasibility, practicability, and effectiveness of the methods. It is worth mentioning that the data obtained by the simulated degradation methods (e.g., bicubic) is completely different from the real data usually collected by two measurement systems (e.g., satellite and radar) with different precision. The lack of well-organized and annotated large-scale dataset hinders the training and verification of more effective and complex deep-learning models for pre- cipitation downscaling. 2) Lack of tailored metrics to evaluate deep-learning based frameworks. Unlike deep learning (DL) communities, scientists in meteorology usually employ maps/charts to assess downscaling models case by case based on domain knowledge [2 56), which hinders the application of RainNet in DL communities. For example, He et. al. (11 uses log- seeiaaaivae (spatial metrics for local precipitation) and quantile-quantile maps to analyze the maps. 3) The efficient deep-learning framework for downscaling should be established. Contrary image data, the proposed real precipitation dataset covers various types of real meteorological phenomena (e.g.. hurricane, squall), and shows the physical characters (e.g., temporal misalignment. temporal sparse and fluid properties) that challenge the downscaling algorithms. Traditional computationally dense physics-driven downscaling methods are powerless to handle the increasing meteorological data size and are flexible to multiple data sources.
Figure I: Metcorological physical process. Deep Iearn- ing framework is one potentially promising solution. comparing to computational dense dynamic methods and spatial- -patial-tempora non-consistent statistical methods. No formal dataset challenges the application of deep learn- ing in downscaling. In contrast to simulated toy-dataset, our RainNet contains more than 62, 400 low- and corre- sponding high-resolution precipitation map pairs.
To alleviate these obstacles, we propose the first large-scale spatial precipitation downscaling dataset named RainNet, which contains more than 62, 400 pairs of high-quality low/high-resolutie precip- itation maps for over 17 years, ready to help the evolution of deep models in spatial precipitation downscaling. The proposed dataset covers more than 9 million square kilometers of land area, which contains both wet and dry seasons and diverse meteorological phenomena. To facilitate DL/ML and other researchers to use RainNet, we introduce 6 most concerning indices to evaluate downscaling models: mesoscale peak precipitation error (MPPE), heavy rain region error (HRRE), cumulative precipitation mean square error (CPMSB), cluster mean distance (CMD), heavy rain transition speed (HRTS) and average miss moving degree (AMMD). In order to further simplify the application of indices, we abstract them into two weighted and summed metrics: Precipitation Error Measure (PEM) and Precipitation Dynamics Error Measure (PDEM). Unlike video super-resolution, the motion of

---

Hurricane Flrrncce
50
80
Aug 25-27,
LR
Squall
Squall
HR
LR
Oct 26-28 2010
Figure 2: Dataset Visualization. Please in the figure for better observation. Please that the details of the precipitation map are partially lost due to file compression. Here we plot groups typical meteorological phenomena (hurricane and squall) in the dataset. To learn more about the dataset, please visit our project website and supplementary material. We show some video data of the dataset in the supplementary material.
the precipitation region non-rigid (i.e., fluid), while video super-resolution mainly concerns rigid body motion estimation. To fully explore how to alleviate the mentioned predicament, propose an implicit dynamics estimation driven downscaling benchmark model. Our model hierarchically aligns adjacent precipitation maps, that is, implicit motion estimation, which is very simple but exhibits highly competitive performance. Based on meteorological science, we also proved that the dataset constructed contained the full information people may need recover the higher resolution observations from lower resolution ones. We believe RainNet provides an imagenet like track for natural science research in the deep-leaming community. The main contributions of this paper are:
To the best of our knowledge, we present the first REAL (non-simulated) large-scale spatial precipitation downscaling dataset for deep learning:
We introduce 2 casy-to-use metrics evaluate the performance of downscaling models;
We propose benchmark downscaling model with strong competitiveness. We evaluate competitive potential solutions on the proposed dataset and analyze the feasibility and effectiveness of these solutions.
2 Background
At the beginning of the 1 century, geoscientists recognized that predicting the state of the atmo- sphere could be treated as an initial value problem mathematical physics, wherein future weather is determined by integrating the governing partial differential equations, starting from the observed current weather. Today, this paradigm translates into solving system of nonlinear differential equations at about half billion points per time step and accounting for dynamic, thermodynamic,

---

radiative, and chemical processes working on scales from hundreds of meters to thousands of kilome- ters and from seconds to weeks [3. The Wavier-Stokes and mass continuity equations (including the effect of the Earth's 's rotation), together with the first law of thermodynamics and the ideal gas law, represent the full set of prognostic equations in the atmosphere, describing the change in space and time of wind, pressure, density and temperature are described (formulas given in supplementary) [ These equations have to be solved numerically using spatial and temporal discretization because of the mathematical intractability of obtaining analytical solutions, and this approximation creates distinction between so-called resolved and unresolved scales of motion.
2.1 Spatial Downscaling of Precipitation
The global weather forecast model is treated as computational problem, and relying on high-quality initial data input. The error of weather forecast would increase exponentially over time from this initial error of the input dataset. Downscaling is one the most important approaches to improving the initial input quality. Precipitation is one of the essential atmospheric variables that are related to daily life. could easily observed, by all means, e.g., gauge station, radar, and satellites. Applying rainfall is far more meaningful
downscaling methods to precipitation and creating high-resolutin than deriving other variables, while it is the most proper initial task to test deep learning's ' power in geo-science. The traditional downscaling methods can be separated into dynamic and statistical downscaling.
Dynamic downscaling treats the downscaling as optimization problem constraint on the physical laws. The dynamic downscaling methods find the most likely precipitation over space and time under the pre-defined physical law. It usually takes over 6 hours to downscale 6-hour precipitation scenario globally on supercomputers [00. As the dynamic downscaling relies on pre-defined known macroscopic physics, more flexible weather downscaling framework that could easily blend
different sources of observations and show the ability to describe more complex physical phenomena on different scales is desperately in need.
Statistical downscaling is trying speed up the dynamic downscaling process. The input of statistical downscaling is usually dynamic model results or two different observation datasets on different scales. However, due to the quality of statistical downscaling results, people rarely apply statistical downscaling to weather forecasts. These methods are currently applied in the tasks not requiring high data quality but more qualitative understanding. e.g., climate projection, which forecasts the weather for hundreds of years on coarse grids and uses statistical downscaling to get detailed knowledge of medium-scal scale future climate system.
3 RainNet: Spatial Precipitation Downscaling Imagery Dataset
3.1 Data Collection and Processing
To build up standard realistic (non-simulated) downscaling dataset for computer vision, we selected the eastern coast of the United States, which covers large region (7 million km²; 105" ~ 65 25 50 N, GNU Free Documentation License 1.2) and has 20-year high-quality precipitation observations. It is worth mentioning that this region is selected for two reasons: on the one hand, this region has both LR and HR instrument observations, and on the other hand, it covers the various meteorological phenomena. We collected two precipitation data sources from National Stage IV QPE Product (StagelV (37]; high resolution at 0.04 (approximately 4kma), GNU Free Documentation License 1.2) and North American Land Data Assimilation System (NIDAS (62) low resolution at 0.125 (approximately 13km)). StageIV is mosaicked into a national product at National Centers for Environmental Prediction (NCEP), from the regional hourly/6-hour hourly multi-sensor (radar+gauges) precipitation analyses (MPEs) produced by the 12 River Forecast Centers over the continental United States with some manual quality control done at the River Forecast Centers (RFCs). NLDAS is constructed
consistent datasets from the gauges and remote sensors to support modeling activities. Both products are hourly updated and both available from 2002 to the current age.
In our dataset, we further selected the eastern coast region for rain season (July November, covering hurricane season; hurricanes pour over 10% annual rainfall in less than 10 days). We matched the coordinate system to the lat-lon system for both products and further labeled all the

---

hurricane periods happening in the last 17 years. These heavy rain events are the largest challenge for weather forecasting and downscaling products. As heavy rain could stimulate a spreaddin flood, which threatens local lives and arouses public evacuation. If people underestimate the rainfall, a potential flood would be underrated; while over-estimating the rainfall would lead to unnecessary evacuation orders and flood protection, which is also costly.
3.2 Dataset Statistics
At the time of this work, we collected and processed precipitation data for the rainy season for 17 years from 2002 to 2018. One precipitation map pair per hour, 24 precipitation map pairs per day. In detail, we have collected 85 months 62424 hours, totaling 62424 pairs of high- resolution and low-resolution precipitation maps. The size of the high-resolution precipitation map is 624 999, and the size of the low-resolution is 208 x 333. Various meteorological phenomena and precipitation conditions (e.g., hurricanes, squall lines.) are covered in these data. The precipitation map pairs in RainNet are stored in HDF5 files that make up 360 GB of disk space. We select 2 typical meteorological phenomena and visualize them in Fig. 2 Our data is collected from satellites, radars, gauge stations, etc., which covers the inherent working characteristics of different meteorological measurement systems. Compared with traditional methods that generate data with different resolutions through physical model simulation, our dataset is of great help for deep models to learn real meteorological laws. The proposed dataset can be used under the Creative Common License (Attribution CC BY).
3.3 Dataset Analysis
In order to help design a more appropriate and effective precipitation downscaling model, we have explored the property of the dataset in depth. As mentioned above, our dataset is collected from multiple sensor sources (e.g., satellite, weather radar), which makes the data show a certain extent of misalignment. Our efforts here are not able to vanquish the misalignment. This is an intrinsic problem brought about by the fusion of multi-sensor meteorological data. Limited by observation methods (e.g., satellites can only collect data when they fly over the observation area), meteorological data is usually temporal sparse, e.g., in our dataset, the sampling interval between two precipitation maps is one hour. The temporal sparse leads to serious difficulties in the utilization of precipitation sequences. Additionally, the movement of the precipitation position is directly related to the cloud, It a fluid movement process that completely different from the rigid body movement concerned in image/video super-resolution. At the same time, the cloud will grow or dissipate in the process of flowing and even form new clouds, which further complicates the process. In the nutshell, although existed SR is a potential solution for downscaling, there is big difference between the two. Especially, the three characteristics of downscaling mentioned above: temporal misalignment, temporal sparse, fluid properties, which make the dynamic estimation of precipitation more challenging.
4 Evaluation Metrics
Due to the difference between downscaling and traditional image/video super-resolution, the metrics that work well under SR tasks may not be sufficient for precipitation downscaling. By gathering the metrics from the meteorologic literature (the literature includes are [66] 35 21
40, [1], we select 6 most common metrics metric may have multiple names in different literature) to reflect the downscaling quality: mesoscale peak precipitation error (MPPE), cumulative precipitation mean square error (CPMSE), heavy rain region error (HRRE). cluster mean distance (CMD), heavy rain transition speed (HRTS) and average miss moving degree (A (AMMD). These 6 metrics can be separated as reconstruction metries: MPPE, HRRE, CPMSE, AMMD, and dynamic metries: HRTS and CMD.
The MPPE
is calculated as the difference of top quantile between the generated/real rainfall dataset which considers both spatial and temporal properties of mesoscale meteorological systems, e.g., hurricane, squall. This metric is used in most of these papers (for example [66] 35 [4
21 40
61] suggest the quantile analysis to evaluate the downscaling quality).
The CPMSE (mm² 2) measures the cumulative rainfall difference on each pixel over the time-axis of the test set, which shows the spatial reconstruction property. Similar metries are used
5

---

Ground Truth 1
A
UpSempling
Ax-3
Conr
Cozs
RRDB
Ag-2
L1 Percetual Loss
Results 1
Shared W eights Implicit Dynamic Estimation Module
Ingut Prames
Dovnscaling Backhone
Figure The pipeline of our proposed baseline model for spatial precipitation downscaling.
in [66 35 61 calculated as the pixel level difference of monthly rainfall and used in 21 level difference of cumulative rainfall with different lengths of the record.
pixel
The HRRE (km²) measures the difference in heavy rain coverage each time slide between generated and labeled test set, which shows the temporal reconstruction ability of the models. The AMMD (radian) measures the average angle difference between main rainfall clusters. Similar metrics are used in [66 35 [1] as rainfall coverage of a indefinite number precipitation level and
used in 21 40
continuous spatial analysis.
As a single variable dataset, hard to evaluate the ability of different models to capture the precipitation dynamics when temporal information is not included (a multi-variable dataset may have wind speed, a typical variable representing dynamics, included). So here we introduce the first-order temporal and spatial variables to evaluate the dynamical property of downscaling results. Similar approaches are suggested in B5 14 40]. The CMD (km) physically compares the location difference of the main rainfall systems between the generated and labeled test which could be also understood as the RMSE of the first-order derivative of precipitation data in spatial directions. The HRTS (km/ (ho///rr) measures the difference between the main rainfall system moving speed between the generated and labeled test set which shows the ability models to capture the dynamic property, which could be also understood as the RMSE the first-order derivative precipitation data on the temporal direction. Similar metrics are suggested in [35 14, 40 as the auto-regression analysis and the differential analysis.
More details about the metrics and their equations are given in supplementary materials. One metrics group (MPPE, HRRE, CPMSE, AMMD) mainly measures the rainfall deviation between the gener- ated precipitation maps and GT. The other group (HRTS and CMD) mainly measures the dynamic deviation of generated precipitation maps. So many measures will create difficulties in application and understanding. In order to further simplify the application of indices, we finally abstract them into two weighted and summed metrics: Precipitation Error Measure (PEM) and Precipitation Dynamics Error Measure (PDEM). We first align the dimensions of these two groups of metrics respectively. The first group of metrics (MPPE, HRRE, CPMSE, AMMD) is normalized, weighted and summed to get the precipitation error measure (PEM). According to [17]. all the metrics are transferred to Percent Bias (PBIAS) be suitable for metrics weighting. The original definition of PBIAS the bias divided by observation, as PBIAS IQ model Swt.e//oolllaaa.a viding the metrics with annual mean observations of the original variables (AMO), as PBIAS |Mericss P&M | |/MM O, PEM Metrics PPM = [MPPE, H RRE, CPMSE, AMMD) DD. In our dataset, AMO 64, AMOPEN HRRE 533, AMO = 0.64, AMO = 332, AMOPEM P.M. = 15, AMOC PEA 26. The metrics then are ensembled to a single metric (PEM) with equal weight, as
Here we rewrite the original metrics to PBIAS by di-
PEM =, 0.25 PBIAS.EM Following the same procedure, we then ensemble the second group of dynamic metrics (HRTS and CMD) to single metries PDEM Σ, 0.5 PBIAS, We also include the most commonly used metric RMSE as one single metric in our metrics list. RMSE could evaluate both the reconstruction and dynamic property of the downscaling results.
Benchmark of RainNet
As a potential solution, Super-Resolution (SR) frameworks are generally divided into the Single- Image Super- Resolution (SISR) and the Video Super-Resolution (VSR). Video Super-Resolution

---

is able to leverage multi-frame information to restore images, which better matches the nature of downscaling. We will demonstrate this judgment in Sec. Sec.6.1] The VSR pipeline usually contains three components: deblurring, inter-frame alignment, and super-resolution. Deblurring and inter-frame alignment are implemented by the motion estimation module. There are four motion estimation
frameworks: 1) RNN based [29, 49 25 19: 2) Optical Flow [65]; 3) Deformable Convolution
based [5] 63 | 4) Temporal Concatenation [26] 6 32). In fact, there is another motion estimation scheme proposed for the first time in the noise reduction task [50], which achieves an excellent video noise reduction performance. Inspired by [50], we design an implicit dynamics estimation model for the spatial precipitation downscaling. is worth mentioning that our proposed model and the above four frameworks together form a relatively complete candidate set of dynamic estimation solutions.
Proposed Framework. As shown in Fig. 3 our framework consists of two components: Implicis dynamic estimation module and downscaling Backbone. These two parts are trained jointly. Suppose there are N adjacent low-resolution precipitation maps {T/ _ 1.-,, Iy..... T+ 2 }- Here, we
employ superscript .f denotes the high resolution maps and _I. for the low resolution counterpart. The task is to reconstruct the high- nigh-resolution precipitation map 1/ of 14. The implicit dynamic estimation module is composed of multiple vanilla networks A AN 3 (N paper) sharing weights. Each vanilla network receives three adjacent frames as input, outputs, and intermediate results. The intermediate result can be considered as a frame with implicit dynamic aligmment. We concatenate all the intermediate frames as the input of the next module. The specific structure of the vanilla network downscaling backbone is to restore the high-resolution precipitation map I based on the aligned intermediate frames. In order to make full use of multi-scale information, in-Residual Dense Blocks (58) in the network. We employ the pnterollationtoovoluite the up- sampling operator to reduce the checkerboard artifacts. After processing by downscaling backbone get the final estimated HR map 1
in this
be found in the supplementary materials. The main task of the
multiple Residual- [39] as
Model objective. The downscaling task is essentially to restore high-resolution precipitation maps. We learn from the super-resolution task and also apply L1 and perceptual loss [27 as the training loss of our model. The model objective is shown below:
(1) =
+A o(t)) x(T lla,
(L)
where & denotes the pre-trained VGG19 network 47), we select the Relu5 4 (without the activa- tor [58]) as the output layer. Å is the coefficient to balance the loss terms. In our framework, we set 20 to balance the order of magnitude between losses.
Experimental Evaluation
We conduct spatial precipitation downscaling experiments illustrate the application of our pro- posed RainNet and evaluate the effectiveness of the benchmark downscaling frameworks. Following the mainstream evaluation protocol of DL/ML communities, cross-validation is employed. In
detail, we divide the dataset into 17 parts (2002.7~2002.11, 2003,7 2004.7~~204.11
2005.7~~000.11, 2006.7~~006.11, 2007.7~-007.11, 2008.7~2008.10 2009.7~~200..11 2010.7--010.11, 2015.7~2015.11 2011.7~2011.11, 2012.7-2012.11, 2013.7~2013.11, 2014.7--011.111,
by year, and sequentially employ each year as the test set and the remaining 16 years as the training set, that is, 17-fold vross-validation. All models maintain the same training settings and hyperparameters during the training phase. These data cover various complicated precipitation situations such as hurricanes, squall lines, different levels of rain, and sunny days. It is sufficient to select the rainy season of the year as the test set from the perspective of meteorology, as the climate of one area is normally stable.
6.1 Baselines
The SISR/VSR and the spatial precipitation downscaling are similar to some extent, so we argue that the SR models can be applied to the task as the benchmark models. The input of SISR is a single image, and the model infers a high-resolution image from it. Its main focus to generate high-quality texture details to achieve pleasing visual effects. In contrast, VSR models input multiple frames of
7

---

Approach
MPPF4
HRREJ.
AMMD
CPMSEL
HRTS)
CMD1
PEM.
PDEM.
RMSEX 100 1
Kriging
4.036
339.641
0.204
4.891
9.958
12.277
0.259
0.568
0.372
Bicubic
4.600
306.996
0.208
3.678
10.453
12.389
0.247
0.587
0.345
SRCNN
5,333
296.950
0.225
3.929
10.091
12.396
0.252
0.575
0.405
SRGAN
14.125
298.290
0.221
91.464
9,429
11.891
0.352
0.543
0.603
EDSR
4.748
288.354
0.204
3.292
9.605
12.259
0.236
0.556
0.329
ESRGAN
6.205
407.848
0.219
4.483
10.201
17.035
0.305
0.668
0.563
DBPN
6.596
302.278
0.212
5.692
9.869
11.336
0.256
0.547
0.380
RCAN
4.709
172.189
0.200
3,062
9.772
12.055
0.227
0.558
0.325
SRGAN-V
10.007
291.546
0.210
35.932
8.276
10.448
0.286
0.477
0.557
EDSR-V
4.592
289.331
0.201
3.269
8.484
11.214
0.235
0.498
0.323
ESRGAN-V
7.187
413.398
0.213
4.010
7.887
10.695
0.309
0.469
0.399
RBPN
4.816
287.214
0.201
2.680
8.267
11.244
0.235
0.492
0.317
EDVR
2.148
213.034
0.179
1.352
8.479
10.060
0.180
0.476
0.329
Ours
4.198
221.859
0.191
1.890
7.723
9.568
0.197
0.441
0.312
Table 1: Cross-validation results. Comparison with state-of-the art super resolution approaches. The best performance marked with red (Ist best), blue (2nd best), In practical applications, we recommend using only PEM, PDEM, and RMSE to evaluate model performance. For ease of use, we provide packages in our open source project that directly compute these metrics.
images (e.g.. frames, frames.). In our experiments, we employ 5 frames. The core idea of VSR models is to increase the resolution by complementing texture information between different frames. is worth mentioning that VSR models generally are equipped with a motion estimation module to alleviate the challenge of object motion to inter-frame information registration.
We evaluated 7 state-of-the-arr SISR
frameworks (i.e.. Bicubic [29]. SR-
ESRGAN
CNN 12], SRGAN 30, EDSR (33,,
ES-
RGAN (58). DBPN 18), RCAN 667
16
Heavy rain transition speed
and VSR frameworks (i.e., SRGAN-
1.5
V. ED ESRGAN-V, RBPN [19],
EDVR (57) of which VSR methods (i.e., SRGAN-V, EDSR-V, ESRGAN-V) are modified from SISR. In order to ver- ify the impact of increasing the number of input frames on performance, we build SRGAN-V, EDSR-V V and ESRGAN-V by concatenating multiple frames of precip- itation maps the input of the model. In addition, we also evaluate Kriging [48]. which is the de-facto standard method in the meteorological community. The men- tioned 9 metrics used to quantitatively evaluate the performance of these SR mod- els and our method. Further, we select some disastrous weather as samples for qualitative analysis to test the model's s ability to learn the Pytorch for Bicubie. We use 4 NVIDIA 2080 Ti GPUs for training. We train all models with following setting. The batch size is set as 24. Precipitation maps are random crop into 64 x 64. We employ the Adam optimizer, betal is 0.9, and beta2 is 0.99. The initial learning rate is 0.001, which is reduced to 1/10 every 50 epochs, and total of 200 epochs are trained. We evaluate benchmark frameworks with 17-fold cross-validation. The downscaling performances are shown in Tab. [ We divide the indicators mentioned above into two groups. PDEM measures the model's 's ability to learn the dynamics of precipitation. PEM illustrates the model's ability to reconstruct precipitation.
14
13
Kriging Bicubic
EDSR
12
RBPN
SRGAN
O
RCAN
SRCNN
OEDSR-V
о DBPN
11
GESGGANV
SRGAN-V
10
Ours
EDVR
10
Cluster's mean distance
Figure 4: The dynamic property of benehmark algorithms. The frameworks of VSR are gathered in the lower-left corner of the figure, which demonstrates that VSR methods are superior to SISR and traditional methods in dynamic properties.
dynamic properties of the weather system. And we employ the implementation
8

---

LR(green) HH
Bicubic
SRCNN
SRGAN
EDSR
ESRGAN
DBPN
RCAN
SRGAN-V
ESRGAN-V
EDVR
Figure 5: Visual comparison with statc-of-the-ar Super Resolution approaches. Please zoom in the figure for better observation. More results can be found in suppl. We show some vidco results the suppl.
From Tab. [ we can learn that the overall performance of the VSR methods are better than SISR models, which shows that the dynamic properties mentioned above are extremely important for the downscaling model. Furthermore. can be seen from Fig. that the SISR method is clustered in the upper right corner of the scatter plot, and the VSR method is concentrated the lower-left corner, which further shows that the dynamic properties of the VSR methods are overall better than the SISR methods. In addition, our method achieves the 1st best performance RMSE, PDEM, and achieve the second- performance PEM, which demonstrates that the proposed implicit dynamic estimation framework used is feasible and effective. is worth mentioning that the traditional downscaling method Kriging performs better than many deep learning models (e.g.. SRGAN, ESRGAN).
6.1.1 Qualitative analysis
We visualized the tropical cyclone precipitation map of the 166th hour (6th) in September 2010 and the high-resolution precipitation map generated by different methods. As shown in Fig. 5 the best perceptual effects generated by EDVR and Our framework. Zooming in the result image, the precipitation maps generated by SRGAN and EDSR present obvious checkerboard artifacts. The reason for the checkerboard artifacts should be the relatively simple and sparse texture pattern in precipitation maps. The results generated by Bicubic, RCAN, Kriging, and SRCNN are over-smooth. DBPN even cannot reconstruct the eye of the hurricane. Especially, the result generated by Kriging is as fuzzy as the input LR precipitation map. In conclusion, the visual effects generated by the VSR methods are generally better than the SISR methods and the traditional method. From the perspective of quantitative and qualitative analysis, the dynamics estimation framework is very critical for downscaling.
Related Works
Downscaling in Geoscience. Downscaling is fundamental task in geoscience and meteorology, which researchers have been interested in for a long time [60]. Most statistical downscaling methods
regard this problem as piset-wise regression [36] (33 46 direct maximum likelihood estimation of

---

the high-resolution data from the low-resolution data [2]. Bürger et. al. [ compared correctio spatial disaggregation (BCSD), quantile regression networks, and expanded downscaling (XSD) for climate downscaling. Four fundamental statistical methods and three more advanced machine learning methods downscale daily precipitation in the Northeast United States were compared in (33 Recently, deep learning methods have been applied to solve the climate downscaling problem.
DeepSD [54] tried to employ SRCNN [12] on precipitation downscaling. White et. al. [59] tested the performance of Generative Adversarial Networks (GANs) downscaling weather models. ClimAlign [6] proposed a novel deep learning method for statistical downscaling, treating it as an unsupervised domain alignment problem without using paired www/high-resolution training data.
Single Image huper-Resolution(SSHH For deep learning based method, Dong et. al. first proposed
an end-to-end model SRCNN [22 using deep convolutional network. SRGAN [30 proposed by
Ledig et. uses perceptual loss for finer texture. The enhanced version ESRGAN (58) introduces RRDB as basic network unit, using relative discriminator and perceptual loss of features before activation.
Video Super-Resolution VSR utilizes the temporal information of image sequences. Wang et. al. proposed a framework EDVR [57], which devises PCD module to handle large motion alignment, and the TSA fusion module with temporal and spatial attention. Xie et. al. proposed the tempoGAN (64) for fluid flow Resolttesolution, synthesizing 4-D physics fields with a temporal discriminator.
8 Future Work and Limitations
The recent success of vision based transformer (i.e., ViT) models [3 34 31 7
15 in image generation tasks has sparked performance revolution. With its larger effective receptive field and high-order interaction, VIT greatly surpasses the previous dominant convolutional neural networks (i.e., CNNs) 20 23
67 [3] in both video and image processing. Witnessing this situation, we believe that the introduction of the VTT model will be of great help to the real downscaling
24
12
30
58 57
task presented by RainNet. At the same time, the diffusion based models [22] g 45
41
45) also
shows unparalleled performance potential in generating images/videos (e.g., text to image), despite
its notorious high computational complexity. In future work, we will deeply explore the tailored application of ViT model and diffusion-bait model in RainNet to better solve this important meteorological problem.
Conclusion
In this paper, built the first large-scale real precipitation downscaling dataset for the deep learning community. This dataset has 62424 pairs of HR and LR precipitation maps in total. We believe this dataset will further accelerate the research on precipitation downscaling. Furthermore, we analyze the problem in-depth and put forward three key challenges: temporal misalignment, temporal sparse and fluid properties. In addition, above challenges. At the same time, we evaluated the mainstream SISR and VSR models and found that none of these models can perfectly solve RainNet's problems. Therefore, the downscaling task on this dataset is still very challenging. This work still remains several open problems. Currently, the data domain of this research is limited to the eastern U.S. In future research, we would enlarge the dataset to a larger domain. The dataset is only a single variable now. In future research, we may include more variables, e.g. temperature and wind speed.
propose implicit dynamic estimation model alleviate the
Acknowledgements
This work was supported by National Science Foundation of China (U20B2072, 61976137). We also appreciate the supporting of computing resources by the student innovation center of Shanghai Jiao Tong University.
References
[1] Rico Angell and Daniel R. Sheldon. Inferring latent velocities from weather radar data using gaussian processes. In Advances in Neural Information Processing Systems 31: Annual
10

---

Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 8998-9007, 2018.
[2] Jorge Luis Baño-Medina, Rodrigo García Manzanas, José Manuel Gutiérrez Llorente, et al. Configuration and intercomparison of deep learning neural models for statistical downscaling. 2020.
[3] Peter Bauer, Alan Thorpe, and Gilbert Brunet. The quiet revolution of numerical weather prediction. Nature, 2015.
[4] Berrisford, D.P. Dee, P. Poli, R. Brugge, Mark Fielding, Manuel Fuentes, P.W. Kållberg, S. Kobayashi, S. Uppala, and Adrian Simmons. The era-interim archive version 2.0. 2011.
[5] Bürger, TQ Murdock, AT Werner, SR Sobie, and AJ Cannon. Downscaling extremes- an intercomparison of multiple statistical methods for present climate. Journal of Climate, 2012.
[6] Jose Caballero, Christian Ledig, Andrew P. Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. Real-time video super-reso resolution with spatio-temporal networks and motion compensation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2017.
[7] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, 2021.
[8] Xuanhong Chen, Hang Wang, and Bingbing Ni. X-volution: On the unification of convolution and self-attention. CoRR, abs/2106.02253. 2021.
[9] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: conditioning method for denoising diffusion probabilistic models. In 202/ IEEE/CVF Interna- tional Conference Computer Vision, ICCV 2021, 2021.
[10] PHILIPPE Courtier, J-N Thépaut, and Anthony Hollingsworth. A strategy for operational implementation of 4d-var, using an incremental approach. Quarterly Journal of the Royal Meteorological Society, 1994.
[11] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022.
[12] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang, Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell., 2016
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[14] Marie Ekström. Metrics identify meaningful downscaling skill in wrf simulations of intense rainfall events. Environmental Modelling & Software, 79:267-284, 2016.
[15] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, 2021.
[16] Brian Groenke, Luke Madaus, and Claire Monteleoni. Climalign: Unsupervised statistical downscaling of climate variables via normalizing flows. CoRR, 2020.
[17] Hoshin Vijai Gupta, Soroash Sorooshian, and Patrice Ogou Yapo. Status of automatic calibration for hydrologic models: Comparison with multilevel expert calibration. Journal of hydrologic engineering, 4(2):135-143 1999.

---

[18] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Deep Sack-projection networks for super-resolution. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018.
[19] Muhammad Haris, Gregory Shakhnarovich, and Norimichi Ukita. Recurrent projeprroe network for video super-resolution In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, Computer Vision Foundation / IEEE, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, 2016.
[21] Xiaogang He, Nathamiel W Chancy, Mare Schleiss, and Justin Sheffield. Spatial downscaling of precipitation using adaptable random forests. Water resources research, 2016.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, 2020.
[23] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1 044611 2017.
[24] Xiwei Hu, Xuanhong Chen, Bingbing Ni, Teng Li, and Yutian Liu. Bi-volution: A static and dynamic coupled filter. In Thirty-Sixth AAAI Conference Artificial Intelligence, AAAI 2022, 2022.
[25] Yan Huang, Wei Wang, and Liang Wang. Bidirectional recurrent convolutional networks for super-resolution. In Advances in Neural Information Processing Systems, 2015.
[26] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA. June 18-22, 2018 IEEE Computer Society, 2018
[27] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 11, 2016.
[28] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Paredes. Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure predic- tion with AlphaFold. Nature, 2021.
[29] Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acousties, speech, and signal processing, 1981.
[30] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017. IEEE Computer Society, 2017.
[31] Jingyun Liang. Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. IEFF/VV International Conference on Computer Vision Workshops, ICCVW 2021, 2021.
12

---

[32] Renjie Liao, Xin Tao, Ruiyu Li. Ziyang Ma, and Jiaya Jia. Video super-resolution via deep draft-ensemble learning. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 2015.
[33] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEFE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July
21-26, 2017. IEEE Computer Society, 2017.
[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF Interaational Conference on Computer Vision, ICCV 2021, 2021.
[35] Douglas Maraun, Martin Widmann, José M Gutiérrez, Sven Kotlarski, Richard E Chandler, Elke Hertig, Joanna Wibig, Radan Huth, and Renate AI Wilcke. Value: A framework to validate downscaling approaches for climate change studies. Earth's 's Future, 3(1):1-14, 2015.
[36] Mohammad Reza Najafi, Hamid Moradkhani, and Susan A Wherry, Statistical downscaling of precipitation using machine learning with optimal predictor selection. Journal of Hydrologic Engineering. 2011.
[37] Brian R. Nelson, Olivier P. Prat, D.-J. Seo, and Emad Habib. Assessment and Implications of NCEP Stage IV Quantitative Precipitation Estimates for Product Intercomparisons. Weather and Forecasting, 2016.
[38] NSF. Nsf geosciences directorate funding by institution type. AGI Report, 2020.
[39] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 2016.
[40] SC Pryor and JT Schoof. Differential credibility assessment for statistical downscaling. Journal of Applied Meteorology and Climatolgyy, 5933):1333-134, 2020.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.001155, 2022.
[42] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 2021.
[43] Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, al. Skillful precipitation nowcasting using deep generative models of radar. Nature, 2021.
[44] Markus Reichstein, Gustau Camps-Vall Valls, Bjom Stevens, Martin Jung, Joachim Denzler, Nuno Carvalhais, et al. Deep learning and process understanding for data-driven earth system science. Nature, 2019.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, 2022.
[46] DA Sachindra, Khandakar Ahmed, Md Mamunur Rashid, S Shahid, and BJC Perera. Statistical downscaling of precipitation using machine learning techniques. Atmospheric research, 2018.
[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[48] Michael I. Stein. Interpolation of spatial data: some theory for kriging. Springer Science Business Media, 2012.
13

---

[49] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revealing deep video super-resolution In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEFE Computer Society, 2017.
[50] Matias Tassano, Julie Delon, and Thomas Veit. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
[51] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. TDAN: -emporallydeformmable alignment network for video super-resolution. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
[52] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde. Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. Nature communications, 2022.
[53] Thomas Vandal, Evan Kodra, and Auroop R Ganguly. Intercomparison of machine learning methods for statistical downscaling: the case of daily and extreme precipitation. Theoretical and Applied Climatology, 2019.
[54] Thomas Vandal, Evan Kodra, Sangram Ganguly, Andrew Michaelis, Ramakrishna Nemani, and Auroop R Ganguly. Deepsd: Generating high resolution climate change projections through single image super-resolution. In Proceedings of the 23rd acm sigkad international conference on knowledge discovery and data mining, pages 1663-1672, 2017.
[55] Mark S. Veillette, Siddharth Samsi, and Christopher J. Mattioli. SEVIR: A storm event imagery dataset for deep learning applications in radar and satellite meteorology. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[56] Daniel Walton, Neil Berg, David Pierce, Ed Maurer, Alex Hall, Yen-Heng Lin, Stefan Rahimi, and Dan Cayan. Understanding differences in california climate projections produced by dynamical and statistical downscaling. Journal of Geophysical Research: Atmospheres, 2020.
[57] Xintao Wang. Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
[58] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceed- ings of the European Conference on Computer Vision (ECCV), pages 0-0, 2018.
[59] BL White, A Singh, and A Albert. Downscaling numerical weather models with gans, AGUFM, 2019.
[60] Robert I. Wilby, TML. Wigley, D Conway, PD Jones, BC Hewitson, J Main, and DS Wilks. Statistical downscaling of general circulation model output: A comparison of methods. Water resources research, 1998.
[61] Adrienne M Wootten, Elias C Massoud, Agniv Sengupta, Duane E Waliser, and Huikyo Lee. The effect of statistical downscaling on the weighting of multi-model ensembles of precipitation. Climate, 8(1 133, 2020.
[62] Youlong Xia, Kenneth Mitchell, Michael Ek, Justin Sheffield, Brian Cosgrove, Eric Wood, Lifeng Luo, Charles Alonge, Helin Wei, Jesse Meng, Ben Livneh, Dennis Lettenmaier, Victor Koren, Qingyun Duan, Kingtse Mo, Yun Fan, and David Mocko. Continental-scafe water and energy flux analysis and validation for the north american land data assimilation system project phase 2 (nldas-2): 1. intercomparison and application of model products. Journal of Geophysical Research: Amoopheres, 2012.
[63] Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P. Allebach, and Chenliang Xu. Zooming slow-mo: Fast and accurate oneg space-time video super-resolution. In 2020 IEEF/CVF Conference on Computer Vision and Pottern Recognition, CVPR 2020, Seattle WA, USA, June 13-19, 2020, IEEE, 2020.
14

---

[64] You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempogan: A temporally coherent, volumetric gan for super-resolution fluid flow. ACM Transactions on Graphies (TOG), 2018.
[65] Tianfan Xue, Baian Chen, Jiajun Wu, Dongla Wei, and William T. Freeman. Video enhancement with task-trie flow. Int. J. Comput. Vis., 2019.
[66] Xuebin Zhang and Feng Yang. Relimdex (1.0) user manual. Climate Research Branch Environ- ment Canada, 22, 2004.
[67] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang. Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, Lecture Notes in Computer Science. Springer, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No]. [ or [N/A]. You are strongly encouraged include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section 3.2
Did you include the license the code and datasets? [Yes] See Section
Did you include the license to the code and datasets? [Yes]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Discussed in Conclusion section
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as a URL)? [Yes] See supplemen- tal material
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Sec.4.2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Sec.4.2
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Sec.2.2
(b) Did you mention the license of the assets? [Yes] See Sec.2.2
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
15

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "yXth6737L-j", "reviewer": "Reviewer_aP2Q", "summary": "The paper proposed a large-scale spatial precipitation downscaling dataset named SPDNet. The dataset contains more than 62400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, and covers more than 9 million squre kilometers of land area. The author also introduced 6 metrics that evaluate different aspects of the downscaling models, and 2 summary metrics that combine these 6 individual metrics. The author viewed the task as a video super-resolution problem and compared 14 methods. From the experimental results, the overall performance of the video super-resolution (VSR) models are better than Single-Image Super-Resolution (SISR) models.\n", "questions": "1. The author mentioned that the dataset contains lots of different events such as hurricane, squall. Are the sequences in the dataset marked with the event type?\n", "limitations": "1. The paper will be limited regarding the coverage of the baseline methods. However, it is difficult to cover all the latest image super-resolution methods so it is acceptable.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Stengths:\n\n1. The paper proposed the first large-scale precipitation downscaling dataset. This is an important scientific problem and a large-scale dataset can help move the area forward. In addition, the author pointed out the unique characteristics of the task such as temporal misalignment, temporal sparse, and fluid properties.\n2. The paper proposed 6 metrics for evaluating the models, including 4 reconstruction metrics that focuses on evaluating if the predicted high-resolution precipitation map matches the ground-truth, and 2 dynamic metrics that evaluate the dynamics of the predicted precipitation (via first order dynamics).\n3. The paper compared 14 models, including the Kriging method that has been widely used in the geospatial community, and other SISR and VSR methods. The author also extended SRGAN, EDSR, ESRGAN to be VSR methods.\n\nWeaknesses\n1. It seems that RMSE is itself a very good summary metric. Thus, it is not clear why we will still need PEM / PDEM.\n2. Currently, the state-of-the-art image super-resolution model is based on vision Transformers (e.g., SwinIR) and the author need to reference the latest progress in this area.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jjMUDMTFaxK", "reviewer": "Reviewer_5jsP", "summary": "This paper presents a large scale dataset for spatial precipitation downscaling which contains more than 62, 400 pairs of high-quality low/high-resolution precipitation maps for over 17 years, ready to help the evolution of deep learning models in precipitation\ndownscaling. The precipitation maps  collected in the dataset  cover various meteorological phenomena such as hurricane and  squall. The data are organised in time series of maps, with 720 maps/month. Comprehensive metrics are also provided to evaluate the performances of models. ", "questions": "Eastern coast of US has been selected for data collections. What about other regions ?", "limitations": "The dataset lacks precipitation maps for several regions", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well written and brings comprehensive dataset for spatial precipitation. However  the technical novelty is low.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "WBmpyaytXL", "reviewer": "Reviewer_9mB6", "summary": "There is not much to say here. The paper organizes a large precipitation dataset from both high res and low res sources and illustrates some baselines. The problem addressed is both interesting to ML (esp. to those interested in mixing physics and ml), and important. The question then, is; given that this data can be obtained directly from the original sources, is a new organization of it necessary and does it warrant a publication in an ML conference. Although I expect other reviewers will disagree, I have a positive view. Publishing this paper will probably increase interest in ML in this set of problems, and some in the community will find the dataset useful.", "questions": "SPDnet is not a very good name…", "limitations": "It says in the checklist that the code and the data are proprietary. This needs to be clarified. What’s the point of publishing a dataset if it cannot be used by others? ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "See above", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1mMfRqp4pty", "reviewer": "Reviewer_WVYT", "summary": "This paper proposed a dataset consists of precipitation image sequences named SPDNet for spatial precipitation downscaling as well as a novel implicit dynamics estimation driven model. The proposed model as well as baseline models are evaluated on SPDNet with task specific metrics. ", "questions": "1. What are the advantages of using the real data collected by two different systems for training super resolution models than using simple downsampling algorithms? If the two domains need to be similar, directly downsampling the high resolution data is both cheap and effective. If the two domains need to differ a lot from each other, it would be better to categorize the task as domain transferring instead of \"spatial precipitation downscaling\".\n2. The qualitative results shown in Figure 4 are hard to distinguish. Could the authors provide more evidence to demonstrate that the models achieving better PEM and PDEM generate better predictions? ", "limitations": "Listed in **Weaknesses**", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. **Valuable dataset**: The large scale dataset SPDNet is in high resolution and in sequence, which is a valuable contribution to data-driven meteorological research.\n2. **Benchmark**: The authors have evaluate SOTA super resolution models on the proposed dataset ,and thus provide a good benchmark.\n\n**Weaknesses**\n1. **Necessity of low resolution data**: The authors claim in Introduction that the data obtained by simulated degradation is different from the real data collected by two different systems. However, there is no further discussion on it. It is insufficient to argue that their approach is better than obtaining low resolution data by simply downsampling the high resolution data. The author should compare these two approaches empirically, e.g., demonstrate that models trained with real data are able to reconstruct better high resolution data and hence boost the performance on downstream tasks.\n2. **Unconvincing evaluation**: There are no clues about the effectiveness of proposed novel compound metrics PEM and PDEM. It would be much more convincing to conduct empirical studies to prove that models achieving better PEM/PDEM demonstrate better ability on addressing some concerned issues.\n3. **Missing training details**: Selected baselines are not designed for precipitation data. It is necessary to (at least slightly) modify and tune the models for fair comparison. However, there is no information about these details except for a single statement \"we also adjust the hype parameters of these models for better performance\" in Supl. Sec.5.2.\n4. **Formatting errors**: There are some language errors like \"e.t.c.\" $\\rightarrow$ \"etc.\", and misleading notations in mathematical expressions such as missing brackets in Eqn.1, missing description of \"L\", \"H\", \"T\" in line 250. Scores of 6 commonly used metrics in Table 1 should also be highlighted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept