RainNet: A Large-Scale Imagery Dataset for Spatial Precipitation Downscaling

Contemporary deep learning frameworks have been applied to solve meteorolog-1 ical problems ( e.g. , front detection , synthetic radar generation , precipitation now-2 casting , e.t.c . ) and have achieved highly promising results . Spatial precipitation3 downscaling is one of the most important meteorological problems . However,4 the lack of a well-organized and annotated large-scale dataset hinders the training5 and verification of more effective and advancing deep-learning models for precip-6 itation downscaling . To alleviate these obstacles , we present the first large-scale7 spatial precipitation downscaling dataset named RainNet , which contains more8 than 62 , 400 pairs of high-quality low/high-resolution precipitation maps for over9 17 years , ready to help the evolution of deep models in precipitation downscal-10 ing . Specifically , the precipitation maps carefully collected in RainNet cover var-11 ious meteorological phenomena ( e.g. , hurricane , squall , e.t.c . ) , which is of great12 help to improve the model generalization ability . In addition , the map pairs in13 RainNet are organized in the form of image sequences ( 720 maps per month or14 1 map/hour ) , showing complex physical properties , e.g. , temporal misalignment,15 temporal sparse , and fluid properties . Two machine-learning-oriented metrics are16 specifically introduced to evaluate or verify the comprehensive performance of the17 trained model , ( e.g. , prediction maps reconstruction accuracy ) . To illustrate the18 applications of RainNet , 14 state-of-the-art models , including deep models and19 traditional approaches , are evaluated . To fully explore potential downscaling so-20 lutions , we propose an implicit physical estimation framework to learn the above21 characteristics . Extensive experiments demonstrate that the value of RainNet in22 training and evaluating downscaling models.23 1 INTRODUCTION24 Deep learning has made an enormous breakthrough in the field of computer vision , which is ex-25 tremely good at extracting valuable knowledge from numerous amounts of data . In recent years,26 with computer science development , a deluge of Earth system data is continuously being obtained,27 coming from sensors all over the earth and even in space . These ever-increasing massive amounts of28 data with different sources and structures challenge the geoscience community , which lacks practi-29 cal approaches to understand and further utilize the raw data ( Reichstein et al . ( 2019 ) ) . Specifically,30 several preliminary works ( Groenke et al . ( 2020 ) ; White et al . ( 2019 ) ; He et al . ( 2016 ) ; Ravuri et al.31 ( 2021 ) ; Angell & Sheldon ( 2018 ) ; Veillette et al . ( 2020 ) ) try to introduce machine learning and deep32 learning frameworks to solve meteorological problems , e.g. , spatial precipitation downscaling.33 In this paper , we focus on the spatial precipitation downscaling task . Spatial precipitation down-34 scaling is a procedure to infer high-resolution meteorological information from low-resolution vari-35 ables , which is one of the most important upstream components for meteorological task ( Bauer et al.36 ( 2015 ) ) . The precision of weather and climate prediction is highly dependent on the resolution and37 reliability of the initial environmental input variables , and spatial precipitation downscaling is the38 most promising solution . The improvement of the weather/climate forecast and Geo-data quality39 saves tremendous money and lives ; with the fiscal year 2020 budget over $ 1 billion , NSF funds40 thousands of colleges in the U.S. to research on these topics ( NSF ( 2020 ) ) .41 Unfortunately , there are looming issues hinders the research of spatial precipitation downscaling42 in the machine learning community : 1 ) . Lack of ” machine-learning ready ” datasets . The existing43 machine-learning-based downscaling methods are only applied to ideal retrospective problems and44 verified on simulated datasets ( e.g. , mapping bicubic of precipitation generated by weather fore-45 cast model to original data ( Berrisford et al . ( 2011 ) ) ) , which significantly weakens the credibility46 of the feasibility , practicability , and effectiveness of the methods . It is worth mentioning that the47 data obtained by the simulated degradation methods ( e.g. , bicubic ) is completely different from the48 real data usually collected by two measurement systems ( e.g. , satellite and radar ) with different49 precision . The lack of a well-organized and annotated large-scale dataset hinders the training and50 verification of more effective and complex deep-learning models for precipitation downscaling . 2 ) .51 Lack of tailored metrics to evaluate machine-learning-based frameworks . Unlike deep learning ( DL ) 52 and machine learning ( ML ) communities , scientists in meteorology usually employ maps/charts to53 assessing downscaling models case by case based on domain knowledge ( He et al . ( 2016 ) ; Walton54 et al . ( 2020 ) ) , which hinders the application of Rainnet in DL/ML communities . For example , ( He55 et al . ( 2016 ) ) use log-semivariance ( spatial metrics for local precipitation ) , quantile-quantile maps56 to analyzing the maps . 3 ) . an efficient downscaling deep-learning framework should be established.57 Contrary to image data , this real precipitation dataset covers various types of real meteorological58 phenomena ( e.g. , Hurricane , Squall , e.t.c . ) , and shows the physical characters ( e.g. , temporal mis-59 alignment , temporal sparse and fluid properties , e.t.c . ) that challenge the downscaling algorithms.60 Traditional computationally dense physics-driven downscaling methods are powerless to handle the61 increasing meteorological data size and flexible to multiple data sources.62 To alleviate these obstacles , we propose the first large-scale spatial precipitation downscaling dataset63 named RainNet , which contains more than 62 , 400 pairs of high-quality low/high-resolution precip-64 itation maps for over 17 years , ready to help the evolution of deep models in spatial precipitation65 downscaling . The proposed dataset covers more than 9 million square kilometers of land area , which66 contains both wet and dry seasons and diverse meteorological phenomena . To facilitate DL/ML and67 other researchers to use RainNet , we introduce 6 most concerning indices to evaluate downscaling68 models : mesoscale peak precipitation error ( MPPE ) , heavy rain region error ( HRRE ) , cumulative69 precipitation mean square error ( CPMSE ) , cluster mean distance ( CMD ) , heavy rain transition speed70 ( HRTS ) and average miss moving degree ( AMMD ) . In order to further simplify the application of in-71 dices , we abstract them into two weighted and summed metrics : Precipitation Error Measure ( PEM ) 72 and Precipitation Dynamics Error Measure ( PDEM ) . Unlike video super-resolution , the motion of73 the precipitation region is non-rigid ( i.e. , fluid ) , while video super-resolution mainly concerns rigid74 body motion estimation . To fully explore how to alleviate the mentioned predicament , we propose75 an implicit dynamics estimation driven downscaling deep learning model . Our model hierarchi-76 cally aligns adjacent precipitation maps , that is , implicit motion estimation , which is very simple77 but exhibits highly competitive performance . Based on meteorological science , we also proved that78 the dataset we constructed contained the full information people may need to recover the higher79 resolution observations from lower resolution ones.80 The main contributions of this paper are:81 • To the best of our knowledge , we present the first REAL ( non-simulated ) Large-Scale Spa-82 tial Precipitation Downscaling Dataset for deep learning ; 83 • We introduce 2 simple metrics to evaluate the downscaling models ; 84 • We propose a downscaling model with strong competitiveness . We evaluate 14 competitive85 potential solutions on the proposed dataset , and analyze the feasibility and effectiveness of86 these solutions.87 2 BACKGROUND88 At the beginning of the 19th century , geoscientists recognized that predicting the state of the atmo-89 sphere could be treated as an initial value problem of mathematical physics , wherein future weather90 is determined by integrating the governing partial differential equations , starting from the observed91 current weather . Today , this paradigm translates into solving a system of nonlinear differential92 equations at about half a billion points per time step and accounting for dynamic , thermodynamic,93 radiative , and chemical processes working on scales from hundreds of meters to thousands of kilo-94 meters and from seconds to weeks ( Bauer et al . ( 2015 ) ) . The Navier–Stokes and mass continuity95 equations ( including the effect of the Earth ’ s rotation ) , together with the first law of thermodynamics96 and the ideal gas law , represent the full set of prognostic equations in the atmosphere , describing the97 change in space and time of wind , pressure , density and temperature is described ( formulas given in98 supplementary ) ( Bauer et al . ( 2015 ) ) . These equations have to be solved numerically using spatial99 and temporal discretization because of the mathematical intractability of obtaining analytical solu-100 tions , and this approximation creates a distinction between so-called resolved and unresolved scales101 of motion.102 2.1 SPATIAL DOWNSCALING OF PRECIPITATION103 The global weather forecast model , treated as a computational problem , relying on high-quality104 initial data input . The error of weather forecast would increase exponentially over time from this105 initial error of input dataset . Downscaling is one of the most important approaches to improve the106 initial input quality . Precipitation is one of the essential atmospheric variables that are related to daily107 life . It could easily be observed , by all means , e.g. , gauge station , radar , and satellites . Applying108 downscaling methods to precipitation and creating high-resolution rainfall is far more meaningful109 than deriving other variables , while it is the most proper initial task to test deep learning ’ s power110 in geo-science . The traditional downscaling methods can be separated into dynamic and statistical111 downscaling.112 Dynamic downscaling treats the downscaling as an optimization problem constraint on the physical113 laws . The dynamic downscaling methods find the most likely precipitation over space and time114 under the pre-defined physical law . It usually takes over 6 hours to downscale a 6-hour precipitation115 scenario globally on supercomputers ( Courtier et al . ( 1994 ) ) . As the dynamic downscaling relying116 on pre-defined known macroscopic physics , a more flexible weather downscaling framework that117 could easily blend different sources of observations and show the ability to describe more complex118 physical phenomena on different scales is desperately in need.119 Statistical downscaling is trying to speed up the dynamic downscaling process . The input of statisti-120 cal downscaling is usually dynamic model results or two different observation datasets on different121 scales . However , due to the quality of statistical downscaling results , people rarely apply statistical122 downscaling to weather forecasts . These methods are currently applied in the tasks not requir-123 ing high data quality but more qualitative understanding , e.g. , climate projection , which forecasts124 the weather for hundreds of years on coarse grids and using statistical downscaling to get detailed125 knowledge of medium-scale future climate system.126 3 RAINNET : SPATIAL PRECIPITATION DOWNSCALING IMAGERY DATASET127 3.1 DATA COLLECTION AND PROCESSING128 To build up a standard realistic ( non-simulated ) downscaling dataset for computer vision , we129 selected the eastern coast of the United States , which covers a large region ( 7 million km2 ; 130 105◦ ∼ 65◦W , 25◦ ∼ 50◦N , GNU Free Documentation License 1.2 ) and has a 20-year high-quality131 precipitation observations . We collected two precipitation data sources from National Stage IV QPE132 Product ( StageIV ( Nelson et al . ( 2016 ) ) ; high resolution at 0.04◦ ( approximately 4km ) , GNU Free133 Documentation License 1.2 ) and North American Land Data Assimilation System ( NLDAS ( Xia134 et al . ( 2012 ) ) ; low resolution at 0.125◦ ( approximately 13km ) ) . StageIV is mosaicked into a na-135 tional product at National Centers for Environmental Prediction ( NCEP ) , from the regional hourly/6-136 hourly multi-sensor ( radar+gauges ) precipitation analyses ( MPEs ) produced by the 12 River Fore-137 cast Centers over the continental United States with some manual quality control done at the River138 Forecast Centers ( RFCs ) . NLDAS is constructed quality-controlled , spatially-and-temporally con-139 sistent datasets from the gauges and remote sensors to support modeling activities . Both products140 are hourly updated and both available from 2002 to the current age.141 In our dataset , we further selected the eastern coast region for rain season ( July ∼ November,142 covering hurricane season ; hurricanes pour over 10 % annual rainfall in less than 10 days ) . We143 matched the coordinate system to the lat-lon system for both products and further labeled all the144 hurricane periods happening in the last 17 years . These heavy rain events are the largest challenge145 for weather forecasting and downscaling products . As heavy rain could stimulus a wide-spreading146 flood , which threatening local lives and arousing public evacuation . If people underestimate the147 rainfall , a potential flood would be underrated ; while over-estimating the rainfall would lead to148 unnecessary evacuation orders and flood protection , which is also costly.149 3.2 DATASET STATISTICS150 At the time of this work , we have collected and processed precipitation data for the rainy season151 for 17 years from 2002 to 2018 . One precipitation map pair per hour , 24 precipitation map pairs152 per day . In detail , we have collected 85 months or 62424 hours , totaling 62424 pairs of high-153 resolution and low-resolution precipitation maps . The size of the high-resolution precipitation map154 is 624 × 999 , and the size of the low-resolution is 208 × 333 . Various meteorological phenomena155 and precipitation conditions ( e.g. , hurricanes , squall lines , e.t.c . ) are covered in these data . The156 precipitation map pairs in RainNet are stored in HDF5 files that make up 360 GB of disk space . We157 select 2 typical meteorological phenomena and visualize them in Fig . 1 . Our data is collected from158 satellites , radars , gauge stations , e.t.c. , which covers the inherent working characteristics of different159 meteorological measurement systems . Compared with traditional methods that generate data with160 different resolutions through physical model simulation , our dataset is of great help for deep models161 to learn real meteorological laws.162 3.3 DATASET ANALYSIS163 In order to help design a more appropriate and effective precipitation downscaling model , we have164 explored the property of the dataset in depth . As mentioned above , our dataset is collected from mul-165 tiple sensor sources ( e.g. , satellite , weather radar , e.t.c . ) , which makes the data show a certain extent166 of misalignment . Our efforts here are not able to vanquish the misalignment . This is an intrinsic167 problem brought by the fusion of multi-sensor meteorological data . Limited by observation meth-168 ods ( e.g. , satellites can only collect data when they fly over the observation area ) , meteorological169 data is usually temporal sparse , e.g. , in our dataset , the sampling interval between two precipitation170 maps is one hour . The temporal sparse leads to serious difficulties in the utilization of precipitation171 sequences . Additionally , the movement of the precipitation position is directly related to the cloud.172 It is a fluid movement process that is completely different from the rigid body movement concerned173 in Super-Resolution . At the same time , the cloud will grow or dissipate in the process of flowing174 and even form new clouds , which further complicates the process . In the nutshell , although existed175 SR is a potential solution for downscaling , there is a big difference between the two . Especially,176 the three characteristics of downscaling mentioned above : temporal misalignment , temporal sparse,177 fluid properties , which make the dynamic estimation of precipitation more challenging.178 4 EVALUATION METRICS179 Due to the difference between downscaling and traditional figure super-resolution , the metrics that180 work well under SR tasks may not be sufficient for precipitation downscaling . By gathering the181 metrics from the meteorologic literature ( the literature includes are Zhang & Yang ( 2004 ) ; Maraun182 et al . ( 2015 ) ; Ekström ( 2016 ) ; He et al . ( 2016 ) ; Pryor & Schoof ( 2020 ) ; Wootten et al . ( 2020 ) ) ,183 we select and rename 6 most common metrics ( a metrics may have multiple names in different184 literature ) to reflect the downscaling quality : mesoscale peak precipitation error ( MPPE ) , cumulative185 precipitation mean square error ( CPMSE ) , heavy rain region error ( HRRE ) , cluster mean distance186 ( CMD ) , heavy rain transition speed ( HRTS ) and average miss moving degree ( AMMD ) .These 6187 metrics can be separated as reconstruction metrics : MPPE , HRRE , CPMSE , AMMD , and dynamic188 metrics : HRTS and CMD.189 The MPPE ( mm/hour ) is calculated as the difference of top quantile between the generated/real190 rainfall dataset which considering both spatial and temporal property of mesoscale meteorological191 systems , e.g. , hurricane , squall . This metric is used in most of these papers ( for example Zhang192 & Yang ( 2004 ) ; Maraun et al . ( 2015 ) ; Ekström ( 2016 ) ; He et al . ( 2016 ) ; Pryor & Schoof ( 2020 ) ; 193 Wootten et al . ( 2020 ) suggest the quantile analysis to evaluate the downscaling quality ) .194 The CPMSE ( mm2/hour2 ) measures the cumulative rainfall difference on each pixel over the time-195 axis of the test set , which shows the spatial reconstruction property . Similar metrics are used in196 Zhang & Yang ( 2004 ) ; Maraun et al . ( 2015 ) ; Wootten et al . ( 2020 ) calculated as the pixel level197 difference of monthly rainfall and used in He et al . ( 2016 ) as a pixel level difference of cumulative198 rainfall with different length of record.199 The HRRE ( km2 ) measures the difference of heavy rain coverage on each time slide between gen-200 erated and labeled test set , which shows the temporal reconstruction ability of the models . The201 AMMD ( radian ) measures the average angle difference between main rainfall clusters . Similar202 metrics are used in Zhang & Yang ( 2004 ) ; Maraun et al . ( 2015 ) ; Wootten et al . ( 2020 ) as rainfall203 coverage of a indefinite number precipitation level and used in He et al . ( 2016 ) ; Pryor & Schoof204 ( 2020 ) as a continuous spatial analysis.205 As a single variable dataset , it is hard to evaluate the ability of different models to capture the206 precipitation dynamics when temporal information is not included ( a multi-variable dataset may207 have wind speed , a typical variable representing dynamics , included ) . So here we introduce the208 first-order temporal and spatial variables to evaluate the dynamical property of downscaling results.209 Similar approaches are suggested in Maraun et al . ( 2015 ) ; Ekström ( 2016 ) ; Pryor & Schoof ( 2020 ) .210 The CMD ( km ) physically compares the location difference of the main rainfall systems between211 the generated and labeled test set , which could be also understand as the RMSE of the first order212 derivative of precipitation data on spatial directions.The HRTS ( km/hour ) measures the difference213 between the main rainfall system moving speed between the generated and labeled test set which214 shows the ability for models to capture the dynamic property , which could be also understand as the215 RMSE of the first order derivative of precipitation data on temporal direction.Similar metrics are216 suggested in Maraun et al . ( 2015 ) ; Ekström ( 2016 ) ; Pryor & Schoof ( 2020 ) as the auto-regression217 analysis and the differential analysis.218 More details about the metrics and their equations are given in supplementary materials . One met-219 rics group ( MPPE , HRRE , CPMSE , AMMD ) mainly measures the rainfall deviation between the220 generated precipitation maps and GT . The other group ( HRTS and CMD ) mainly measures the221 dynamic deviation of generated precipitation maps . In order to further simplify the application222 of indices , we abstract them into two weighted and summed metrics : Precipitation Error Mea-223 sure ( PEM ) and Precipitation Dynamics Error Measure ( PDEM ) . We first align the dimensions224 of these two groups of metrics respectively . The first group of metrics ( MPPE , HRRE , CPMSE,225 AMMD ) is normalized , weighted and summed to get the precipitation error measure ( PEM ) . Ac-226 cording to Gupta et al . ( 1999 ) , all the metrics are transferred to Percent Bias ( PBIAS ) to be suit-227 able for metrics weighting . The original definition of PBIAS is the bias divided by observation , as228 PBIAS = |Qmodel − Qobs|/|Qobs| . Here we rewrite the original metrics to PBIAS by dividing229 the metrics with annual mean observations of the original variables ( AMO ) , as PBIASPEMi =230 |MetricsPEMi |/|AMOPEMi | , MetricsPEMi = { MPPE , HRRE , CPMSE , AMMD } . In our231 dataset , AMOPEMMPPE = 64 , AMO PEM HRREM = 533 , AMO PEM CPMSE = 0.64 , AMO PEM AMMD = 332,232 AMOPEMHRTS = 15 , AMO PEM CMD = 26 . The metrics then are ensembled to a single metric233 ( PEM ) with equal weight , as PEM = ∑ i 0.25 · PBIASPEMi . Following the same procedure,234 we then ensemble the second group of dynamic metrics ( HRTS and CMD ) to a single metrics235 PDEM = ∑ i 0.5 · PBIASPDEMi .236 We also include the most common used metrics RMSE as one single metrics in our metrics list.237 RMSE could evaluate both reconstruction and dynamic property of the downscaling result.238 5 APPLICATIONS OF RAINNET IN SPATIAL PRECIPITATION DOWNSCALING239 As a potential solution , Super-Resolution ( SR ) frameworks are generally divided into the Single-240 Image Super-Resolution ( SISR ) and the Video Super-Resolution ( VSR ) . Video Super-Resolution is241 able to leverage multi-frame information to restore images , which better matches the nature of down-242 scaling . We will demonstrate this judgment in Sec . 6.1 . The VSR pipeline usually contains three243 components : deblurring , inter-frame alignment , and super-resolution . Deblurring and inter-frame244 alignment are implemented by the motion estimation module . There are four motion estimation245 frameworks : 1 ) . RNN based ( Keys ( 1981 ) ; Tao et al . ( 2017 ) ; Huang et al . ( 2015 ) ; Haris et al.246 ( 2019 ) ) ; 2 ) . Optical Flow ( Xue et al . ( 2019 ) ) ; 3 ) . Deformable Convolution based ( Tian et al . ( 2020 ) ; 247 Xiang et al . ( 2020 ) ; Wang et al . ( 2019 ) ) ; 4 ) . Temporal Concatenation ( Jo et al . ( 2018 ) ; Caballero248 et al . ( 2017 ) ; Liao et al . ( 2015 ) ) . In fact , there is another motion estimation scheme proposed for249 the first time in the noise reduction task ( Tassano et al . ( 2020 ) ) , which achieves an excellent video250 noise reduction performance . Inspired by ( Tassano et al . ( 2020 ) ) , we design an implicit dynamics251 estimation model for the spatial precipitation downscaling . It is worth mentioning that our proposed252 model and the above four frameworks together form a relatively complete candidate set of dynamic253 estimation solutions.254 Proposed Framework . As shown in Fig . 2 , our framework consists of two components : Implicit255 dynamic estimation module and downscaling Backbone . These two parts are trained jointly . Suppose256 there areN adjacent low-resolution precipitation maps { IL T−N−12 , .. , ILT , ... , I L T+N−12 } . The task is to257 reconstruct the high-resolution precipitation map IHT of I L T . The implicit dynamic estimation module258 is composed of multiple vanilla networks A = { A1 , ... , AN−2 } ( N = 5 in this paper ) sharing259 weights . Each vanilla network receives three adjacent frames as input , outputs , and intermediate260 results . The intermediate result can be considered as a frame with implicit dynamic alignment . We261 concatenate all the intermediate frames as the input of the next module . The specific structure of262 the vanilla network can be found in the supplementary materials . The main task of the downscaling263 backbone is to restore the high-resolution precipitation map IHT based on the aligned intermediate264 frames . In order to make full use of multi-scale information , we use multiple Residual-in-Residual265 Dense Blocks ( Wang et al . ( 2018 ) ) in the network . We employ the interpolation+convolution ( Odena266 et al . ( 2016 ) ) as the up-sampling operator to reduce the checkerboard artifacts . After processing by267 downscaling backbone we get the final estimated HR map ÎHT .268 Model objective . The downscaling task is essentially to restore high-resolution precipitation maps.269 We learn from the super-resolution task and also applyL1 and perceptual loss ( Johnson et al . ( 2016 ) ) 270 as the training loss of our model . The model objective is shown below:271 L ( ̂IHT , IHT ) =‖ ÎHT − IHT ‖1 +λ ‖ φ ( ̂IHT ) − φ ( IHT ) ‖2 , ( 1 ) where φ denotes the pre-trained VGG19 network ( Simonyan & Zisserman ( 2015 ) ) , we select the272 Relu5 − 4 ( without the activator ( Wang et al . ( 2018 ) ) ) as the output layer . λ is the coefficient to273 balance the loss terms . λ = 20 in our framework.274 6 EXPERIMENTAL EVALUATION275 We conduct spatial precipitation downscaling experiments to illustrate the application of our276 proposed RainNet and evaluate the effectiveness of the benchmark downscaling frameworks . Fol-277 lowing the mainstream evaluation protocol of DL/ML communities , cross-validation is employed.278 In detail , we divide the dataset into 17 parts ( 2002.7∼2002.11 , 2003.7∼2003.11 , 2004.7∼2004.11,279 2005.7∼2005.11 , 2006.7∼2006.11 , 2007.7∼2007.11 , 2008.7∼2008.11 , 2009.7∼2009.11,280 2010.7∼2010.11 , 2011.7∼2011.11 , 2012.7∼2012.11 , 2013.7∼2013.11 , 2014.7∼2014.11,281 2015.7∼2015.11 , 2016.7∼2016.11 , 2017.7∼2017.11 , 2018.7∼2018.11 ) by year , and sequentially282 employ each year as the test set and the remaining 16 years as the training set , that is , 17-fold283 cross-validation . All models maintain the same training settings and hyperparameters during the284 training phase . These data cover various complicated precipitation situations such as hurricanes,285 squall lines , different levels of rain , and sunny days . It is sufficient to select the rainy season of286 the year as the test set from the perspective of meteorology , as the climate of one area is normally287 stable.288 6.1 BASELINES289 The SISR/VSR and the spatial precipitation down-290 scaling are similar to some extent , so we argue that291 the SR models can be applied to the task as the292 benchmark models . The input of SISR is a single293 image , and the model infers a high-resolution image294 from it . Its main focus is to generate high-quality295 texture details to achieve pleasing visual effects . In296 contrast , VSR models input multiple frames of im-297 ages ( e.g. , 3 frames , 5 frames , e.t.c. ) . In our experi-298 ments , we employ 5 frames . The core idea of VSR299 models is to increase the resolution by complement-300 ing texture information between different frames . It301 is worth mentioning that VSR models generally are302 equipped with a motion estimation module to alle-303 viate the challenge of object motion to inter-frame304 information registration.305 We evaluated 7 state-of-the-art SISR frameworks306 ( i.e. , Bicubic ( Keys ( 1981 ) ) , SRCNN1 ( Dong307 et al . ( 2016 ) ) , SRGAN2 ( Ledig et al . ( 2017 ) ) ,308 1https : //github.com/yjn870/SRCNN-pytorch 2https : //github.com/leftthomas/SRGAN EDSR3 ( Lim et al . ( 2017 ) ) , ESRGAN4 ( Wang et al . ( 2018 ) ) , DBPN5 ( Haris et al . ( 2018 ) ) ,309 RCAN6 ( Zhang et al . ( 2018 ) ) and 5 VSR frameworks ( i.e. , SRGAN-V , EDSR-V , ESRGAN-V,310 RBPN7 ( Haris et al . ( 2019 ) ) , EDVR8 ( Wang et al . ( 2019 ) ) , of which 3 VSR methods ( i.e. , SRGAN-311 V , EDSR-V , ESRGAN-V ) are modified from SISR . In particular , we build SRGAN-V , EDSR-V and312 ESRGAN-V by concatenating multiple frames of precipitation maps as the input of the model . In313 addition , we also evaluated the traditional statistics method Kriging ( Stein ( 2012 ) ) , which is widely314 applied in weather forecasting . The mentioned 8 metrics are used to quantitatively evaluate the315 performance of these SR models and our method . Further , we select some disastrous weather as316 samples for qualitative analysis to test the model ’ s ability to learn the dynamic properties of the317 weather system . And we employ the implementation of Pytorch for Bicubic . We use 4 NVIDIA318 2080 Ti GPUs for training . We train all models with following setting . The batch size is set as 24.319 Precipitation maps are random crop into 64× 64 . We employ the Adam optimizer , beta1 is 0.9 , and320 beta2 is 0.99 . The initial learning rate is 0.001 , which is reduced to 1/10 every 50 epochs , and a total321 of 200 epochs are trained . We evaluate benchmark frameworks with 17-fold cross-validation . The322 downscaling performances are shown in Tab . 1 . We divide the indicators mentioned above into two323 groups . PDEM measures the model ’ s ability to learn the dynamics of precipitation . PEM illustrates324 the model ’ s ability to reconstruct precipitation.325 From Tab . 1 , we can learn that the overall performance of the VSR methods are better than SISR326 models , which shows that the dynamic properties mentioned above are extremely important for the327 downscaling model . Furthermore , it can be seen from Fig . 3 that the SISR method is clustered in the328 upper right corner of the scatter plot , and the VSR method is concentrated in the lower-left corner,329 which further shows that the dynamic properties of the VSR methods are overall better than the SISR330 methods . In addition , our method achieves the 1st best performance in RMSE , PDE , and achieve the331 second-best performance on PEM . The score shows that the implicit dynamic estimation framework332 3https : //github.com/sanghyun-son/EDSR-PyTorch 4https : //github.com/xinntao/ESRGAN 5https : //github.com/alterzero/DBPN-Pytorch 6https : //github.com/yulunzhang/RCAN 7https : //github.com/alterzero/RBPN-PyTorch 8https : //github.com/xinntao/EDVR used is feasible and effective . It is worth mentioning that the traditional downscaling method Kriging333 performs better than many deep learning models ( e.g. , SRGAN , ESRGAN ) 334 6.1.1 QUALITATIVE ANALYSIS335 We visualized the tropical cyclone precipitation map of the 166th hour ( 6th ) in September 2010336 and the high-resolution precipitation map generated by different methods . As shown in Fig . 4 , the337 best perceptual effects are generated by EDVR and Our framework . Zooming in the result image,338 the precipitation maps generated by SRGAN and EDSR present obvious checkerboard artifacts.339 The reason for the checkerboard artifacts should be the relatively simple and sparse texture pattern340 in precipitation maps . The results generated by Bicubic , RCAN , Kriging , and SRCNN are over-341 smooth . DBPN even can not reconstruct the eye of the hurricane . Especially , the result generated by342 Kriging is as fuzzy as the input LR precipitation map . In conclusion , the visual effects generated343 by the VSR methods are generally better than the SISR methods and the traditional method . From344 the perspective of quantitative and qualitative analysis , the dynamics estimation framework is very345 critical for downscaling.346 7 CONCLUSION347 In this paper , we built the first large-scale real precipitation downscaling dataset for the deep learning348 community . This dataset has 62424 pairs of HR and LR precipitation maps in total . We believe this349 dataset will further accelerate the research on precipitation downscaling . Furthermore , we analyze350 the problem in-depth and put forward three key challenges : temporal misalignment , temporal sparse,351 fluid properties . In addition , we propose an implicit dynamic estimation model to alleviate the above352 challenges . At the same time , we evaluated the mainstream SISR and VSR models and found that353 none of these models can solve RainNet ’ s problems well . Therefore , the downscaling task on this354 dataset is still very challenging.355 This work still remains several open problems . Currently , the data domain of this research is limited356 to the eastern U.S . In future research , we would enlarge the dataset to a larger domain . The dataset357 is only a single variable now . In future research , we may include more variables , e.g . temperature358 and wind speed.359 REFERENCES360 Rico Angell and Daniel R. Sheldon . Inferring latent velocities from weather radar data using gaus-361 sian processes . In Advances in Neural Information Processing Systems 31 : Annual Conference362 on Neural Information Processing Systems 2018 , NeurIPS 2018 , December 3-8 , 2018 , Montréal,363 Canada , pp . 8998–9007 , 2018.364 Peter Bauer , Alan Thorpe , and Gilbert Brunet . The quiet revolution of numerical weather prediction.365 Nature , 2015.366 P. Berrisford , D.P . Dee , P. Poli , R. Brugge , Mark Fielding , Manuel Fuentes , P.W . Kållberg,367 S. Kobayashi , S. Uppala , and Adrian Simmons . The era-interim archive version 2.0 . 2011.368 Jose Caballero , Christian Ledig , Andrew P. Aitken , Alejandro Acosta , Johannes Totz , Zehan Wang,369 and Wenzhe Shi . Real-time video super-resolution with spatio-temporal networks and motion370 compensation . In 2017 IEEE Conference on Computer Vision and Pattern Recognition , CVPR371 2017 , Honolulu , HI , USA , July 21-26 , 2017 . IEEE Computer Society , 2017.372 PHILIPPE Courtier , J-N Thépaut , and Anthony Hollingsworth . A strategy for operational imple-373 mentation of 4d-var , using an incremental approach . Quarterly Journal of the Royal Meteorolog-374 ical Society , 1994.375 Chao Dong , Chen Change Loy , Kaiming He , and Xiaoou Tang . Image super-resolution using deep376 convolutional networks . IEEE Trans . Pattern Anal . Mach . Intell. , 2016.377 Marie Ekström . Metrics to identify meaningful downscaling skill in wrf simulations of intense378 rainfall events . Environmental Modelling & Software , 79:267–284 , 2016.379 Brian Groenke , Luke Madaus , and Claire Monteleoni . Climalign : Unsupervised statistical down-380 scaling of climate variables via normalizing flows . CoRR , 2020.381 Hoshin Vijai Gupta , Soroosh Sorooshian , and Patrice Ogou Yapo . Status of automatic calibration382 for hydrologic models : Comparison with multilevel expert calibration . Journal of hydrologic383 engineering , 4 ( 2 ) :135–143 , 1999.384 Muhammad Haris , Gregory Shakhnarovich , and Norimichi Ukita . Deep back-projection networks385 for super-resolution . In 2018 IEEE Conference on Computer Vision and Pattern Recognition,386 CVPR 2018 , Salt Lake City , UT , USA , June 18-22 , 2018 . IEEE Computer Society , 2018.387 Muhammad Haris , Gregory Shakhnarovich , and Norimichi Ukita . Recurrent back-projection net-388 work for video super-resolution . In IEEE Conference on Computer Vision and Pattern Recogni-389 tion , CVPR 2019 , Long Beach , CA , USA , June 16-20 , 2019 . Computer Vision Foundation / IEEE,390 2019.391 Xiaogang He , Nathaniel W Chaney , Marc Schleiss , and Justin Sheffield . Spatial downscaling of392 precipitation using adaptable random forests . Water resources research , 2016.393 Yan Huang , Wei Wang , and Liang Wang . Bidirectional recurrent convolutional networks for multi-394 frame super-resolution . In Advances in Neural Information Processing Systems , 2015.395 Younghyun Jo , Seoung Wug Oh , Jaeyeon Kang , and Seon Joo Kim . Deep video super-resolution396 network using dynamic upsampling filters without explicit motion compensation . In 2018 IEEE397 Conference on Computer Vision and Pattern Recognition , CVPR 2018 , Salt Lake City , UT , USA,398 June 18-22 , 2018 . IEEE Computer Society , 2018.399 Justin Johnson , Alexandre Alahi , and Li Fei-Fei . Perceptual losses for real-time style transfer and400 super-resolution . In Computer Vision - ECCV 2016 - 14th European Conference , Amsterdam , The401 Netherlands , October 11-14 , 2016 , Proceedings , Part II , 2016.402 Robert Keys . Cubic convolution interpolation for digital image processing . IEEE transactions on403 acoustics , speech , and signal processing , 1981.404 Christian Ledig , Lucas Theis , Ferenc Huszar , Jose Caballero , Andrew Cunningham , Alejandro405 Acosta , Andrew P. Aitken , Alykhan Tejani , Johannes Totz , Zehan Wang , and Wenzhe Shi . Photo-406 realistic single image super-resolution using a generative adversarial network . In 2017 IEEE407 Conference on Computer Vision and Pattern Recognition , CVPR 2017 , Honolulu , HI , USA , July408 21-26 , 2017 . IEEE Computer Society , 2017.409 Renjie Liao , Xin Tao , Ruiyu Li , Ziyang Ma , and Jiaya Jia . Video super-resolution via deep draft-410 ensemble learning . In 2015 IEEE International Conference on Computer Vision , ICCV 2015,411 Santiago , Chile , December 7-13 , 2015 . IEEE Computer Society , 2015.412 Bee Lim , Sanghyun Son , Heewon Kim , Seungjun Nah , and Kyoung Mu Lee . Enhanced deep resid-413 ual networks for single image super-resolution . In 2017 IEEE Conference on Computer Vision414 and Pattern Recognition Workshops , CVPR Workshops 2017 , Honolulu , HI , USA , July 21-26,415 2017 . IEEE Computer Society , 2017.416 Douglas Maraun , Martin Widmann , José M Gutiérrez , Sven Kotlarski , Richard E Chandler , Elke417 Hertig , Joanna Wibig , Radan Huth , and Renate AI Wilcke . Value : A framework to validate418 downscaling approaches for climate change studies . Earth ’ s Future , 3 ( 1 ) :1–14 , 2015.419 Brian R. Nelson , Olivier P. Prat , D.-J . Seo , and Emad Habib . Assessment and Implications of420 NCEP Stage IV Quantitative Precipitation Estimates for Product Intercomparisons . Weather and421 Forecasting , 2016.422 NSF . Nsf geosciences directorate funding by institution type . AGI Report , 2020.423 Augustus Odena , Vincent Dumoulin , and Chris Olah . Deconvolution and checkerboard artifacts.424 Distill , 2016.425 SC Pryor and JT Schoof . Differential credibility assessment for statistical downscaling . Journal of426 Applied Meteorology and Climatology , 59 ( 8 ) :1333–1349 , 2020.427 Suman Ravuri , Karel Lenc , Matthew Willson , Dmitry Kangin , Remi Lam , Piotr Mirowski , Megan428 Fitzsimons , Maria Athanassiadou , Sheleem Kashem , Sam Madge , et al . Skillful precipitation429 nowcasting using deep generative models of radar . Nature , 2021.430 Markus Reichstein , Gustau Camps-Valls , Bjorn Stevens , Martin Jung , Joachim Denzler , Nuno Car-431 valhais , et al . Deep learning and process understanding for data-driven earth system science.432 Nature , 2019.433 Karen Simonyan and Andrew Zisserman . Very deep convolutional networks for large-scale image434 recognition . In Yoshua Bengio and Yann LeCun ( eds . ) , 3rd International Conference on Learning435 Representations , ICLR 2015 , San Diego , CA , USA , May 7-9 , 2015 , Conference Track Proceed-436 ings , 2015.437 Michael L Stein . Interpolation of spatial data : some theory for kriging . Springer Science & Business438 Media , 2012.439 Xin Tao , Hongyun Gao , Renjie Liao , Jue Wang , and Jiaya Jia . Detail-revealing deep video super-440 resolution . In IEEE International Conference on Computer Vision , ICCV 2017 , Venice , Italy,441 October 22-29 , 2017 . IEEE Computer Society , 2017.442 Matias Tassano , Julie Delon , and Thomas Veit . Fastdvdnet : Towards real-time deep video denois-443 ing without flow estimation . In 2020 IEEE/CVF Conference on Computer Vision and Pattern444 Recognition , CVPR 2020 , Seattle , WA , USA , June 13-19 , 2020 . IEEE , 2020.445 Yapeng Tian , Yulun Zhang , Yun Fu , and Chenliang Xu . TDAN : temporally-deformable alignment446 network for video super-resolution . In 2020 IEEE/CVF Conference on Computer Vision and447 Pattern Recognition , CVPR 2020 , Seattle , WA , USA , June 13-19 , 2020 . IEEE , 2020.448 Mark S. Veillette , Siddharth Samsi , and Christopher J. Mattioli . SEVIR : A storm event imagery449 dataset for deep learning applications in radar and satellite meteorology . In Advances in Neu-450 ral Information Processing Systems 33 : Annual Conference on Neural Information Processing451 Systems 2020 , NeurIPS 2020 , December 6-12 , 2020 , virtual , 2020.452 Daniel Walton , Neil Berg , David Pierce , Ed Maurer , Alex Hall , Yen-Heng Lin , Stefan Rahimi , and453 Dan Cayan . Understanding differences in california climate projections produced by dynamical454 and statistical downscaling . Journal of Geophysical Research : Atmospheres , 2020.455 Xintao Wang , Ke Yu , Shixiang Wu , Jinjin Gu , Yihao Liu , Chao Dong , Yu Qiao , and Chen456 Change Loy . Esrgan : Enhanced super-resolution generative adversarial networks . In Proceedings457 of the European Conference on Computer Vision ( ECCV ) , pp . 0–0 , 2018.458 Xintao Wang , Kelvin CK Chan , Ke Yu , Chao Dong , and Chen Change Loy . Edvr : Video restoration459 with enhanced deformable convolutional networks . In Proceedings of the IEEE Conference on460 Computer Vision and Pattern Recognition Workshops , 2019.461 BL White , A Singh , and A Albert . Downscaling numerical weather models with gans . AGUFM,462 2019.463 Adrienne M Wootten , Elias C Massoud , Agniv Sengupta , Duane E Waliser , and Huikyo Lee . The464 effect of statistical downscaling on the weighting of multi-model ensembles of precipitation . Cli-465 mate , 8 ( 12 ) :138 , 2020.466 Youlong Xia , Kenneth Mitchell , Michael Ek , Justin Sheffield , Brian Cosgrove , Eric Wood , Lifeng467 Luo , Charles Alonge , Helin Wei , Jesse Meng , Ben Livneh , Dennis Lettenmaier , Victor Koren,468 Qingyun Duan , Kingtse Mo , Yun Fan , and David Mocko . Continental-scale water and energy469 flux analysis and validation for the north american land data assimilation system project phase470 2 ( nldas-2 ) : 1. intercomparison and application of model products . Journal of Geophysical Re-471 search : Atmospheres , 2012.472 Xiaoyu Xiang , Yapeng Tian , Yulun Zhang , Yun Fu , Jan P. Allebach , and Chenliang Xu . Zooming473 slow-mo : Fast and accurate one-stage space-time video super-resolution . In 2020 IEEE/CVF474 Conference on Computer Vision and Pattern Recognition , CVPR 2020 , Seattle , WA , USA , June475 13-19 , 2020 . IEEE , 2020.476 Tianfan Xue , Baian Chen , Jiajun Wu , Donglai Wei , and William T. Freeman . Video enhancement477 with task-oriented flow . Int . J. Comput . Vis. , 2019.478 Xuebin Zhang and Feng Yang . Rclimdex ( 1.0 ) user manual . Climate Research Branch Environment479 Canada , 22 , 2004.480 Yulun Zhang , Kunpeng Li , Kai Li , Lichen Wang , Bineng Zhong , and Yun Fu . Image super-481 resolution using very deep residual channel attention networks . In Computer Vision - ECCV482 2018 - 15th European Conference , Munich , Germany , September 8-14 , 2018 , Proceedings , Part483 VII , Lecture Notes in Computer Science . Springer , 2018.484

Summary: This paper proposes a dataset called RainNet for studying precipitation downscaling. RainNet consists of ~62k pairs of low-res/high-res precipitation maps from various meteorological events over the east coast of the US over 17 years. This dataset was created from two existing products: NLDAS and the NCEP's Stage IV 4km gridded precipitation data. Further, the paper proposes two metrics, Precipitation Error Measure (PEM) and Precipitation Dynamics Error Measure (PDEM). Finally, the paper compares several superresolution methods from computer vision literature, traditional statistical approaches to downscaling, as well as the proposed architecture/method on the proposed dataset to form a set of benchmark results.