# Weather Radar Image Superresolution Using a Nonlocal Residual Network

Authors: Haoxuan Yuan; Qiangyu Zeng; Jianxin He
Journal: Journal of Mathematics
 (2021)

---

## Abstract

Accurate and high-resolution weather radar images reflecting detailed structure information of radar echo are vital for analysis and forecast of extreme weather. Typically, this is performed by using interpolation schemes, which only use several neighboring data values for computational approximation to get the estimated value regardless of the large-scale context feature of weather radar images. Inspired by the striking performance of the convolutional neural network (CNN) applied in feature extraction and nonlocal self-similarity of weather radar images, we proposed a nonlocal residual network (NLRN) on the basis of CNN. The proposed network mainly consists of several nonlocal residual blocks (NLRB), which combine short skip connection (SSC) and nonlocal operation to train the deep network and capture large-scale context information. In addition, long skip connection (LSC) added in the network avoids learning low-frequency information, making the network focus on high-level features. Extensive experiments of ×2 and ×4 super-resolution reconstruction demonstrate that NLRN achieves superior performance in terms of both quantitative evaluation metrics and visual quality, especially for the reconstruction of the edge and detailed information of the weather radar echo.


---

## Body

## 1. Introduction

Doppler weather radar with high temporal and spatial resolution e.g., China Next Generation Weather Radar (CINRAD) provides measurements with high temporal (approx. 6 minutes) and spatial (approx. 1 × 1 km) resolution and have been widely applied in operational research and forecasts on medium-scale and intense precipitation weather phenomena.However, single weather radar is susceptible to beam blocking, ground clutter, and reduced resolution at long distances due to beam broadening and averaging. As shown in Figure1, the beam width increases with the detection distance which leads to a loss of information on sudden changes in radar echoes such as velocity changes in tornadoes and mesocyclones, as well as information on extreme precipitation intensity and gradients when detection target occurs at a distance from the observing radar [1, 2]. It is, therefore, worthwhile to improve the resolution of weather radar data by upgrading the observation equipment and by postprocessing the observation data such as interpolation or superresolution reconstruction. Due to the long cycle and high cost of the first scheme, many scholars have conducted a lot of research on the second scheme.Figure 1
The effect of beam broadening.In terms of interpolation methods, Ruzanski and Chandrasekar [3] proposed a kernel-based Fourier interpolation method, which effectively improves the spatial and temporal resolution of weather radar by adding windows to screen the effective input data to be interpolated. Sharifi et al. [4] proposed a downscaling method based on spline interpolation to address the problem of too coarse spatial-temporal resolution when satellite precipitation estimates are applied to small areas, which effectively improves the resolution of precipitation data products while accurately capturing detailed precipitation patterns and information. Considering the non-Gaussian and locally coherent structure of weather radar reflectivity data in the wavelet domain, Kou et al. [5] proposed an interpolation method to improve the resolution of radar reflectivity data, which effectively use the hidden Markov tree (HMT) model as priority information to well capture the multiscale statistical characteristics of radar reflectivity data in small-scale intense precipitation condition.Inspired by the sparsity of weather radar precipitation images in the wavelet domain, dictionary-based sparse regularization methods have been applied to statistical down scaling, converting coarse observation data into more refined [6, 7]. Based on sparse regularization, Zhang et al. [8] further proposed a nonlocal self-similar sparse representation (NSSR) model, which effectively uses the nonlocal self-similarity of weather radar echo during the reconstruction process. Similarly, in the research article, Yuan et al. [9] proposed adaptive regularized sparse representation for weather radar echo super-resolution reconstruction. Based on dictionary learning, Xu et al. [10] proposed a downscale method to obtain more refined short-duration precipitation data.With the rapid progress of deep learning, the atmospheric research community has already taken advantage of the convolutional neural network (CNN)’s ability [11], such as the application of super resolution is related to the statistical downscaling of climate data. Vandal et al. [12] first applied CNN to climate data downscalling and improved the accuracy of precipitation data from 1° (100 km) to 1/8° (12.5 km) by stacking CNN-based super-resolution models. Cheng et al. [13] innovatively integrated the residual dense block (RDB) [14] with the Laplacian pyramid super-resolution network (LapSRN) [15] to exploit hierarchical features from all convolutional layers and generate a more refined climate image than Vandal’s method. Geiss and Hardin [16] combined the classic “up-net” [17] with the “dense network” [18] and proposed a new deep convolutional neural network to learn large-scale precipitation features in weather radar images. The reconstructed weather radar image is superior to the traditional interpolation method in terms of both objective evaluation metrics and visual quality. Stengel et al. [19] also used adversarial learning to significantly enhance the resolution of wind and solar data, which shows the notable performance of the generative adversarial network (GAN) in downscaling climate data.However, there is an issue existing in most CNN-based super-resolution models. These networks use local convolutional operations to extract features, which result in a relatively small receptive field size of the network and the inability to capture the large-scale context information of weather radar image, such as the recurrence features of small-scale organized precipitation within and across different storm environments.To address this issue, we proposed a nonlocal residual network (NLRN) on the basis of CNN, which increases the depth of the network and the efficiency in exploiting the large-scale context information of the weather radar image by applying residual learning and nonlocal operation.The main contributions of this work are as follows:(i)
According to the nonlocal self-similarity of weather radar images, this research endowed the network’s ability that allows them to fuse nonlocal information by stacking the nonlocal residual block (NLRB). In order to achieve a nonlocal operation, it embeds the nonlocal block (NLB).(ii)
The outstanding performance of residual learning has inspired us to introduce the long and short skip connections in the overall structure of the network and NLRB, respectively. It trains a deeper network, which effectively facilitates the flow of information and solves the gradient vanishing problems.(iii)
This study conducted loads of experiments on reflectivity data under different weather conditions. They justified that weather radar images reconstructed by NLRN achieved better quantitative results and visual quality by the proposed method than other SR methods mentioned in the experimental section.The remaining part of the article proceeds as follows: Section2 gives a brief introduction to the background that is related to the article. The framework and details of the proposed NLRN are described in Section 3. Section 4 presents the implementation details of NLRN. Several experimental results and discussions are presented to validate the effectiveness of the proposed NLRN in Section 5. Section 6 concludes the article.


## 2. Background

First, this section gives a concise overview of the SR methods that are related to the article. Second, we discuss the characteristic of a weather radar image. Finally, to facilitate the understanding of subsequent experimental parts by readers without a background in meteorology, we give a brief introduction to the hook echo.


### 2.1. Image Super-Resolution Methods

According to the means of implementation, the image super-resolution methods can be divided into three types: interpolation-based, reconstruction-based, and learning-based image super-resolution methods.


#### 2.1.1. Methods Based on Interpolation

The basic idea of classical interpolation methods such as nearest interpolation, bilinear interpolation, and bicubic interpolation [20] is to approximate the lost image information by using the basis function or interpolation kernel, which only exploits the neighboring information. These methods often result in blurred edges and loss of high-frequency detail while reducing computational complexity.


#### 2.1.2. Methods Based on Reconstruction

Reconstruction-base methods apply prior knowledge of image as constraint terms to the process of super-resolution reconstruction, which effectively solves the ill-posed problem of super-resolution reconstruction. Classical reconstruction-based methods are iterative back projection (IBP) [21], maximum a posterior estimation (MAP) [22], etc.


#### 2.1.3. Methods Based on Shallow Learning

Before the deep learning is applied to image super resolution, the most learning-based methods are proposed on the basis of sparse representation such as nonlocally centralized sparse representation (NCSR) [23] and structure-modulated sparse representation (SMSR) [24], which mainly includes dictionary learning and sparse coding.


#### 2.1.4. Methods Based on Deep Learning

In recent years, with the rapid progress of deep learning, learning methods represented by convolutional neural networks (CNNs) quickly dominate the computer vision field and have been widely applied in the fields of image recognition and segmentation [25–29]. Deep learning methods based on CNN have been popularly applied to the image super-resolution (SR) field since Dong et al. first proposed to combine CNN and super-resolution reconstruction (SRCNN) [30]. Dong et al. [31] replaced the large convolution kernel with more and smaller convolutional layers on the basis of SRCNN and introduced a deconvolution layer at the end of the network to greatly improve the training speed of the model. Benefiting from the increasing depth and complexity of networks, super-resolution networks possess increasingly better performance. The structure of the residual network (ResNet) [32] has been widely applied to effectively solve the gradient vanishing and gradient exploding problems faced by deep networks. The network proposed by Kim et al. (VDSR) [33] increases the convolutional layers to 20 layers, which apply global skip connection as well as adjustable gradient cropping strategy to solve the problem of difficult convergence of deep networks and achieve striking reconstruction results. An enhanced deep super-resolution network (EDSR) [34] further improves the results by removing the unnecessary batch normalization modules. Haris et al. proposed a deep back-projection network (DBPN) [35] exploiting iterative up-and-down sampling layers, which effectively use the interdependence of LR and HR images.


### 2.2. Weather Radar Image Characteristic

Statistics show that the weather radar image contains much redundant information. As shown in Figure2, the red box in the left PPI (Plane Position Indicator) and the black box in the right PPI indicate the given patch and the patches that are nonlocally similar to it, respectively. Many similar and redundant structures can be observed between two patches (the example data are the reflectivity data of the first layer elevation angle of CINRAD-SA radar (Yancheng, Jiangsu, China, 11:48, June 23, 2016). These data have 360 radials, and each radial has 460 range bins. This nonlocal redundant information has the effect of improving the quality of weather radar image reconstructed. As discussed above, limited by the size of the local receptive field, the convolution operation has deficiencies in the fusion of nonlocal information, resulting in the failure to effectively capture the long-range dependencies. Therefore, the network that can better fuse the nonlocal information of weather radar images has better super-resolution reconstruction performance.Figure 2
Example of nonlocally similar weather radar image patches.


### 2.3. Hook Echo

The Hook Echo is a well-researched radar reflectivity signature that must involve descending precipitation curtains as hydrometeors fall relative to the ambient air. As shown on the left part of Figure3, the falling precipitation occurs on the back side of the updraft, which can be regarded as the low-level “extension of the bounded weak echo region (BWER)” [36]. It is commonly believed that the falling precipitation interacts with the mesocyclone of the storm, so it descends along an arc [37]. The right part of Figure 3 shows the radar reflectivity signatures of the hook echo. A common belief is that the hook echo is the area where tornadoes may occur in supercell thunderstorms [38, 39]. Therefore, capturing as much detailed information as possible about the hook echo can help in tornado detection and early warning forecasting.Figure 3
Conceptual model and radar reflectivity signature of hook echo.


## 2.1. Image Super-Resolution Methods

According to the means of implementation, the image super-resolution methods can be divided into three types: interpolation-based, reconstruction-based, and learning-based image super-resolution methods.


### 2.1.1. Methods Based on Interpolation

The basic idea of classical interpolation methods such as nearest interpolation, bilinear interpolation, and bicubic interpolation [20] is to approximate the lost image information by using the basis function or interpolation kernel, which only exploits the neighboring information. These methods often result in blurred edges and loss of high-frequency detail while reducing computational complexity.


### 2.1.2. Methods Based on Reconstruction

Reconstruction-base methods apply prior knowledge of image as constraint terms to the process of super-resolution reconstruction, which effectively solves the ill-posed problem of super-resolution reconstruction. Classical reconstruction-based methods are iterative back projection (IBP) [21], maximum a posterior estimation (MAP) [22], etc.


### 2.1.3. Methods Based on Shallow Learning

Before the deep learning is applied to image super resolution, the most learning-based methods are proposed on the basis of sparse representation such as nonlocally centralized sparse representation (NCSR) [23] and structure-modulated sparse representation (SMSR) [24], which mainly includes dictionary learning and sparse coding.


### 2.1.4. Methods Based on Deep Learning

In recent years, with the rapid progress of deep learning, learning methods represented by convolutional neural networks (CNNs) quickly dominate the computer vision field and have been widely applied in the fields of image recognition and segmentation [25–29]. Deep learning methods based on CNN have been popularly applied to the image super-resolution (SR) field since Dong et al. first proposed to combine CNN and super-resolution reconstruction (SRCNN) [30]. Dong et al. [31] replaced the large convolution kernel with more and smaller convolutional layers on the basis of SRCNN and introduced a deconvolution layer at the end of the network to greatly improve the training speed of the model. Benefiting from the increasing depth and complexity of networks, super-resolution networks possess increasingly better performance. The structure of the residual network (ResNet) [32] has been widely applied to effectively solve the gradient vanishing and gradient exploding problems faced by deep networks. The network proposed by Kim et al. (VDSR) [33] increases the convolutional layers to 20 layers, which apply global skip connection as well as adjustable gradient cropping strategy to solve the problem of difficult convergence of deep networks and achieve striking reconstruction results. An enhanced deep super-resolution network (EDSR) [34] further improves the results by removing the unnecessary batch normalization modules. Haris et al. proposed a deep back-projection network (DBPN) [35] exploiting iterative up-and-down sampling layers, which effectively use the interdependence of LR and HR images.


## 2.1.1. Methods Based on Interpolation

The basic idea of classical interpolation methods such as nearest interpolation, bilinear interpolation, and bicubic interpolation [20] is to approximate the lost image information by using the basis function or interpolation kernel, which only exploits the neighboring information. These methods often result in blurred edges and loss of high-frequency detail while reducing computational complexity.


## 2.1.2. Methods Based on Reconstruction

Reconstruction-base methods apply prior knowledge of image as constraint terms to the process of super-resolution reconstruction, which effectively solves the ill-posed problem of super-resolution reconstruction. Classical reconstruction-based methods are iterative back projection (IBP) [21], maximum a posterior estimation (MAP) [22], etc.


## 2.1.3. Methods Based on Shallow Learning

Before the deep learning is applied to image super resolution, the most learning-based methods are proposed on the basis of sparse representation such as nonlocally centralized sparse representation (NCSR) [23] and structure-modulated sparse representation (SMSR) [24], which mainly includes dictionary learning and sparse coding.


## 2.1.4. Methods Based on Deep Learning

In recent years, with the rapid progress of deep learning, learning methods represented by convolutional neural networks (CNNs) quickly dominate the computer vision field and have been widely applied in the fields of image recognition and segmentation [25–29]. Deep learning methods based on CNN have been popularly applied to the image super-resolution (SR) field since Dong et al. first proposed to combine CNN and super-resolution reconstruction (SRCNN) [30]. Dong et al. [31] replaced the large convolution kernel with more and smaller convolutional layers on the basis of SRCNN and introduced a deconvolution layer at the end of the network to greatly improve the training speed of the model. Benefiting from the increasing depth and complexity of networks, super-resolution networks possess increasingly better performance. The structure of the residual network (ResNet) [32] has been widely applied to effectively solve the gradient vanishing and gradient exploding problems faced by deep networks. The network proposed by Kim et al. (VDSR) [33] increases the convolutional layers to 20 layers, which apply global skip connection as well as adjustable gradient cropping strategy to solve the problem of difficult convergence of deep networks and achieve striking reconstruction results. An enhanced deep super-resolution network (EDSR) [34] further improves the results by removing the unnecessary batch normalization modules. Haris et al. proposed a deep back-projection network (DBPN) [35] exploiting iterative up-and-down sampling layers, which effectively use the interdependence of LR and HR images.


## 2.2. Weather Radar Image Characteristic

Statistics show that the weather radar image contains much redundant information. As shown in Figure2, the red box in the left PPI (Plane Position Indicator) and the black box in the right PPI indicate the given patch and the patches that are nonlocally similar to it, respectively. Many similar and redundant structures can be observed between two patches (the example data are the reflectivity data of the first layer elevation angle of CINRAD-SA radar (Yancheng, Jiangsu, China, 11:48, June 23, 2016). These data have 360 radials, and each radial has 460 range bins. This nonlocal redundant information has the effect of improving the quality of weather radar image reconstructed. As discussed above, limited by the size of the local receptive field, the convolution operation has deficiencies in the fusion of nonlocal information, resulting in the failure to effectively capture the long-range dependencies. Therefore, the network that can better fuse the nonlocal information of weather radar images has better super-resolution reconstruction performance.Figure 2
Example of nonlocally similar weather radar image patches.


## 2.3. Hook Echo

The Hook Echo is a well-researched radar reflectivity signature that must involve descending precipitation curtains as hydrometeors fall relative to the ambient air. As shown on the left part of Figure3, the falling precipitation occurs on the back side of the updraft, which can be regarded as the low-level “extension of the bounded weak echo region (BWER)” [36]. It is commonly believed that the falling precipitation interacts with the mesocyclone of the storm, so it descends along an arc [37]. The right part of Figure 3 shows the radar reflectivity signatures of the hook echo. A common belief is that the hook echo is the area where tornadoes may occur in supercell thunderstorms [38, 39]. Therefore, capturing as much detailed information as possible about the hook echo can help in tornado detection and early warning forecasting.Figure 3
Conceptual model and radar reflectivity signature of hook echo.


## 3. Network Structure

The architecture of NLRN is illustrated in Figure4(a), which consists of four parts: shallow feature extraction, deep feature extraction, upsampling, and reconstruction layer. ILR and ISR denote the input and output of NLRN. The first part extracts the shallow features F0 from the input LR image ILR.(1)F0=HSFILR,where HSF denotes the two convolution operation, each followed by an activation function. To effectively solve the vanishing gradients and dying ReLU problem, we choose leaky rectified linear unit (LReLU) as the activation function. The shallow features F0 are then used as input to the nonlinear mapping network part that consists of G nonlocal residual blocks (NLRBs) to learn mapping relations with sufficient representational ability. The structure of NLRB will be given in detail later. We formulate the deep feature-extraction process as follows:(2)Fg=HRNAB,gFg−1,g=1,…,G,where HRNAB,g⋅ represents the operation of gth NLRB, Fg−1 and Fg denote the input and output of gth NLRB, respectively. To make the network more focused on high-frequency information, we avoid learning low-frequency information by adding long skip connection (LSC), which also effectively fuses shallow and deep features. The process is expressed as follows:(3)FG+1=HFG+F0,where H represents a convolution operation and FG+1 as the input to the upscale module. As the previous work [34], we apply ESPCNN [40] to upscale the deep features and then use one final convolution layer with three filters to provide the final reconstructed images. The output of NLRN can be obtained by(4)ISR=HRHUPFG+1=HNLANILR,where HUP and HR denote the upsampling and the convolution operation, respectively. HNLAN denotes the function of our NLRN.Figure 4
(a) The overall architecture of the proposed nonlocal residual network (NLRN). (b) The inner structure of nonlocal residual block (NLRB).
(a)(b)


### 3.1. Nonlocal Residual Block

Restricted by the local receptive field size, most CNN-based SR methods have deficiencies in the fusion of nonlocal information, which result in low-efficiency utilization of self-similarity properties in images. Inspired by the classical nonlocal means [41], Wang et al. [42] proposed a nonlocal block (NLB) to fuse the nonlocal information of the image by using the self similarity of the nonlocal patches of the image, which obtain promising results in image recognition. Returning to the task of super-resolution reconstruction, in order to fully exploit the input information through the network, we propose a nonlocal residual block (NLRB). As shown in Figure 4(b), the first part of the NLRB is three convolutions with a size of 3 × 3, each of them is followed by LReLU. The input of NLRB and output of NLB are combined by residual learning through a short skip connection (SSC), which also effectively solves the vanishing gradient problems faced by the deep network and facilitates the flow of information.


### 3.2. Nonlocal Block

Now, we present details about NLB. Nonlocal operations can transform the response of the current position into the weighted sum of all position features in the input feature map, so the nonlocal mean can fuse the nonlocal information of the image by using the self similarity of the local patch of the image. The generalization formula of the nonlocal operation in neural networks is defined as(5)yi=∑∀jfxi,xjgxj∑∀jfxi,xj,where i is the location index of the target output y, j is the index of all possible positions in this operation. x is the input feature mapping; y is the output feature mapping, and x and y have the same size. fxi,xj represents the scalar relationship between the positions i and j. The unary function gxj represents the eigenvalues of the feature mapping x at the position index j. Inspired by nonlocal means and bilateral filters, we use embedded Gaussian functions to compute similarity.(6)fxi,xj=expuxiTvxj,where uxi=Wuxi, vxj=Wvxj, Wu and Wv denote the weight matrices. This can be seen from the fact that for a given i, 1/∑∀?jfxi,xjfxi,xj becomes the SoftMax computation along the dimension j. So, we have yi=softmaxxiTWuTWvxjgxj. For simplicity, we consider a linear embedding for gxj:gxj=Wgxj, and Wg denotes the weight matrix. We further transform nonlocal operations into a nonlocal block (NLB). The definition of NLB is as follows:(7)zi=Wzyi+xi,where yi has been given in equation (5), +xi represents a residual connection, which allows us to embed a new NLB in any pretraining model without changing its original structure. Then, the output z at the position i of the (NLB) is calculated as follows:(8)zi=Wzyi+xi=WzsoftmaxxiTWθTWϕxjgxj+xi,where Wz denotes the weight matrix and is initialized as zeros. The structure of NLB is illustrated in Figure 5.Figure 5
The framework of nonlocal block. Red solid line denotes matrix reshaping.H×W×C means C features with height H and width W, and ⊗ denotes the matrix multiplication. ⊕ denotes element-wise addition.


### 3.3. Loss Function

When training NLRN, the optimal set of network parameters is obtained by minimizing the loss between training samples. To show the effectiveness of our NLRN, we choosel1 loss function for our network optimization to provide better convergence [40]. Now, for a batch of N training images i.e.ISRi,ILRii=1N, the aim is to minimize the l1 loss function as(9)Lθ=1N∑i=1NIHRi−ISRi1,where θ denotes the parameters to be learned by the network. The Adam algorithm is applied to optimize the loss function.


## 3.1. Nonlocal Residual Block

Restricted by the local receptive field size, most CNN-based SR methods have deficiencies in the fusion of nonlocal information, which result in low-efficiency utilization of self-similarity properties in images. Inspired by the classical nonlocal means [41], Wang et al. [42] proposed a nonlocal block (NLB) to fuse the nonlocal information of the image by using the self similarity of the nonlocal patches of the image, which obtain promising results in image recognition. Returning to the task of super-resolution reconstruction, in order to fully exploit the input information through the network, we propose a nonlocal residual block (NLRB). As shown in Figure 4(b), the first part of the NLRB is three convolutions with a size of 3 × 3, each of them is followed by LReLU. The input of NLRB and output of NLB are combined by residual learning through a short skip connection (SSC), which also effectively solves the vanishing gradient problems faced by the deep network and facilitates the flow of information.


## 3.2. Nonlocal Block

Now, we present details about NLB. Nonlocal operations can transform the response of the current position into the weighted sum of all position features in the input feature map, so the nonlocal mean can fuse the nonlocal information of the image by using the self similarity of the local patch of the image. The generalization formula of the nonlocal operation in neural networks is defined as(5)yi=∑∀jfxi,xjgxj∑∀jfxi,xj,where i is the location index of the target output y, j is the index of all possible positions in this operation. x is the input feature mapping; y is the output feature mapping, and x and y have the same size. fxi,xj represents the scalar relationship between the positions i and j. The unary function gxj represents the eigenvalues of the feature mapping x at the position index j. Inspired by nonlocal means and bilateral filters, we use embedded Gaussian functions to compute similarity.(6)fxi,xj=expuxiTvxj,where uxi=Wuxi, vxj=Wvxj, Wu and Wv denote the weight matrices. This can be seen from the fact that for a given i, 1/∑∀?jfxi,xjfxi,xj becomes the SoftMax computation along the dimension j. So, we have yi=softmaxxiTWuTWvxjgxj. For simplicity, we consider a linear embedding for gxj:gxj=Wgxj, and Wg denotes the weight matrix. We further transform nonlocal operations into a nonlocal block (NLB). The definition of NLB is as follows:(7)zi=Wzyi+xi,where yi has been given in equation (5), +xi represents a residual connection, which allows us to embed a new NLB in any pretraining model without changing its original structure. Then, the output z at the position i of the (NLB) is calculated as follows:(8)zi=Wzyi+xi=WzsoftmaxxiTWθTWϕxjgxj+xi,where Wz denotes the weight matrix and is initialized as zeros. The structure of NLB is illustrated in Figure 5.Figure 5
The framework of nonlocal block. Red solid line denotes matrix reshaping.H×W×C means C features with height H and width W, and ⊗ denotes the matrix multiplication. ⊕ denotes element-wise addition.


## 3.3. Loss Function

When training NLRN, the optimal set of network parameters is obtained by minimizing the loss between training samples. To show the effectiveness of our NLRN, we choosel1 loss function for our network optimization to provide better convergence [40]. Now, for a batch of N training images i.e.ISRi,ILRii=1N, the aim is to minimize the l1 loss function as(9)Lθ=1N∑i=1NIHRi−ISRi1,where θ denotes the parameters to be learned by the network. The Adam algorithm is applied to optimize the loss function.


## 4. Implementation Details

The implementation details on datasets, degradation process, training details, and evaluation metrics are presented in this section.


### 4.1. Datasets

We use reflectivity data from China’s New Generation Weather Radar (CINRAD-SA) as a dataset. For the elevation of each layer, there are 360 radials, 460 range bins reflectivity data for each radial, and the range resolution of 1 km. To enable NLRN to learn large-scale precipitation features and small-scale extreme weather features, we use the precipitation experimental data of South China in Guangdong on May-June, 2016, the tornado and hail data of Yancheng in Jiangsu on May 27, 2008, the tornado data of Yancheng in Jiangsu on June 23, 2016 and Nantong in Jiangsu on July 06, 2016, and the typhoon data of Xuzhou in Jiangsu in August 2018. The total number of high resolution (HR) datasets with a size of 512 × 512 is 1800, of which 1600 images are used for training, 20 images for validation, and 180 images for testing. We mainly study the super-resolution reconstruction performance of NLRN under ×2 and ×4 conditions, so low resolution (LR) images with a size of 256 × 256 and 128 × 128 are obtained as input of NLRN by applying the degradation method to HR images.


### 4.2. Degradation Process

The weather radar image degradation process includes three processes: blurring, downsampling, and system noise. The degradation process can be formulated by the equation as follows:(10)y=Ax+n,where A represents the degradation operation (e.g., blurring kernel and downsampling operation), and n represents the weather radar receiver noise, which obeys the zero-mean Gaussian distribution.


### 4.3. Parameter Settings

During the training, common data enhancement methods are also used in training, such as random horizontal rotations of 90°, 180°, and 270°. For every training batch, 16 LR patches with the size of 20 × 20 are randomly extracted as inputs. The convolutional kernels present in the network are all 3 × 3 in size and 64 in number of filters. To explore the most appropriate number ofG, we have counted the average PSNR under different weather conditions (e.g., the large weather system, small weather system, and cloudless) with different parameter G. As shown in Figure 6, when G≥4, the growth of PSNR becomes slow or even decreases. For balancing performance and computational complexity, we set the number of G as 4. The negative slope of leaky rectified linear units (LReLUs) is 0.1. The Adam algorithm with β1=0.9,β2=0.09,ε=10−8 is adopted to optimize the network. The initial learning rate is 10−4 and decreases by half for every 200 epochs. Training a NLRN on PyTorch (1.01) framework roughly takes two days with two Tesla P40 GPUs for 500 epochs.Figure 6
The average PSNR performance of NLRN on validation set with different parameterG (the total numbers of NLRB).


### 4.4. Evaluation Metrics

In order to test the effectiveness of the proposed NLRN, we compare NLRN with several SR methods, including Bicubic, IBP, NCSR, VDSR, and EDSR. Bicubic is a classic interpolation method. IBP is a classical reconstruction method based on gradual iteration. NCSR is a traditional learning method based on sparse dictionary learning. VDSR and EDSR are CNN-based deep learning methods. Peak-signal-to-noise ratio (PSNR) (dB) and structural similarity (SSIM) [43] are used to quantitatively evaluate the SR methods.


## 4.1. Datasets

We use reflectivity data from China’s New Generation Weather Radar (CINRAD-SA) as a dataset. For the elevation of each layer, there are 360 radials, 460 range bins reflectivity data for each radial, and the range resolution of 1 km. To enable NLRN to learn large-scale precipitation features and small-scale extreme weather features, we use the precipitation experimental data of South China in Guangdong on May-June, 2016, the tornado and hail data of Yancheng in Jiangsu on May 27, 2008, the tornado data of Yancheng in Jiangsu on June 23, 2016 and Nantong in Jiangsu on July 06, 2016, and the typhoon data of Xuzhou in Jiangsu in August 2018. The total number of high resolution (HR) datasets with a size of 512 × 512 is 1800, of which 1600 images are used for training, 20 images for validation, and 180 images for testing. We mainly study the super-resolution reconstruction performance of NLRN under ×2 and ×4 conditions, so low resolution (LR) images with a size of 256 × 256 and 128 × 128 are obtained as input of NLRN by applying the degradation method to HR images.


## 4.2. Degradation Process

The weather radar image degradation process includes three processes: blurring, downsampling, and system noise. The degradation process can be formulated by the equation as follows:(10)y=Ax+n,where A represents the degradation operation (e.g., blurring kernel and downsampling operation), and n represents the weather radar receiver noise, which obeys the zero-mean Gaussian distribution.


## 4.3. Parameter Settings

During the training, common data enhancement methods are also used in training, such as random horizontal rotations of 90°, 180°, and 270°. For every training batch, 16 LR patches with the size of 20 × 20 are randomly extracted as inputs. The convolutional kernels present in the network are all 3 × 3 in size and 64 in number of filters. To explore the most appropriate number ofG, we have counted the average PSNR under different weather conditions (e.g., the large weather system, small weather system, and cloudless) with different parameter G. As shown in Figure 6, when G≥4, the growth of PSNR becomes slow or even decreases. For balancing performance and computational complexity, we set the number of G as 4. The negative slope of leaky rectified linear units (LReLUs) is 0.1. The Adam algorithm with β1=0.9,β2=0.09,ε=10−8 is adopted to optimize the network. The initial learning rate is 10−4 and decreases by half for every 200 epochs. Training a NLRN on PyTorch (1.01) framework roughly takes two days with two Tesla P40 GPUs for 500 epochs.Figure 6
The average PSNR performance of NLRN on validation set with different parameterG (the total numbers of NLRB).


## 4.4. Evaluation Metrics

In order to test the effectiveness of the proposed NLRN, we compare NLRN with several SR methods, including Bicubic, IBP, NCSR, VDSR, and EDSR. Bicubic is a classic interpolation method. IBP is a classical reconstruction method based on gradual iteration. NCSR is a traditional learning method based on sparse dictionary learning. VDSR and EDSR are CNN-based deep learning methods. Peak-signal-to-noise ratio (PSNR) (dB) and structural similarity (SSIM) [43] are used to quantitatively evaluate the SR methods.


## 5. Experimental Results and Analysis

In this section, we compare NLRN with other SR methods in terms of the training process, visual quality, and quantitative results.


### 5.1. Training Comparison

To further compare the convergence and accuracy when training VDSR, EDSR, and NLRN, the training loss curves of three networks are plotted in Figure7(a). From Figure 7(a), it can be seen that after 100 epochs, the loss function loss decreases more slowly with iterations increasing and finally stabilizes between 6.0 and 7.0, indicating that the network has converged on the training dataset. It can be seen that NLRN has both faster convergence and better accuracy than VDSR and EDSR when training. Figure 7(b) represents the validation PSNR values of several SR methods mentioned above during the training. It can be seen that NLRN exhibited considerable advantages over the conventional methods (Bicubic, IBP, and NCSR) and CNN-based methods (EDSR and VDSR).Figure 7
(a) Trainingl1 loss of VDSR, EDSR, and NLRN for scale ×4. (b) Validation PSNR values of Bicubic, IBP, NCSR, VDSR, EDSR, and NLRN during training for scale ×2.
(a)(b)


### 5.2. Visual Quality Comparison

In order to test the performance of NLRN under the large-scale weather system, we choose precipitation and typhoon reflectivity data as test data. For weather radar reflectivity data, the intense precipitation convective cells are often embedded in a lower intensity region, which shows high aggregation and sparse correlation. Figure8 shows some of the visual results of the different SR methods under an intense precipitation condition (Heyuan, Guangdong, China, 11:48, June 15, 2016). The deep learning-based output is subjectively superior to the conventional methods. The approach is not able to recover all of the very fine-scale precipitation structure that is lost when the original weather radar image is degraded. However, the deep leaning methods preserve more of the fine-scale structure and is notably better at preserving sharp edges associated with the larger features. Compared with VDSR and EDSR, NLRN effectively exploits the self similarity of radar precipitation images through the nonlocal operation, which is very useful for the identification and monitoring of intense convective precipitation echo by reconstructing high-frequency details that are prone to variability in the map, making the echo structure more refined and highlighting the location of intense echo. As shown in Figure 9, NLRN also achieves the best subjective output under the typhoon conditionFigure 8
Visual comparison of ×4 super-resolution reconstruction results of intense precipitation data. The best results are in bold.Figure 9
Visual comparison of ×4 super-resolution reconstruction results of typhoon data. The best results are in bold.(Xuzhou, Jiangsu, China, 09:30, August 18, 2018) by reconstructing more intense echo information. To test the performance of NLRN under the small-scale weather system, we choose tornado data (Yancheng, Jiangsu, China, 14:08, June 23, 2016) as test data. From Figure10, it can be seen that although both the conventional methods and the deep learning methods can reconstruct most of the hook echo detail information, the hook echo reconstructed by the NLRN is closer to the original hook echo, which helps forecasters to analyze small- and medium-scale extreme weather and make timely forecast and warning condition (Xuzhou, Jiangsu, China, 09:30, August 18, 2018) by reconstructing more intense echo information.Figure 10
Visual comparison of ×4 super-resolution reconstruction results of tornado data. The best results are in bold.


### 5.3. Quantitative Result Comparison

Average PSNR (dB) comparison of VDSR, EDSR, and NLRN is listed in Table1. It can be seen from the result that both EDSR and NLRN can achieve PSNR more than 0.2 dB higher than VDSR, which indicates that EDSR and NLRN with SSC have better performance in feature fusion compared with VDSR only using LSC. Furthermore, NLRN achieves better performance in exploiting large-scale context information by adding the nonlocal operation, which leads to the PSNR gain of NLRN over EDSR up to 0.07 dB. To further validate the effectiveness of NLRN, we compare the proposed NLRN with various SR methods on degraded weather radar images under different weather conditions (e.g., large- and small-scale weather system, and cloudless). All the results on ×2 and ×4 reconstruction are shown in Table 2, from which we can see that the NLRN proposed in this article can get the highest PSNR and SSIM values compared with other SR methods. Especially under the small-scale weather system, the PSNR gain of NLRN over Bicubic is up to 2 dB under ×2 reconstruction.Table 1
Average PSNR (dB) of VDSR, EDSR, and NLRN for×4 super-resolution reconstruction on test set.
NetworkVDSREDSRNLRNLong skip Connection (LSC)√√√Short skip Connection (SSC)×√√Nonlocal operation××√PSNR25.1725.3825.45Table 2
Quantitative results of the compared methods on different weather conditions.
MethodScaleLarge weather systemSmall weather systemCloudlessPSNRSSIMPSNRSSIMPSNRSSIMBicubic×226.180.793326.560.801227.890.8458IBP×226.670.793527.130.807928.450.8562NCSR×227.280.823327.650.826328.980.8749VDSR×228.050.849628.110.849229.310.8961EDSR×228.260.853128.480.856929.420.9013NLRN×228.410.854628.570.861229.530.9042Bicubic×423.210.673923.660.674325.120.7268IBP×423.770.678424.130.680325.430.7327NCSR×424.690.689824.510.707825.810.7513VDSR×424.830.732524.920.740526.220.7857EDSR×424.980.738225.080.748726.410.7893NLRN×425.110.742125.190.749826.480.8013The best results are in bold.


## 5.1. Training Comparison

To further compare the convergence and accuracy when training VDSR, EDSR, and NLRN, the training loss curves of three networks are plotted in Figure7(a). From Figure 7(a), it can be seen that after 100 epochs, the loss function loss decreases more slowly with iterations increasing and finally stabilizes between 6.0 and 7.0, indicating that the network has converged on the training dataset. It can be seen that NLRN has both faster convergence and better accuracy than VDSR and EDSR when training. Figure 7(b) represents the validation PSNR values of several SR methods mentioned above during the training. It can be seen that NLRN exhibited considerable advantages over the conventional methods (Bicubic, IBP, and NCSR) and CNN-based methods (EDSR and VDSR).Figure 7
(a) Trainingl1 loss of VDSR, EDSR, and NLRN for scale ×4. (b) Validation PSNR values of Bicubic, IBP, NCSR, VDSR, EDSR, and NLRN during training for scale ×2.
(a)(b)


## 5.2. Visual Quality Comparison

In order to test the performance of NLRN under the large-scale weather system, we choose precipitation and typhoon reflectivity data as test data. For weather radar reflectivity data, the intense precipitation convective cells are often embedded in a lower intensity region, which shows high aggregation and sparse correlation. Figure8 shows some of the visual results of the different SR methods under an intense precipitation condition (Heyuan, Guangdong, China, 11:48, June 15, 2016). The deep learning-based output is subjectively superior to the conventional methods. The approach is not able to recover all of the very fine-scale precipitation structure that is lost when the original weather radar image is degraded. However, the deep leaning methods preserve more of the fine-scale structure and is notably better at preserving sharp edges associated with the larger features. Compared with VDSR and EDSR, NLRN effectively exploits the self similarity of radar precipitation images through the nonlocal operation, which is very useful for the identification and monitoring of intense convective precipitation echo by reconstructing high-frequency details that are prone to variability in the map, making the echo structure more refined and highlighting the location of intense echo. As shown in Figure 9, NLRN also achieves the best subjective output under the typhoon conditionFigure 8
Visual comparison of ×4 super-resolution reconstruction results of intense precipitation data. The best results are in bold.Figure 9
Visual comparison of ×4 super-resolution reconstruction results of typhoon data. The best results are in bold.(Xuzhou, Jiangsu, China, 09:30, August 18, 2018) by reconstructing more intense echo information. To test the performance of NLRN under the small-scale weather system, we choose tornado data (Yancheng, Jiangsu, China, 14:08, June 23, 2016) as test data. From Figure10, it can be seen that although both the conventional methods and the deep learning methods can reconstruct most of the hook echo detail information, the hook echo reconstructed by the NLRN is closer to the original hook echo, which helps forecasters to analyze small- and medium-scale extreme weather and make timely forecast and warning condition (Xuzhou, Jiangsu, China, 09:30, August 18, 2018) by reconstructing more intense echo information.Figure 10
Visual comparison of ×4 super-resolution reconstruction results of tornado data. The best results are in bold.


## 5.3. Quantitative Result Comparison

Average PSNR (dB) comparison of VDSR, EDSR, and NLRN is listed in Table1. It can be seen from the result that both EDSR and NLRN can achieve PSNR more than 0.2 dB higher than VDSR, which indicates that EDSR and NLRN with SSC have better performance in feature fusion compared with VDSR only using LSC. Furthermore, NLRN achieves better performance in exploiting large-scale context information by adding the nonlocal operation, which leads to the PSNR gain of NLRN over EDSR up to 0.07 dB. To further validate the effectiveness of NLRN, we compare the proposed NLRN with various SR methods on degraded weather radar images under different weather conditions (e.g., large- and small-scale weather system, and cloudless). All the results on ×2 and ×4 reconstruction are shown in Table 2, from which we can see that the NLRN proposed in this article can get the highest PSNR and SSIM values compared with other SR methods. Especially under the small-scale weather system, the PSNR gain of NLRN over Bicubic is up to 2 dB under ×2 reconstruction.Table 1
Average PSNR (dB) of VDSR, EDSR, and NLRN for×4 super-resolution reconstruction on test set.
NetworkVDSREDSRNLRNLong skip Connection (LSC)√√√Short skip Connection (SSC)×√√Nonlocal operation××√PSNR25.1725.3825.45Table 2
Quantitative results of the compared methods on different weather conditions.
MethodScaleLarge weather systemSmall weather systemCloudlessPSNRSSIMPSNRSSIMPSNRSSIMBicubic×226.180.793326.560.801227.890.8458IBP×226.670.793527.130.807928.450.8562NCSR×227.280.823327.650.826328.980.8749VDSR×228.050.849628.110.849229.310.8961EDSR×228.260.853128.480.856929.420.9013NLRN×228.410.854628.570.861229.530.9042Bicubic×423.210.673923.660.674325.120.7268IBP×423.770.678424.130.680325.430.7327NCSR×424.690.689824.510.707825.810.7513VDSR×424.830.732524.920.740526.220.7857EDSR×424.980.738225.080.748726.410.7893NLRN×425.110.742125.190.749826.480.8013The best results are in bold.


## 6. Conclusion

In this article, we proposed a nonlocal residual network (NLRN) for weather radar image super resolution, where the nonlocal residual block (NLRB) allows NLRN to integrate the nonlocal dependencies and structural information (e.g., the local and nonlocal correlations of intense precipitation echo) by inserting nonlocal blocks (NLBs) in the network. Extensive experiments on different weather conditions show the effectiveness of our NLRN in terms of quantitative and visual results. Although NLRN cannot reconstruct exactly the same echo geometry, it can recover more accurate echo edges and details than other SR methods listed in the experimental part and highlight the structure, location, and development of intense echo, which contributes to fine detection and prediction of small- and medium-scale intense convection processes. It is worth noting that the deep learning-based super resolution of weather radar image relies not only on an excellent model structure but also on the completeness and diversity of the dataset, and that both the NLRN and the training dataset need to be further optimized to meet the increased demand for monitoring and forecasting of severe convective weather events.



---

*Source: 4483907-2021-12-30.xml*