Uncertainty-Driven Loss for Single Image Super-Resolution
Authors: Qian Ning
Weisheng Dong
Xin Li
Jinjian Wu
Guangming Shi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Uncertainty-Driven Loss
Uncertainty Learning
Single Image Super-Resolution

Abstract: In low-level vision such as single image super-resolution (SISR), traditional MSE or L_1 loss function treats every pixel equally with the assumption that the importance of all pixels is the same. However, it has been long recognized that texture and edge areas carry more important visual information than smooth areas in photographic images. How to achieve such spatial adaptation in a principled manner has been an open problem in both traditional model-based and modern learning-based approaches toward SISR. In this paper, we propose a new adaptive weighted loss for SISR to train deep networks focusing on challenging situations such as textured and edge pixels with high uncertainty. Specifically, we introduce variance estimation characterizing the uncertainty on a pixel-by-pixel basis into SISR solutions so the targeted pixels in a high-resolution image (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, uncertainty estimation allows us to leverage conventional wisdom such as sparsity prior for regularizing SISR solutions. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance to visual quality. For the first time, we demonstrate that such uncertainty-driven loss can achieve better results than MSE or L_1 loss for a wide range of network architectures. Experimental results on three popular SISR networks show that our proposed uncertainty-driven loss has achieved better PSNR performance than traditional loss functions without any increased computation during testing. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/UDL-SR.htm

Uncertainty-Driven Loss for Single Image Super-Resolution
Qian Ning', Weisheng Dong'; Xin Li', Jinjian Wu', Guangming Shi!
Schooo of Artificial Intelligence, Xidian University, Xi' an 710071, China
Lane Dep. of CSEE, West Virginia University, Morgantown WV 26506, USA
ningqiani antutu. x di an edu. cn, (wsdong, jinjian. we @ma i xidi an edu. cn xin 11 @mail 1 edu, gmshi @xidian edu.cn
Abstract
In low-level vision such as single image super-resolution (SISR), traditional MSE or C loss function treats every pixel equally with the assumption that the importance all pixels the same. However, has been long recognized that texture and edge areas carry more important visual information than smooth areas in photographic images. How to achieve such spatial adaptation in a principled manner has been open problem in both traditional model-based and modern learning-bai based approaches toward SISR. In this paper, we propose a new adaptive weighted loss for SISR to train deep networks focusing challenging situations such as textured and edge pixels with high uncertainty. Specifically, we introduce variance estimation characterizing the uncertainty on pixel-by-pixel basis into SISR solutions so the targeted pixels in high-resolution image (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, uncertainty estimation allows us to leverage conventional wisdom such as sparsity prior for regularizing SISR solutions. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance visual quality. For the first time, demonstrate that such mmceraintt--rrvve loss can achieve better results than MSE or L1 loss for wide range of network architectures. Experimental results on three popular SISR networks show that our proposed incertainty-driverr loss has achieved better PSNR performance than traditional loss functions without any increased computation during testing. The code is available at httpp:://se.xidiiatt edu.
1
Introduction
Single image super-resolution (SISR) aims at reconstructing high-resolution (HR) images from their corresponding degraded low-resolution (LR) images. Since the publication of super-resolution with convolutional neural network (SRCNN) 1 there has been a flurry of works on deep learning-based
approaches toward SISR e.g., EDSR 2), DPDNN [3, RCAN (4) SAN (5), and MoG-DUN [6]. The unifying theme along this line of research appears to be that deeper, bigger, and more complex networks can achieve improved SISR performance by facilitating the reconstruction of high-frequency details such as textures and edges in photographic images. Such improvement has been achieved by novel network architectures (c.g., skip connections (2)), new attention mechanism (c.g., residue channel attention [4), and closed-loop supervision (7]. Surprisingly. most of these existing methods have adopted MSE or L, loss optimize the parameters of networks.
The commonly used practice, such as MSE or L1 loss, treats every pixel equally regardless of whether the pixel is in texture/edge regions smooth areas. The optimality of such non-adaptive loss function
Conresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Go bug. 007 from Setl4
(b) TIR image
(c) SR image f(x)
(d) Difference map
f(y))
Figure 1: Illustraiion of the difference (d) between HR image (b) and SR image (c) reconstructed by EDSR network [2] on dataset Set14 (8. The image reconstructed by EDSR network is shown in (c)
and (d) shows the absolute difference between the HR image and SR image. Best viewed in color.
has been questioned in the literature of SISR calling for the proposition of perceptual loss function
(e.g., [9). From a Bayesian perspective, the assumption underlying the MSE or L1 loss is that each pixel obeys the independent and identically distribution with the same variance. Taking L₁ loss as an example, the likelihood of all pixels in an image can be formulated as
II cexp(- ||x00 f(W) (1))),
p( y. W)
where and y denote the pair of HR and LR image, (.) denotes an arbitrary SISR network parameterized by W, and c, a denote spatially invariant constants. However, such assumption of stationarity or spatial invariance of image prior model is invalid for photographic images in the real world. For instance, if one compares the ground-truth (HR image) and the SR image reconstructed by EDSR (2 shown in Fig. [ (c). it can be observed that texture areas (c.g.. hair of baboon) are not restored as good as smooth areas (c.g.. nose of baboon). Fig. [ (d) depicts the absolute difference between the HR image and reconstructed SR image, from which we can observe spatial variation of the difference map. Such observation implies that the uncertainty texture and edge areas as characterized by the variance is much larger than that in smooth areas. How to address such uncertainty-dree driiee loss for SISR sets up the stage for this paper.
In this paper, we propose new adaptive weighted loss (uncertainty-drive) loss) for SISR by assigning texture and edge areas with higher weights during the training process. Unlike previous work of perceptual loss (9] focusing on characterizing content and style consistency, we target at explicitly estimating the variance field underlying the unknown HR image in the first step, which can be exploited consequence of our two-step learning approach is that delivers not only higher visual quality but also improved objective performance such PSNR and SSIM. Moreover, uncertainty estimation perspective allows us to easily incorporate existing models such as Jefferey's 's prior 10, 11 into the proposed SISR solution. It follows that the network training boils down two sequential steps in which the variance map is estimated from the first step and serves as the attention signal for the second step. The main technical contributions summarized as follows.
auxiliary signal for guiding the SISR solution in the second step. A direct
Uncertainty modeling and estimation. We propose to cast SISR into a Bayesian estimation framework under which SR image (mean) and uncertainty (variance) are derived simulta- neously. Unlike previous works in which pixels with large uncertainty are attenuated for high-level vision tasks, we advocate to prioritize them for low-level vision tasks such as SISR.
Uncertainty-driee driven loss (UDL). The estimation of variance map facilitates the training of SISR network by dividing it into two steps. In the first step, an estimating sparsity uncertainty (ESU) loss function was derived from the classical Jeffrey's prior to estimate the variance map. In the second step. the estimated variance map serves as the guidance signal leading to adaptive weighted loss named uncertainty-dif
loss LUDI.-
Universality of UDL. The proposed uncertainty loss can easily be employed in any existing SISR network to improve performance and do not increase any additional computation cost during testing.
Experimental results on three different baseline networks show that our proposed uncertainty- driven loss has achieved better PSNR performance than traditional MSE or C1 loss.

---

2 Related Work
2.1 Uncertainty in Deep Learning
Many works [12 have introduced uncertainty into the regression with input-dependent noises problems, and studied the nature and behavior of uncertainty for a long time. More recently, modeling uncertainty in deep learning have improved the performance and robustness of deep networks in many computer vision tasks [15H17] such as image classification [18], image segmentation [15] and face recognition [7 19 The uncertainty in deep learning can be roughly divided into two categories [20]. Epistemic/model uncertainty describes how much the model is uncertain about its predictions. Another type aleatoric/data uncertainty which refers to noise inherent in observation data. In (15), they presented a Bayesian deep learning framework combining alcatoric uncertainty with epistemic uncertainty for per-pixel semantic segmentation and depth regression tasks. Chang al. [17] investigated the data uncertainty with estimated mean and variance in face recognition. Those uncertainty-baat based loss function proposed by those works 17 can be summarized as
N
D
f(y)) ll2 1 In o
C
2 20²
where f14) and of denote the learned mean and variance respectively. Using above loss function indeed improved their robustness to noisy data. In those tasks, the pixels with high uncertainty were regarded as unreliable pixels which would bear loss attenuation. On the contrary, in SISR tasks, the pixels with high uncertainty (e.g., complex texture or edge areas) should be prioritized since those regions visually more important than pixels in smooth areas. That can explain why applying above loss into SISR directly leads performance decline.
2.2 Modeling Uncertainty for SISR
To the best of our knowledge, only two works have studied the behavior of uncertainty for SISR in the open literature. [22] used
uncertainty to analyze SISR uncertainty, improving the robustness of the network against adversarial attack. The most recent advance related to our work is Gradient Rescaling Attention Model (GRAM) [21] which analyses the effect of aleatoric/data uncertainty on SISR reconstruction. By decreasing the loss attenuation of large variance pixels, GRAM achieves better results than applying above uncertainty loss into SISR directly. However, GRAM 21 loss remains attenuated when the variance of pixels is high, which contradicts the intuition of prioritizing texture and edge pixels. Thus, GRAM [21] is still inferior baseline methods since the proposed method fails to prioritize the pixels large variance. Different from GRAM, we propose a novel uncertainty-driven loss (UDL) to enforce the network concentrating more on the pixels with large variance aiming at better reconstruction of texture and edge regions. By quantifying the uncertainty in SISR under deep Bayesian framework, our proposed method has achieved better results than baseline methods.
3 Methodology
Unlike traditional MSE or Li loss treating every pixel equally, the proposed new adaptive weighted loss for SISR aims at prioritizing texture and edge pixels that are visually more important than pixels in smooth areas. Toward this objective, we first introduce approach of estimating intermediate results of SR image (mean) and uncertainty (variance) simultancously in SISR. Then, with Jeffrey's s prior term, regularized approach of estimating sparse uncertainty is proposed for more accurate uncertainty estimation. An important new insight brought by this paper is that unlike high-level vision tasks where pixels with large uncertainty are assigned lower weights to attenuate their impact (T5), one should prioritize these pixels in low-level vision tasks such as SISR. Such observation implies that the attenuation of weighting coefficients in loss function needs to be properly translated into the attention mechanism given the specific vision problem as the context.
In previous study [5] it has been shown that explicitly representing aleatoric uncertainty can lead to performance and robustness improvement to noise data in high-level vision tasks such as image segmentation. Such improvement can be explained away by attenuating the weights of pixels with
3

---

large uncertainty. However, attenuation has to go the opposite direction in low-level vision tasks such as SISR i.e. larger weights should be assigned the pixels with high uncertainty (e.g., texture and edge pixels) because they are visually more important than pixels in smooth regions. It should be noted that existing work such as gradient rescaling strategy in GRAM [2]] fails to recognize such difference and does not prioritize pixels with high uncertainty. In this paper, we propose new adaptive weighted loss named uncertainty-driven loss (UDL) for properly turning attenuation into attention for SISR.
Step1
Upsampling Network
f(y)
Conv
Conv
Conv
Conv
ESU
y
Neaeett
Upsumpling
Backbone Network
Conv
ELU
Step2
Upsampling Network
f(y)
Conv
Conv
Conv
Conv
Lupe
y
(b)
Buckbone Network
Figure 2: The overview of training SISR network with proposed LUDL loss. The whole training process can divided into two steps; the first step estimates the uncertainty precisely and the second step generates the final mean value (t) In stepl shown in (a), the mean value f(y) and variance are pretrained by LESU loss. During step2, as shown in (b), the mean value ('y) network trained
by LUDI. loss, while the network of inferring variance 0 is fixed. Note that the mean value ( network of step2 starts training from the pretrained network of stepl. The Nearest Upsampling denotes interpolation operator.
3.1 Estimating Uncertainty (EU) in SISR.
As discussed in [15]. there are two classes of uncertainty in Bayesian modeling: aleatoric uncertainty capturing noise inherent in observation data and epistemic uncertainty accounting for uncertainty of model about its predictions. We opt to study the former (aleatoric uncertainty) and explore its application into SISR by designing new uncertainty-drive order to better quantify aleatoric uncertainty in SISR, we use y: æ; denote the low-resolution (LR) image and the corresponding high-resolution (HR) image respectively. Let f(-) denotes an arbitrary SISR network and the aleatoric uncertainty can denoted by an additive term 0,. This way, the overall observation model can be formulated as
loss (UDL) functions in this paper. In
fv:) + O;,
where represents the Laplace distribution with zero-mean and unit- varianee Existing deep-learning based SISR methods target at training network to Iearn the SR image (mean) f(y:) only. To more accurately characterize alcatoric uncertainty for SISR, we propose estimate not only the SR image (mean) f(y,) but also the uncertainty (variance) 0, simultancously.
For given LR image y: and corresponding HR image æ; Laplace distribution is assumed for characterizing the likelihood function by
|| - f(1))l1)
p(a;, 0.ly))
2 20; exp(
(4)
0,
where f(y:) and 0 denote the SR image (mean) and the uncertainty (variance) which are learned by deep neural networks (DNNs) respectively. Then, the log likelihood can be formulated as follows,
In p(x;, 0,1y:) = x f(y)ll In 0, - In 2
(5)
0
The most commonly used loss function in SISR Li loss which refers to Laplace distribution.

---

0
(A)
(6) EUSR-S
(c) HUSR-S-Cru
EDSR-S- S-DSS---CggS Cgsu
Cn: Lacertuatty
(8) PESU Uncertainty
Figure 3: SISR visual quality comparisons of EDSR-S 2 with different loss function on 'lmg_005 005'
from Set5 24 (bicubie-downsampling x4). Best viewed in color.
For numerical stability, we train the networks to estimate log variance 8; = In 0, as shown in Fig. 2 (a). At last, the maximum likelihood estimation of (5 can be reformulated the minimization of following loss function for estimating uncertainty (EU) in SISR.
LEU Σ exp( 8:) ||:0 f(y))l
Jeffrey's Prior for Estimating Sparse Uncertainty (ESU) in SISR. The loss function LEU includes two terms; the first one associated with fidelity term and the second one prevents the network from predicting infinite uncertainty for all pixels. Those two terms reach equilibrium but there is no prior that imposed on the uncertainty estimation. Therefore, based on the observation that the uncertainty we propose impose Jeffrey's prior [T0]
is sparse in view of the whole image as shown in Fig. [ on uncertainty 0 which can be expressed as p( p(w) x w
p(a,, 0, y.) = p(x.///.... (.)) x 20: exp( >
1
y
1
2
f(y
exp(
0,
0,
2
O,
Then the log likelihood and loss function can be separately formulated as follows,
| f(y.)1l
In p( y.)
2 In 0: In
(8)
0:
1 CESU =
N Σ exp( (s:) ||x; - -f(y))lll + st
(9)
The limitations of LEU and CESU loss. Applying LEu and CESU loss leads to more accurate es- timation of uncertainty (variance field), but counter-intuitively, they do not directly improve the performance SISR. We have conducted experiments comparing those three different loss functions to verify the above claim. As shown in Tab. [ the average PSNR and SSIM results of LESU and LEu are notably lower than the original results. The reason behind this observation is that both Leu and LESU loss functions have incorporated the variance term (0:) into the divisor of the absolution difference term. Consequently, pixel with large variance will penalized after the division and has less impact on the overall loss function. Note that such attenuation of pixels with large uncertainty is preferred for high-level vision tasks, as demonstrated in previous works [15 on image classification [18) image segmentation 16), and face recognition [7 19].
Low-level vision tasks such as SISR are much different. As shown in Fig. pixels with large uncertainty carry visually important information such as textured and edges. They need to be prioritized (opposite to attenuation) and given larger instead of smaller weights. To verify such claim, we have presented simple example comparing the visual results between LEU and LESU as shown Fig- 3 can seen that the uncertainty captured by LesU loss is better than LA:: loss. The improvement of LESU in Eq. (9) over LEU in Eq. (6) is attributed to the prioritization of pixels with large uncertainty (s, values). Fig. 3 (f) clearly demonstrate superiority of exploiting the sparsity constraint with the uncertainty estimation.
5

---

Table 1: Average PSNR and SSIM results for BI degradation five datasets for investigating three
different loss. The best performance is shown in bold. We record the results in 1.2 x 10 iterations.
Base Model
Scale
Loss
FSNK
Sct5
SSIM
PSNR
Set14 [8]
SSIM
PSNR
BSDTO0
SSTM
PSNR
Unban
SSIM
PSNR
Manga 109 26
SSIM
Original
30,93
0,8700
27.80
0.7627
27.05
0.7190
24.71
0.7351
28.14
0.8693
S
Cx:
30.19
0.8627
27.29
0.7538
26.78
0.7120
0.7179
0.8481
30.31
0.8637
27.39
0.7543
26.83
0.7124
24.27
0.7192
26.92
0.8496
3.2
Uncertainty-Driver Loss (UDL) for SISR
Improvement of LESU over Leu inspired us to go one step further. To better prioritize pixels with large uncertainty, we propose a new adaptive weighted loss named uncertainty-driven loss (UDL) for SISR. Unlike LesU loss putting larger weight to the second term than LEU, we suggest that the first term can also be modified to directly associate the aleatoric/data uncertainty of ((ys). That is, instead of using exp(-s; s;) to attenuate the importance of pixels with large uncertainty, we need to use a monotonically increasing function to prioritize them. Linear scaling would be a natural option, which leads to the following loss function
1 N
LUDE. =
8 |1. - f(y))),
(10)
where 8; = 8; ) is a non-negative linear scaling function. To prevent uncertainty value from degenerating into zeros, the result of uncertainty estimation network in the first step will be passed to the second step as the attention signal (s = In 0), as shown in Fig. 2 By leveraging the log variance represent the challenging and cumbersome pixels with higher uncertainty, we propose a new weighted loss named uncertainty driven loss LUDL. In LUDL loss, texture and edge pixels with higher uncertainty tend have larger weights than those in smooth regions. In summary, the uncertainty estimation 0 serves as the bridge connecting two steps: it is the output of the first step; but passed on to the second step as the guidance required for calculating LUDL. loss.
3.3 Two-step Training of Dual Networks
As shown in Fig. 2 the whole training process can be divided into two steps; the first step estimates the uncertainty 0 precisely and the second step generates the final mean value f(y) with the aid from the estimated uncertainty 0 from step1. More specifically, the mean value f(y) and variance 0 are pre-trained by LESU loss during stepl as shown in Fig. 2 (a). After the uncertainty has been estimated, the mean value f(y) network is trained by LUDL loss with variance shown in Fig. (b), while the network of inferring variance is fixed.
Note that the mean value fyy network of step2 starts training from the pre-trained network of stepl. Such partial parameter sharing is a salient property of our proposed dual networks with parallel symmetric attention (27). In theory, we can extend the two-step training into multiple-step training by alternating between the estimation of uncertainty (variance 0) and mean value fv)) Conceptually, an improved estimation of unknown HR image leads to improved estimation of alcatoric uncertainty and vice versa. This line of reasoning will lead to the pursuit of a deep equilibrium model 28 for SISR; but is beyond the scope of this paper.
3.4
Discussions: Why UDL Outperforms GRAM?
To the best of our knowledge, only one work GRAM 21 has studied data uncertainty in SISR, which is the most related to our work. We will discuss connections and differences between proposed UDL and GRAM (21) here. First, both GRAM (21) and our work has found out that applying the traditional uncertainty loss designed for high-level computer vision tasks into SISR task directly results in performance decline. For high-level computer vision tasks, the pixels with higher uncertainty indicates less confidence in final inference, which needs loss attenuation. However, for SISR tasks, the pixels with higher uncertainty (e.g., texture and edge pixels) should be prioritized with larger weights because they are visually more important than pixels smooth regions.
solve this problem,
GRAM 21] proposes to use uncertainty to generate an attention mask that decreases loss attenuation. However, GRAM 21 loss still attenuated when the variance of pixels is high. Thus, GRAM 21 is still inferior to bascline method since it still does not prioritize pixels with high uncertainty.

---

Different from GRAM, we propose an uncertainty- loss to assign the pixels with high variance more weight to prioritize them. Besides, modeling uncertainty under Bayesian framework allows us to leverage sparsity prior for more precise estimation of uncertainty. Ultimately, our proposed method consists of those two technical contributions that achieve better results than baseline methods and outperform GRAM.
4 Experiments
4.1 Experimental Settings
Datasets and Metrics. 800 high-quality (2K resolution) images from the DIV2K dataset [2] have
been used for training. Following EDSR (2] five standard benchmark datasets: Set5 [23], Sct (8),
BSD100(24), Urban 100 [25]. Manga109 [26] are used for testing. Performance evaluation in terms
of of PSNR and SSIM
330 metries is conducted on the luminance (Y) channel only.
Training Setting. We randomly select 16 RGB LR patches sized by 48 x 48 as the inputs. The image patches are randomly rotated by 90º 180% 270 flipped horizontally. The ADAM algorithm []] with 3 = 0.9, 82 0.999, E = 10-8 8 is adopted to optimize the network. The initial learning rate is 10-4 and decreases by half for every 2 x 10s minibatch updates.
Degradation models. To demonstrate the effectiveness of our proposed uncertainty-drive loss in varying degradation scenarios, we have designed the following experiments with two different degradation models. Let BI denotes bicubic downsampling. The second one is BD which uses Gaussian blur followed by nearest downsampling to generate LR images. Specifically, we apply 11 11 sized Gaussian kernel with standard deviation 1.6 for blurring our experiments.
SISR Networks. We choose three different networks to verify the effectiveness of proposed LUDL. loss. The first one is EDSR-S or called baseline network in (2). EDSR-S [2] mainly consists of 16 Resblock with 64 channels, having 1.5. parameters. The second one is DPDNN where denoiser network is U-net under model-guided framework. The last one is big network EDSR(2, consisting of 32 Resblock with 256 channels, having 43M parameters. The analysis of training cost can be found in our supplementary material.
4.2 Ablation Study
Table 2: Average PSNR and SSIM results for BI degradation on five datasets for investigating three
different loss. The best performance is shown in bold. We record the results in 4 x 10 iterations.
Base Model
Scale
Loss
PSNR
23
SSIM
PSNR
SSIM
B
PSNR
BSDT00 24
SSIM
PSNR
Urhan 25
SSIM
PSNR
Mangalin
SSIM
Original
31.61
0.8862
28.22
0.7721
27.30
0.7271
25.23
0.7575
29.31
0.8907
EDSR-S[]
x:4
31.83
0.8895
28.33
0.7754
27.37
0.7297
25.49
0.7665
29.70
0.8959
Cene
31.90
0.8897
28.37
0.7755
27.40
0.7301
25.54
0.7671
29.77
0.8967
To further verify the effectiveness of sparse uncertainty estimation at stepl, we have conducted an
ablation study to compare the final PSNR/SSIM results of LUDI. with LEU or with Lesu at stepl. In our ablation study, we have used x4 bicubic down-sampling degradation on five frequently-used benchmark datasets with EDSR-S backbone [2] As shown in Tab, 2 both LEu+LUDL and LUDL loss have achieved better performance than original loss. Besides, CUSU+ loss obtains better results than CCU+L due to more accurate uncertainty estimation shown in Fig. 3 (c) and (f).
4.3 Analysis of Different Weighted Loss
There are many different weighted loss guided by different weight maps, such Error_map. Gra- dieni_man _ma which can also reveal the challenging pixels. We have conducted experiments with a weighted loss function where the weight a pixel-wise gradient Error_map. The PSNR results of five benchmark datasets for investigating the influence of different weighted loss functions can be summarized in Tab. 3
The HR gradient_man and LR gradien candien_mppaaa denote calculating gradient map from high-resolution (ground truth) images and low-resolution images respectively. The calculation of gradient can be formulated as
((i.j) 1(i+1,j) (((++,,))-((ii))) H(I,) [(i,j((i))1-)))) G(i = (i( Hli,))ll, (1I)
7

---

Table 3: Average PSNR and A PSNR results with BI degradation on five datasets for investigating
the influence of different weighted loss functions. The best performance is shown bold.
Weighted loss
Sct5
A
Sct14
A
BSDI00
A
Urban 100
A
Manga109
A
Baseline
31.61
0.00
28.22
0.00
27.30
0.00
25.25
0.00
29.31
0.00
Uncertainty(Ours)
31.90
0.29
28.37
0.151
27.40
0.10
25.54
0.291
29.77
0.46
Firor map
31.77
0.16f
28.30
0.087
27.35
0.051
25.40
0.151
29.57
0.267
HR [i__grrdieni_ngg map
31.68
0.071
28.27
0.051
27.35
0.051
25.42
0.17f
29.45
0.147
LR gradient map
31.69
0.087
28.29
0.071
27.35
0.051
25.38
0.131
29.50
0.197
where I denotes pixels value and j.j denotes position of pixels. Note that we adjust the scaling functions of Error_map. HR [iR___rddiet__mm and LR___adggge [R__gaddieii_mee to get the best performance.
From the Tab. one can be observed that other weighted loss functions can indeed improve the PSNR results, but only to certain degrees. Comparing four different weight maps, our proposed uncertainty weighted loss function can bring the biggest improvement. Although the Error Error_map map can represent the variance of single pixel, the Error_map lacks semantic information or local information to capture more precise estimation of variance comparing uncertainty. With regard to the gradient map of HR or LR images, those gradient maps only well match the edges of images and have certain correlation to variance. Comparing the visual results of Error_ Error_map, HR HR_ gradientt _ma and LR_ ARgrrddien _ma with uncertainty map, those maps only detect edges of images and fail reflecting complex texture details which are important to final reconstruction performance. Therefore, uncertainty-weighee. loss can is still valuable for achieving the best performance among other weighted maps.
4.4 Analysis of Different Scaling Functions
We have conducted experiments with several various monotonically increasing functions (including linear and non-linear) and the results can be summarized in Tab.
Table 4: Average PSNR and A PSNR results with BI degradation on five datasets for investigating the influence of different scaling functions. The best performance are shown in bold.
Scaling functions
Ser5
A
Sct14
A
BSDTO0
Urban
Manga (09
Bascline
31.61
0.00
28.22
0.00
27.30
0.00
25.25
0.00
29.31
0.00
31,90
0.297
28.37
0.15f
27.40
0.107
25.54
0.297
29.77
0.467
capls)
31,80
0.19t
28.34
0.12
27.40
0.107
25,53
0.281
29.66
0.351
crp(s) (172)
31.86
0.25t
28. 36
0.141
27.41
0.11
25.55
0.301
29.71
0.401
log(s) (og((og(s))
31.89
0.28
28.39
0.171
27.42
0.127
25.57
0.327
29.74
0.437
The best and second-best performances are shown in bold. Overall, four various monotonically increasing functions have achieved better results than the baseline method. The best two scaling functions are linear scaling and log scaling with a slight difference as shown the above table. Since the linear scaling function achieves comparable performance with low computational cost, we advocate this choice this paper.
(dI EDSR S GRAM (e) EDSK Cym. Ouns Figure 4: SISR visual quality comparisons of EDSR-S 2 with different loss function 'Img_004 004'
(a) HR
1D) HK
(c) EDSK.S
(6 Our Uncertainty
and 'Ing 016' from Urban100 (25) (bicubie-downsampling x x4). Best viewed in color.

---

(A) 095 095
(b) IIR
(d) DPENN
(6) DRDNN-GRAM
(o EDDDNN-LURE Ours
(1) Our Uncertainty
Figure 5: SISR visual quality comparisons of DPDNN B3 with different loss function on 'Img_09 095' from BSD100 [24] tbicubi-doonssapling x x4). Best viewed in color.
Table 5: Average PSNR and SSIM results for BI degradation on five benchmark datasets. The best performance is shown bold. Note that LUDI.-Our denotes adopting LESU stepl and LUDI. at step2 for simplicity.
Set5
23
Setl4
8
BSDT0U
124]
Urban PSNR
100
22
Manga (09) 226
Base Model
Scale
Loss
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
Original
37.66
0.8594
33.22
0.9146
31.95
0.8969
30.71
0,9205
37.79
0,9752
EDSR-S 2
x2
GRAM [2]
37.48
0.9589
32.99
0.9126
31.76
0.8946
30.11
0,9134
37.38
0,9739
-uus
37.95
0,9604
33.50
0.9165
32.13
0.8991
31.54
0,9304
38.38
0.9767
Original
37.75
0.9600
33.30
0.9150
52,09
0.8950
31.50
0.9220
DPDNN 3
x2
GRAM (2]
37.74
0.9597
33.27
0.9148
31.98
0.8973
30.97
0.9238
38.14 14
0.9758
Cure-Ours
38.00
0.9605
33.63
0.9176
32.16
0.8995
31.72
0.9331
38.55
0.9769
Original
38.11
0.9612
33.92
0.9195
32.32
0.9013
32,93
0.9351
39.10
0,9773
EDSR 2
x2
GRAM [21
37.87
0.9604
33.43
0.9164
32.08
0.8990
31.46 .44
0.9301
37,91
0.9765
Cuna-Ours
38.29
0.9615
34.14
0.9236
32.40
0,9027
32.99
0.9446
39.53
0.9787
Original
0.9231
0.8352
0.7996
32.52
0,9369
EDSR-S x
x3
GRAM 2
33.27
0.9178
29.60
0.8298
28.66 28.60
0.7936
26.52
0.8142
31.14
0,9258
-uuse-Ors
34.15
0,9251
30.15
0,8388
28.99
0.8021
27.72
0.8430
32.97
0.9406
OFiginal
33.93
0.9240
30.02
0.8360
29,00
0.8010
27.61
0.84211
DPDNN 3
x3
GRAM 211
33.92
0.9241
30.00
0.8362
28.86
0.8000
27.37
0.8353
32.41
0.9373
Cure-Ours
0.9267
30.31
0.8419
0.8047
33.27
0.9438
Original
34,65
0.9280
30.52
0.8462
29.25
0.8043
28.80
0.8653
34.17
0,9476
EDSR 2
x3
GRAM 21
34.34
0.9270
30.28
0.8412
29.07
0.8044
27.98
0.8789
33.32
0.9432
CUDL-Ours
34,83
0,9312
30.69
0.8497
29.28
0.8109
28.99
0.8697
34.63
0,9502
Original
31.61
0.8862
0.7721
27.30
0.7271
0.7575
0.8907
EDSR-S $
2
x4
GRAM 23
31.08
0.8787
27.89
0.7670
27.12
0.7229
24.81
0.7429
18
0.8762
-uus
31.90
0.8897
28.37
0.7755
27.40
0.7301
25.54
0.7671
29.77
0.8967
Original
31.72
0.8390
28. 28
0.7730
27.44
0.7290
25.53
0,7680
DPDNN 3
x4
GRAM (2]
31.89
0.8913
28.37
0.7772
27.41
0.7314
23
0,7708
29.70
0.9003
Cupe
32.20
0.8944
28.60
0.7819
27.56
0.7356
0.7862
0.9082
Original
32.46
0.8968
28.80
0,7876
27.71
0,7420
26.64
0.8033
31.02
0,9148
EDSR
GRAM [1]
32.32
0.8971
28.73
0.7858
27.66
0.7395
26.35
0.7955
30.73
0,9125
Cupu -Ours
32.59
0.8998
28.87
0.7889
27.78
0.7431
26.75
0.8054
31,24
0.9167
4.5 Results with BI Degradation Model
For bicubic downsampling (BI), we have compared proposed LUDL loss function with GRAM 21 and original loss functions such as MSE Li three different SISR networks. The average PSNR and SSIM results in Tab. (bb.5)ees are cited from corresponding papers or retrained from officially released code. It is casy to see that our proposed CUDI. loss function is superior to GRAM 21 and original
Table Average PSNR and SSIM results for BD degradation on five benchmark datasets. The best performance is shown in bold. Note that -Ours denotes adopting LESU at step1 and LUDL at step2 for simplicity.
Set5
23
Setl4
8
BSDTO0
24
Urban PSNR
TO0 25
Manga (09
26
Base Model
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
Original
31.70
0.8903
28.37
0.7778
27.37
0.7320
25.77
0.7789
29,83
0.9014
EDSR-S
GRAM 21
30.98
0.8791
27.85
0.7667
27.05
0.7225
24.79
0.7452
28.12
0.8773
Eupx-Ours
31.97
0.8927
28.45
0.7793
27.41
0.7321
25.95
0.7842
30.18
0,9053
Original
31.56
0.8923
28.38
0.7780
27.36
0.7311
25.82
0.7812
29.77
0.9033
DPDNN
GRAM 21
31.75
0.8913
28.33
0.7765
0.7302
25.62
0.7739
29.55
0.9003
Eupe-Ours
32,03
0.8949
28.60
0.7828
27.48
0.7355
26.21
0.7931
30.35
0,9097
Original
32.17
0.8975
28.65
0.7856
27.59
0.7400
26.56
0.8043
30.66
0.9133
FDSR
GRAM [2]]
32.13
0.8963
28.57
0.7822
27.49
0.7362
26.19
0.7916
30.48
0.9097
-Ours
32.37
0.8986
27.74
0.7867
27.62
0.7407
26.65
0.8065
30.81
0.9149

---

loss functions such as MSE or L1 in terms of PSNR and SSIM values. Note that the improvements achieved by our proposed method do not bring any additional computation cost during testing time. Comparing EDSR-S (1.5M parameters) with EDSR (43M parameters), our proposed LUDI. can bring lightweight networks with more greater performance improvements than big ones. The visual image comparison results are reported in Fig. and Fig. 5 As shown in Fig. 4 our proposed LUDL has recovered with fewer visible artifacts (e.g.. the circular pattern the roof and the lines on the glassy surface) than original loss and GRAM (21]. Fig. (f) depicts the uncertainty learned by LUDL, revealing the challenging pixels with poor reconstruction performance. From Fig. [ vertical center-lin line of window has been recover and (f), while DPDNN and OPDNN-GRAM [2] failed to discern shown (c) and (d) respectively. More visual comparisons can be found in supplementary material.
clear with precisely estimated uncertainty shown in (e)
4.6 Results with BD Degradation Model
For blur downsampling (BD), we have compared proposed LUDL loss function with GRAM (1 and original loss functions such as MSE or L1 on three different baseline networks. The average PSNR and SSIM results in Tab. are retrained from officially released code. It is easy
that propose LUDI loss function is superior to GRAM (21) and original loss functions such as MSE or L1 in terms of PSNR and SSIM values. The visual image comparison results of BD degradation
reported in Fig. and Fig. 7 Note that the BD degradation involves Gaussian blur, increasing difficulty in recovering structure patterns. From Fig- we can see that SR result (Fig. (e)) of 'Img 109" is the closest to that of the ground-truth. In another challenging image ('Img 078' from Urban10M (25)), our method can recover much more reliable textured details as shown Fig. (e): while all other methods have severe aliasing artifacts (i.e., distorted tile patterns). The visual quality improvement achieved by CUDL mainly due the fact that our proposed method makes full use of the captured uncertainty to train deep networks focusing on the challenging pixels with high uncertainty. More visual comparisons can be found our supplementary material.
(a) Img_109
[bJ HK
(e) DPONN
(d) DPENN-GRAM
(c) DPUNN--UDL -Ours
(f) Oar Uncertainty
Figure 6: SISR visual quality comparisons of DPDNN B with different loss function 'Img_109 109' from Manga109 (26 (blur-downsampling x4). Best viewed color.
(al Ing_078
[b] HK
(c) BUSK
(d) EUSR-GRAM
(e) EUSK-21mx--00
(i) Our Uncertainy
Figure 7: SISR visual quality comparisons EDSR 2 with different loss function on 'Img_07 from Urban (25] (blu--dowssampling x)). Best viewed in color.
Conclusion
In this paper, we propose new adaptive weighted loss LUDL for SISR to train SISR networks focusing on challenging pixels with high uncertainty (c.g., textured and edge pixels). Specifically, variance estimation is introduced into SISR so that the high-resolution images (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, modeling uncertainty under Bayesian framework allows us leverage sparsity prior for a more precise estimation of uncertainty. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance to visual quality. For the first time, we demonstrate that such uncertainty-driver loss can achieve better results than MSE or Li loss. Experimental results three popular SISR networks show that our proposed uncertainty-driven loss achieved better PSNR performance than traditional loss functions without any increased computation during testing.
10

---

Acknowledgement
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0101400 and the Natural Science Foundation of China under Grant 61991451, Grant 61632019, Grant 61621005, and Grant 61836008. Xin Li's work is partially supported by the NSF under grants I15-1951504 and DAC-1140855 the DoJ/NU under grant NIJ 2018-75-CX-0032, and the WV Higher Education Policy Commission Grant (HEPC.dsr. 18.5).
References
[1] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184-199, 2014.
[2] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132-1140, 2017.
[3] Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fanglang Wu, and Xiaotong Lu. Denoising prior driven deep neural network for image restoration. JEEE Transactions on Pattern Analysis and Machine Intelligence, 41(10):2305 2318, 2019.
[4] Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu, Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference Computer Vision (ECCV). pages 286-301, 2018.
[5] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 11065-11074, 2019.
[6] Qian Ning, Weisheng Dong, Guangming Shi, Leida Li, and Xin Li. Accurate and lightweight image super-resolution with model-guided deep unfolding network. IEEE Journal of Selected Topics in Signal Processing, 1522)220--22,, 2021. doi: 10.1 109//STSS2222003077561
[7] Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao. Zeshuai Deng, Yanwu Xu. and Mingkui Tan. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5407-5416, 2020.
[8] Roman Zeyde. Michael Elad, and Matan Protter. On single image scale-up using sparse-representationn. In Proceedings of the 7th international conference Curves and Surfaces, pages 711-730, 2010.
[9] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694 694-711. Springer, 2016.
[10] Mario AT Figueiredo. Adaptive sparseness using jeffreys prior. NIPS, pages 697-704, 2001.
[11] Qian Ning, Weisheng Dong, Fangfang Wu, Jinjian Wu, Jie Lin, and Guangming Shi. Spatiial-emporal gaussian scale mixture modeling for foreground estimation. Proceedings of the AAA/ Conference on Artificial Intelligence, 34(07):1 1791- 11798, Apr. 2020, doi: 10. (609/aaii.344007 .6551. URL https
V/ojs. wat.saaaa...gnnneeeee
php/ AAAI wat.a//aallee/.oooo.ooo.ooa /651611
[12] Paul W Goldberg, Christopher KI Williams, and Christopher Bishop. Regression with input-dependent noise: A gaussian process treatment. Advances in neural information processing systems, 10:493 4999 1997.
[13] WA Wright. Bayesian approach to neural-netword modeling with input uncertainty. IEEE Transactions on Newral Networks, 10(6): 1999.
[14] Christopher M Bishop and Cazhaow S Qazaz. Regression with input-dependent noise: A bayesian treatment. Advances neural information processing systems, pages 347-353, 1997.
[15] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for com- puter vision? In I. Guyon, U. V. Luxburg. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL httpps:://ppocceccc 2650d608 9 6:14 0c 5bw.88822b882268 b5t.ceee.55552222555-222--00
ngs wew.eeeeddinnnununeneeeeeeee www.neurr....papeeeeeee pttp://apppp///////111...//
[16] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- architecture for image sogmentation. IEEE transactions on pattern analysis and machine intelligence, 39 (12):2481224) 1211112245,,22 2017.
[17] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5710-
5719, 2020.
11

---

[18] Yingjie Gu, Zhong Jin, and Steve C Chiu. Active learning combining uncertainty and diversity for multi-class image classification. IET Computer Vision 93)):00-407, 2015.
[19] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the JEEE/CVE International Conference on Computer Vision, pages 6902 6911, 2019.
[20] Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does matter? Structural safety, 31(2): 105-112, 2009.
[21] Changwoo Lee and Ki-Seok Chung. Gram: Gradient rescaling attention model for data uncertainty estimation in single image super resolution. In 2019 18th IEEE International Conference Machine Learning And Applications (ICMLA), pages 13. IEEE, 2019.
[22] Aupendu Kar and Prabir Kumar Biswas. Fast bayesian uncertainty estimation and reduction of batch normalized single image super-resolution network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4957-4966, June 2021.
[23] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based nonnegative neighbor embedding. In British Machine Vision Conference 2012, pages 1-10, 2012.
[24] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume pages 416-423, 2001.
[25] Jia-Bin Bin Huang. Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-cxemplars. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5197-5206, 2015.
[26] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 76 (20::21811221838, 2017.
[27] Yingqian Wang, Xinyi Ying, Longguang Wang, Jungang Yang, Wei An, and Yulan Guo. Symmetric parallax attention for stereo image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 766-775, June 2021.
[28] Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. arXiv preprint arXiv:21022 102.07944, 2021.
[29] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops. pages 1 114-125, 2017.
[30] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 13(4):600-61) 2004.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
URLhttp: /ttp://///xivv/ org/ abc//bb//4440 412. 6980
12


Reviews:
[{"review_id": "y4aibnidRvN", "reviewer": "Reviewer_WXXg", "summary": "The authors proposed a new loss function guiding the SISR network focus more on the uncertain areas of the images. Specifically, the uncertainty map is obtained by the network trained with the proposed L_esu loss enforcing the sparsity in the map, and the final SISR network is initialized from the previous steps, and finetuned with the uncertainty-guided loss. The idea of using uncertainty guidance to train a SISR network will not increase the computational burden during inference, and the performance is demonstrated to be better than the baseline. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros,\n\n- Interesting idea of estimating the uncertainty map and adding the sparsity prior. \n- Better performance of using the guided training with the newly-proposed loss. \n\nCons,\n\n- It demonstrates the uncertainty map is mostly focusing on edges and texture regions of the image. I wonder whether the proposed method estimating the uncertainly map will be equivalent to high-pass filtering or edge detector for image preprocessing. What if there are high-level noises existing in the input images, will it highly influence the uncertainty map?\n- Given a general-purpose loss function potentially good for other image restoration tasks, I wonder what it performs on more SOTA models, and whether it will perform consistently better than the baseline. Current evaluation is not enough with only three models and two tasks. Since it seems fairly easy to plug the training framework into any models, why not run more experiments to make the model more solid?\n\nMy main concern is there can be multiple similar alternatives for the uncertainty estimation, and the authors had better clarify the motivations and compare the current way with some simpler methods. Besides, the evaluated models are not enough to show the effectiveness of the model. Some potential generalization ability is not shown in the limited experiments. \n\nHowever, the simple loss function is intuitive and somehow brings some improvement on some specific models, I currently tend to accept the paper due to its strong intuition and clear presentation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the above comments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.5"}, "extra_text": {}}, {"review_id": "WqQ141LBrh", "reviewer": "Reviewer_oVep", "summary": "This paper proposed a new loss for single image super-resolution by considering the uncertainty of each pixel within an image. Specifically, the authors claimed that pixels with larger uncertainty, such as texture and edge, should be more emphasized, and thus designed a loss function to achieve this goal. Experiments have been conducted to evaluated the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of emphasizing pixels with larger uncertainty is new, and seems effective. Experiments have also showed clear improvements as compared with baselines.\n\nHowever, I do not think technical part this work is clearly clarified. Specifically,\n1. There are many priors can induce sparsity other than Jeffrey's prior, and thus it should be justify why adopt this prior. \n\n2. Also notice that, comparing Eq. (6) and Eq. (9), the difference only exists in the scalar multiplied to $s_i$. Therefore, it is natural to ask what the performance could be if we change this scalar to other values, which is also valid and can be derived similarly as Eq. (7) and Eq. (8), in the sense of generalize priors.\n\n3. The final loss function in Eq. (10) is somewhat heuristic, since there are many ways to set $\\hat{s}_i$ provided monotonically non-decreasing with respect to $s_i$.\n\nBesides, many of the mathematics are not rigorous. For example,\n1. In Eq. (3), the authors assume $\\epsilon$ to be standard normal. However, according to equations after it, e.g., Eq. (4), $\\epsilon$ should follow a Laplace distribution.\n\n2. Eq. (7) is not correct, since $p(x_i|y_i,\\theta_i)p(\\theta_i)$ results in $p(x_i,\\theta_i|y_i)$ according to conditional probability rule, but not $p(x_i|y_i)$.\n\n3. Jeffrey's prior can only be written as $p(w)\\propto\\frac{1}{w}$, since it is not a valid distribution, and thus in Eq. (7) the equal sign after $p(x_i|y_i,\\theta_i)p(\\theta_i)$ should be replaced by \"$\\propto$\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not mention the limitations. As I can see, the major limitation of this method in practice is that it requires a two-step training, which increase the computational complexity.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours."}, "extra_text": {}}, {"review_id": "CIEMJgHrej", "reviewer": "Reviewer_zyeM", "summary": "This paper proposes a new loss function for the training of single image super-resolution. The proposed method is motivated by the uncertainty of predicting different pixels. Firstly, the uncertainty of each pixel in super-resolution is estimated. This uncertainty (variance) was used during training that the pixels with higher uncertainty were weighted with higher weights. A two-step training scheme was designed. The experiments show good results compared with other uncertainty-driven methods, such as GRAM.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major concerns:\n\n1. The novelty of this paper is somehow limited. The author describes a probability-based framework to explain the uncertainty in super-resolution. However, this part of the statement is common. Describe super-resolution under Bayesian statistics introduce no novelty. The Jeffrey's prior part is interesting but the author did not explain too much. The whole method is similar to training SR with weighted data (hard samples have higher weights). The relationship between the proposed method and weighted training is not clear (there is no expariments and statements about it).\n\n2. In the experiment section, the author only presents comparison between the proposed method and GARM. More experiments are expected. Can we weight the pixel using other methods? such as calculate the gradient, high gradient indicates dense textures. Or weight the loss for each pixel based on Wevelet or DCT? Is there any difference in using different methods to weight loss? Why is the proposed weighting method better?\n\n3. More explaination and experiments about the Jeffery's prior and its function in the proposed method is expected.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "A68MpJL2rc", "reviewer": "Reviewer_H49w", "summary": "This paper proposes a novel training scheme for end-to-end single image\nsuper-resolution (SISR) with neural networks. The proposed scheme has two\nsteps. In the first step, a neural network is used to predict per-pixel\naleatoric uncertainty. In the second step, a second neural network is trained,\nand the uncertainty estimated in the first step is used to weigh the per-pixel\nrestoration loss, so that uncertain pixels are more heavily penalized. Because\nsuper-resolving textured regions and edges is typically more uncertain, the\nauthors claim that by making the model focus more on those regions, better\nsuper-resolution results are obtained. Experimental results show consistent by\nmild improvements over the baseline. In my opinion, the theoretical developments\nneed a stronger justification and their presentation needs to be improved.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Super-resolution is a one-to-many mapping so the outcome is inherently\n  uncertain. It is good to see uncertainty being modeled in deep learning based\n  SISR. This is a very recent trend that started with [21]. It also makes sense\n  to me that the authors take into account the problem's heteorscedasticity\n  since one expects uniform regions to exhibit less uncertainty in the output\n  than texture or edges.\n- Experimentally, the authors show  that their two-step training approach\n  obtains a better perfoming model than the baseline (by 0.1 to 0.3db PSNR on\n  average). Qualitative results in two examples show improvements over the\n  baseline.\n\n\nCons:\n- Since the goal of the work is to more strongly penalize textured and edge\n  regions, a natural and simple baseline would be to optimize for the difference\n  of image gradients (or other low-level fiters), or to weigh the penalization\n  according to gradient magnitude. This would be much cheaper than having to\n  recognize texture and edges with a secondary neural network.\n- The theoretical developments in Section 3.1 have weak points that need to be\n  clarified:\n  - In lines 132, 137 authors say that they study only epistemic uncertainty,\n    but it seems to me that what is being modeled is the aleatoric uncertainty:\n    the formulation in (3) models heteroscedastic noise in the\n    observations. Uncertainty of the model parameters (epistemic) does not\n    appear in the presentation, to the best of my understanding.\n  - In line 137 it is said that the observation noise in (3) is Gaussian but the\n    likelihood equation (4) corresponds to Laplacian noise.\n  - A Jeffrey's prior is introduced on one of the model's output (the estimated\n    variance), by simply multiplying the likelihood function by one over the\n    variance. I believe the probabilistic model intended here needs further\n    clarification. Is the goal here to do a MAP formulation for this variance?\n    Equation (7) is certainly confusing. The leftmost equation is invalid unless\n    \\theta_i is integrated over, and is independent of y_i. But then the\n    probabilistic interpretation would not be clear, at least to me, since\n    \\theta_i is itself a function of y_i. Also, by multiplying the likelihood\n    function by a factor it is no longer a probability over the x_i. I think all\n    this deserves a better theoretical justification and an improved\n    presentation.\n  - There is no probabilistic justification for the second training step\n    (weighing the reconstruction loss by the uncertainty).\n- Although Figures 6 and 7 help to show the relative improvement of the proposed\n  approach, it would help to include more examples and more state-of-the-art\n  SISR methods in the comparison, so one can evaluate the validity of the\n  approach as a general SISR method.\n\nMinor:\n- References [14] and [22] (Kendall and Gal 2017) are the same. This is a\n  central reference to this work.\n\n\n\nIn my opinion, the weaknesses listed above outweight the submission's strengths.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new weighted loss function for the training of single image super-resolution. The proposed scheme involves two steps. In the first step, the authors use a Bayesian estimation framework to predict mean and variance for each pixel in the high resolution image (not just a point estimate). Then the authors train a second neural network by weighting the reconstruction error according to the estimated variance. The intuition motivating this idea is that prediction errors on pixels with larger uncertainty (e.g. texture and edges) should be upweighted. Experimental evaluation shows that the new scheme is able to outperform the baseline.

The authors provided a constructive and high quality rebuttal, incorporating new experiments (including baselines and results using different monotonic functions to define the weighting scheme). However, all reviewers still consider the paper not ready to publish at NeurIPS.

Reviewers zyeM, oVep and H49w had concerns regarding the precision of the mathematical formulation. Specifically, several comments on the derivations in Section 3.1 and the lack of a probabilistic motivation for the second estimation step. This led Reviewer WXXg to reduce the score to 4, as she/he believes that the paper would benefit from a second revision. Reviewer H49w also expressed the same opinion.

Reviewer zyeM provided a score of 5 (updated from 4 after the rebuttal), although she/he did not change it in the system. While Reviewer zyeM considers the general direction of the work interesting (the investigation of weighting schemes for the SISR problem) and appreciated the detailed response provided by the authors, she/he still recommends rejecting the paper. The main reason is the lack of significant technical novelty in their approach. 

Reviewers H49w and zyeM, suggested incorporating simple baselines that could identify the textured regions in the image without the need of training a full system. The authors added in their rebuttal convincing baselines, showing that the proposed approach can lead to singnigicantly better results. 

All four reviewers recommend rejecting the work. The AC agrees with the concerns raised by the reviewers and finds no basis for overruling their recommendation. 

Decision: Accept (Poster)