Uncertainty-Driven Loss for Single Image Super-Resolution
Authors: Qian Ning
Weisheng Dong
Xin Li
Jinjian Wu
Guangming Shi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Uncertainty-Driven Loss
Uncertainty Learning
Single Image Super-Resolution

Abstract: In low-level vision such as single image super-resolution (SISR), traditional MSE or L_1 loss function treats every pixel equally with the assumption that the importance of all pixels is the same. However, it has been long recognized that texture and edge areas carry more important visual information than smooth areas in photographic images. How to achieve such spatial adaptation in a principled manner has been an open problem in both traditional model-based and modern learning-based approaches toward SISR. In this paper, we propose a new adaptive weighted loss for SISR to train deep networks focusing on challenging situations such as textured and edge pixels with high uncertainty. Specifically, we introduce variance estimation characterizing the uncertainty on a pixel-by-pixel basis into SISR solutions so the targeted pixels in a high-resolution image (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, uncertainty estimation allows us to leverage conventional wisdom such as sparsity prior for regularizing SISR solutions. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance to visual quality. For the first time, we demonstrate that such uncertainty-driven loss can achieve better results than MSE or L_1 loss for a wide range of network architectures. Experimental results on three popular SISR networks show that our proposed uncertainty-driven loss has achieved better PSNR performance than traditional loss functions without any increased computation during testing. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/UDL-SR.htm

Uncertainty-Driven Loss for Single Image Super-Resolution
Qian Ning', Weisheng Dong'; Xin Li', Jinjian Wu', Guangming Shi!
Schooo of Artificial Intelligence, Xidian University, Xi' an 710071, China
Lane Dep. of CSEE, West Virginia University, Morgantown WV 26506, USA
ningqiani antutu. x di an edu. cn, (wsdong, jinjian. we @ma i xidi an edu. cn xin 11 @mail 1 edu, gmshi @xidian edu.cn
Abstract
In low-level vision such as single image super-resolution (SISR), traditional MSE or C loss function treats every pixel equally with the assumption that the importance all pixels the same. However, has been long recognized that texture and edge areas carry more important visual information than smooth areas in photographic images. How to achieve such spatial adaptation in a principled manner has been open problem in both traditional model-based and modern learning-bai based approaches toward SISR. In this paper, we propose a new adaptive weighted loss for SISR to train deep networks focusing challenging situations such as textured and edge pixels with high uncertainty. Specifically, we introduce variance estimation characterizing the uncertainty on pixel-by-pixel basis into SISR solutions so the targeted pixels in high-resolution image (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, uncertainty estimation allows us to leverage conventional wisdom such as sparsity prior for regularizing SISR solutions. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance visual quality. For the first time, demonstrate that such mmceraintt--rrvve loss can achieve better results than MSE or L1 loss for wide range of network architectures. Experimental results on three popular SISR networks show that our proposed incertainty-driverr loss has achieved better PSNR performance than traditional loss functions without any increased computation during testing. The code is available at httpp:://se.xidiiatt edu.
1
Introduction
Single image super-resolution (SISR) aims at reconstructing high-resolution (HR) images from their corresponding degraded low-resolution (LR) images. Since the publication of super-resolution with convolutional neural network (SRCNN) 1 there has been a flurry of works on deep learning-based
approaches toward SISR e.g., EDSR 2), DPDNN [3, RCAN (4) SAN (5), and MoG-DUN [6]. The unifying theme along this line of research appears to be that deeper, bigger, and more complex networks can achieve improved SISR performance by facilitating the reconstruction of high-frequency details such as textures and edges in photographic images. Such improvement has been achieved by novel network architectures (c.g., skip connections (2)), new attention mechanism (c.g., residue channel attention [4), and closed-loop supervision (7]. Surprisingly. most of these existing methods have adopted MSE or L, loss optimize the parameters of networks.
The commonly used practice, such as MSE or L1 loss, treats every pixel equally regardless of whether the pixel is in texture/edge regions smooth areas. The optimality of such non-adaptive loss function
Conresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Go bug. 007 from Setl4
(b) TIR image
(c) SR image f(x)
(d) Difference map
f(y))
Figure 1: Illustraiion of the difference (d) between HR image (b) and SR image (c) reconstructed by EDSR network [2] on dataset Set14 (8. The image reconstructed by EDSR network is shown in (c)
and (d) shows the absolute difference between the HR image and SR image. Best viewed in color.
has been questioned in the literature of SISR calling for the proposition of perceptual loss function
(e.g., [9). From a Bayesian perspective, the assumption underlying the MSE or L1 loss is that each pixel obeys the independent and identically distribution with the same variance. Taking L₁ loss as an example, the likelihood of all pixels in an image can be formulated as
II cexp(- ||x00 f(W) (1))),
p( y. W)
where and y denote the pair of HR and LR image, (.) denotes an arbitrary SISR network parameterized by W, and c, a denote spatially invariant constants. However, such assumption of stationarity or spatial invariance of image prior model is invalid for photographic images in the real world. For instance, if one compares the ground-truth (HR image) and the SR image reconstructed by EDSR (2 shown in Fig. [ (c). it can be observed that texture areas (c.g.. hair of baboon) are not restored as good as smooth areas (c.g.. nose of baboon). Fig. [ (d) depicts the absolute difference between the HR image and reconstructed SR image, from which we can observe spatial variation of the difference map. Such observation implies that the uncertainty texture and edge areas as characterized by the variance is much larger than that in smooth areas. How to address such uncertainty-dree driiee loss for SISR sets up the stage for this paper.
In this paper, we propose new adaptive weighted loss (uncertainty-drive) loss) for SISR by assigning texture and edge areas with higher weights during the training process. Unlike previous work of perceptual loss (9] focusing on characterizing content and style consistency, we target at explicitly estimating the variance field underlying the unknown HR image in the first step, which can be exploited consequence of our two-step learning approach is that delivers not only higher visual quality but also improved objective performance such PSNR and SSIM. Moreover, uncertainty estimation perspective allows us to easily incorporate existing models such as Jefferey's 's prior 10, 11 into the proposed SISR solution. It follows that the network training boils down two sequential steps in which the variance map is estimated from the first step and serves as the attention signal for the second step. The main technical contributions summarized as follows.
auxiliary signal for guiding the SISR solution in the second step. A direct
Uncertainty modeling and estimation. We propose to cast SISR into a Bayesian estimation framework under which SR image (mean) and uncertainty (variance) are derived simulta- neously. Unlike previous works in which pixels with large uncertainty are attenuated for high-level vision tasks, we advocate to prioritize them for low-level vision tasks such as SISR.
Uncertainty-driee driven loss (UDL). The estimation of variance map facilitates the training of SISR network by dividing it into two steps. In the first step, an estimating sparsity uncertainty (ESU) loss function was derived from the classical Jeffrey's prior to estimate the variance map. In the second step. the estimated variance map serves as the guidance signal leading to adaptive weighted loss named uncertainty-die
loss LUDI.-
Universality of UDL. The proposed uncertainty loss can easily be employed in any existing SISR network to improve performance and do not increase any additional computation cost during testing.
Experimental results on three different baseline networks show that our proposed uncertainty- driven loss has achieved better PSNR performance than traditional MSE or C1 loss.

---

2 Related Work
2.1 Uncertainty in Deep Learning
Many works [12 have introduced uncertainty into the regression with input-dependent noises problems, and studied the nature and behavior of uncertainty for a long time. More recently, modeling uncertainty in deep learning have improved the performance and robustness of deep networks in many computer vision tasks [15H17] such as image classification [18], image segmentation [15] and face recognition [7 19 The uncertainty in deep learning can be roughly divided into two categories [20]. Epistemic/model uncertainty describes how much the model is uncertain about its predictions. Another type aleatoric/data uncertainty which refers to noise inherent in observation data. In (15), they presented a Bayesian deep learning framework combining alcatoric uncertainty with epistemic uncertainty for per-pixel semantic segmentation and depth regression tasks. Chang al. [17] investigated the data uncertainty with estimated mean and variance in face recognition. Those uncertainty-baat based loss function proposed by those works 17 can be summarized as
N
D
f(y)) ll2 1 In o
C
2 20²
where f14) and of denote the learned mean and variance respectively. Using above loss function indeed improved their robustness to noisy data. In those tasks, the pixels with high uncertainty were regarded as unreliable pixels which would bear loss attenuation. On the contrary, in SISR tasks, the pixels with high uncertainty (e.g., complex texture or edge areas) should be prioritized since those regions visually more important than pixels in smooth areas. That can explain why applying above loss into SISR directly leads performance decline.
2.2 Modeling Uncertainty for SISR
To the best of our knowledge, only two works have studied the behavior of uncertainty for SISR in the open literature. [22] used
uncertainty to analyze SISR uncertainty, improving the robustness of the network against adversarial attack. The most recent advance related to our work is Gradient Rescaling Attention Model (GRAM) [21] which analyses the effect of aleatoric/data uncertainty on SISR reconstruction. By decreasing the loss attenuation of large variance pixels, GRAM achieves better results than applying above uncertainty loss into SISR directly. However, GRAM 21 loss remains attenuated when the variance of pixels is high, which contradicts the intuition of prioritizing texture and edge pixels. Thus, GRAM [21] is still inferior baseline methods since the proposed method fails to prioritize the pixels large variance. Different from GRAM, we propose a novel uncertainty-driven loss (UDL) to enforce the network concentrating more on the pixels with large variance aiming at better reconstruction of texture and edge regions. By quantifying the uncertainty in SISR under deep Bayesian framework, our proposed method has achieved better results than baseline methods.
3 Methodology
Unlike traditional MSE or Li loss treating every pixel equally, the proposed new adaptive weighted loss for SISR aims at prioritizing texture and edge pixels that are visually more important than pixels in smooth areas. Toward this objective, we first introduce approach of estimating intermediate results of SR image (mean) and uncertainty (variance) simultancously in SISR. Then, with Jeffrey's s prior term, regularized approach of estimating sparse uncertainty is proposed for more accurate uncertainty estimation. An important new insight brought by this paper is that unlike high-level vision tasks where pixels with large uncertainty are assigned lower weights to attenuate their impact (T5), one should prioritize these pixels in low-level vision tasks such as SISR. Such observation implies that the attenuation of weighting coefficients in loss function needs to be properly translated into the attention mechanism given the specific vision problem as the context.
In previous study [5] it has been shown that explicitly representing aleatoric uncertainty can lead to performance and robustness improvement to noise data in high-level vision tasks such as image segmentation. Such improvement can be explained away by attenuating the weights of pixels with
3

---

large uncertainty. However, attenuation has to go the opposite direction in low-level vision tasks such as SISR i.e. larger weights should be assigned the pixels with high uncertainty (e.g., texture and edge pixels) because they are visually more important than pixels in smooth regions. It should be noted that existing work such as gradient rescaling strategy in GRAM [2]] fails to recognize such difference and does not prioritize pixels with high uncertainty. In this paper, we propose new adaptive weighted loss named uncertainty-driven loss (UDL) for properly turning attenuation into attention for SISR.
Step1
Upsampling Network
f(y)
Conv
Conv
Conv
Conv
ESU
y
Neaeett
Upsumpling
Backbone Network
Conv
ELU
Step2
Upsampling Network
f(y)
Conv
Conv
Conv
Conv
Lupe
y
(b)
Buckbone Network
Figure 2: The overview of training SISR network with proposed LUDL loss. The whole training process can divided into two steps; the first step estimates the uncertainty precisely and the second step generates the final mean value (t) In stepl shown in (a), the mean value f(y) and variance are pretrained by LESU loss. During step2, as shown in (b), the mean value ('y) network trained
by LUDI. loss, while the network of inferring variance 0 is fixed. Note that the mean value ( network of step2 starts training from the pretrained network of stepl. The Nearest Upsampling denotes interpolation operator.
3.1 Estimating Uncertainty (EU) in SISR.
As discussed in [15]. there are two classes of uncertainty in Bayesian modeling: aleatoric uncertainty capturing noise inherent in observation data and epistemic uncertainty accounting for uncertainty of model about its predictions. We opt to study the former (aleatoric uncertainty) and explore its application into SISR by designing new uncertainty-drive order to better quantify aleatoric uncertainty in SISR, we use y: æ; denote the low-resolution (LR) image and the corresponding high-resolution (HR) image respectively. Let f(-) denotes an arbitrary SISR network and the aleatoric uncertainty can denoted by an additive term 0,. This way, the overall observation model can be formulated as
loss (UDL) functions in this paper. In
fv:) + O;,
where represents the Laplace distribution with zero-mean and unit- varianee Existing deep-learning based SISR methods target at training network to Iearn the SR image (mean) f(y:) only. To more accurately characterize alcatoric uncertainty for SISR, we propose estimate not only the SR image (mean) f(y,) but also the uncertainty (variance) 0, simultancously.
For given LR image y: and corresponding HR image æ; Laplace distribution is assumed for characterizing the likelihood function by
|| - f(1))l1)
p(a;, 0.ly))
2 20; exp(
(4)
0,
where f(y:) and 0 denote the SR image (mean) and the uncertainty (variance) which are learned by deep neural networks (DNNs) respectively. Then, the log likelihood can be formulated as follows,
In p(x;, 0,1y:) = x f(y)ll In 0, - In 2
(5)
0
The most commonly used loss function in SISR Li loss which refers to Laplace distribution.

---

0
(A)
(6) EUSR-S
(c) HUSR-S-Cru
EDSR-S- SUDSS---CggS Cggu
Cn: Lacertuatty
(8) PESU Uncertainty
Figure 3: SISR visual quality comparisons of EDSR-S 2 with different loss function on 'lmg_005 005'
from Set5 24 (bicubie-downsampling x4). Best viewed in color.
For numerical stability, we train the networks to estimate log variance 8; = In 0, as shown in Fig. 2 (a). At last, the maximum likelihood estimation of (5 can be reformulated the minimization of following loss function for estimating uncertainty (EU) in SISR.
LEU Σ exp( 8:) ||:0 f(y))l
Jeffrey's Prior for Estimating Sparse Uncertainty (ESU) in SISR. The loss function LEU includes two terms; the first one associated with fidelity term and the second one prevents the network from predicting infinite uncertainty for all pixels. Those two terms reach equilibrium but there is no prior that imposed on the uncertainty estimation. Therefore, based on the observation that the uncertainty we propose impose Jeffrey's prior [T0]
is sparse in view of the whole image as shown in Fig. [ on uncertainty 0 which can be expressed as p( p(w) x w
p(a,, 0, y.) = p(x.///.... (.)) x 20: exp( >
1
y
1
2
f(y
exp(
0,
0,
2
O,
Then the log likelihood and loss function can be separately formulated as follows,
| f(y.)1l
In p( y.)
2 In 0: In
(8)
0:
1 CESU =
N Σ exp( (s:) ||x; - -f(y))lll + st
(9)
The limitations of LEU and CESU loss. Applying LEu and CESU loss leads to more accurate es- timation of uncertainty (variance field), but counter-intuitively, they do not directly improve the performance SISR. We have conducted experiments comparing those three different loss functions to verify the above claim. As shown in Tab. [ the average PSNR and SSIM results of LESU and LEu are notably lower than the original results. The reason behind this observation is that both Leu and LESU loss functions have incorporated the variance term (0:) into the divisor of the absolution difference term. Consequently, pixel with large variance will penalized after the division and has less impact on the overall loss function. Note that such attenuation of pixels with large uncertainty is preferred for high-level vision tasks, as demonstrated in previous works [15 on image classification [18) image segmentation 16), and face recognition [7 19].
Low-level vision tasks such as SISR are much different. As shown in Fig. pixels with large uncertainty carry visually important information such as textured and edges. They need to be prioritized (opposite to attenuation) and given larger instead of smaller weights. To verify such claim, we have presented simple example comparing the visual results between LEU and LESU as shown Fig- 3 can seen that the uncertainty captured by LesU loss is better than LA:: loss. The improvement of LESU in Eq. (9) over LEU in Eq. (6) is attributed to the prioritization of pixels with large uncertainty (s, values). Fig. 3 (f) clearly demonstrate superiority of exploiting the sparsity constraint with the uncertainty estimation.
5

---

Table 1: Average PSNR and SSIM results for BI degradation five datasets for investigating three
different loss. The best performance is shown in bold. We record the results in 1.2 x 10 iterations.
Base Model
Scale
Loss
FSNK
Sct5
SSIM
PSNR
Set14 [8]
SSIM
PSNR
BSDTO0
SSTM
PSNR
Unban
SSIM
PSNR
Manga 109 26
SSIM
Original
30,93
0,8700
27.80
0.7627
27.05
0.7190
24.71
0.7351
28.14
0.8693
S
Cx:
30.19
0.8627
27.29
0.7538
26.78
0.7120
0.7179
0.8481
30.31
0.8637
27.39
0.7543
26.83
0.7124
24.27
0.7192
26.92
0.8496
3.2
Uncertainty-Driver Loss (UDL) for SISR
Improvement of LESU over Leu inspired us to go one step further. To better prioritize pixels with large uncertainty, we propose a new adaptive weighted loss named uncertainty-driven loss (UDL) for SISR. Unlike LesU loss putting larger weight to the second term than LEU, we suggest that the first term can also be modified to directly associate the aleatoric/data uncertainty of ((ys). That is, instead of using exp(-s; s;) to attenuate the importance of pixels with large uncertainty, we need to use a monotonically increasing function to prioritize them. Linear scaling would be a natural option, which leads to the following loss function
1 N
LUDE. =
8 |1. - f(y))),
(10)
where 8; = 8; ) is a non-negative linear scaling function. To prevent uncertainty value from degenerating into zeros, the result of uncertainty estimation network in the first step will be passed to the second step as the attention signal (s = In 0), as shown in Fig. 2 By leveraging the log variance represent the challenging and cumbersome pixels with higher uncertainty, we propose a new weighted loss named uncertainty driven loss LUDL. In LUDL loss, texture and edge pixels with higher uncertainty tend have larger weights than those in smooth regions. In summary, the uncertainty estimation 0 serves as the bridge connecting two steps: it is the output of the first step; but passed on to the second step as the guidance required for calculating LUDL. loss.
3.3 Two-step Training of Dual Networks
As shown in Fig. 2 the whole training process can be divided into two steps; the first step estimates the uncertainty 0 precisely and the second step generates the final mean value f(y) with the aid from the estimated uncertainty 0 from step1. More specifically, the mean value f(y) and variance 0 are pre-trained by LESU loss during stepl as shown in Fig. 2 (a). After the uncertainty has been estimated, the mean value f(y) network is trained by LUDL loss with variance shown in Fig. (b), while the network of inferring variance is fixed.
Note that the mean value fyy network of step2 starts training from the pre-trained network of stepl. Such partial parameter sharing is a salient property of our proposed dual networks with parallel symmetric attention (27). In theory, we can extend the two-step training into multiple-step training by alternating between the estimation of uncertainty (variance 0) and mean value fv)) Conceptually, an improved estimation of unknown HR image leads to improved estimation of alcatoric uncertainty and vice versa. This line of reasoning will lead to the pursuit of a deep equilibrium model 28 for SISR; but is beyond the scope of this paper.
3.4
Discussions: Why UDL Outperforms GRAM?
To the best of our knowledge, only one work GRAM 21 has studied data uncertainty in SISR, which is the most related to our work. We will discuss connections and differences between proposed UDL and GRAM (21) here. First, both GRAM (21) and our work has found out that applying the traditional uncertainty loss designed for high-level computer vision tasks into SISR task directly results in performance decline. For high-level computer vision tasks, the pixels with higher uncertainty indicates less confidence in final inference, which needs loss attenuation. However, for SISR tasks, the pixels with higher uncertainty (e.g., texture and edge pixels) should be prioritized with larger weights because they are visually more important than pixels smooth regions. To solve this problem, GRAM 21] proposes to use uncertainty to generate an attention mask that decreases loss attenuation. However, GRAM 21 loss still attenuated when the variance of pixels is high. Thus, GRAM 21 is still inferior to bascline method since it still does not prioritize pixels with high uncertainty.

---

Different from GRAM, we propose an uncertainty- loss to assign the pixels with high variance more weight to prioritize them. Besides, modeling uncertainty under Bayesian framework allows us to leverage sparsity prior for more precise estimation of uncertainty. Ultimately, our proposed method consists of those two technical contributions that achieve better results than baseline methods and outperform GRAM.
4 Experiments
4.1 Experimental Settings
Datasets and Metrics. 800 high-quality (2K resolution) images from the DIV2K dataset [2] have
been used for training. Following EDSR (2] five standard benchmark datasets: Set5 [23], Sct (8),
BSD100(24), Urban 100 [25]. Manga109 [26] are used for testing. Performance evaluation in terms
of of PSNR and SSIM
330 metries is conducted on the luminance (Y) channel only.
Training Setting. We randomly select 16 RGB LR patches sized by 48 x 48 as the inputs. The image patches are randomly rotated by 90º 180% 270 flipped horizontally. The ADAM algorithm []] with 3 = 0.9, 82 0.999, E = 10-8 8 is adopted to optimize the network. The initial learning rate is 10-4 and decreases by half for every 2 x 10s minibatch updates.
Degradation models. To demonstrate the effectiveness of our proposed uncertainty-drive loss in varying degradation scenarios, we have designed the following experiments with two different degradation models. Let BI denotes bicubic downsampling. The second one is BD which uses Gaussian blur followed by nearest downsampling to generate LR images. Specifically, we apply 11 11 sized Gaussian kernel with standard deviation 1.6 for blurring our experiments.
SISR Networks. We choose three different networks to verify the effectiveness of proposed LUDL. loss. The first one is EDSR-S or called baseline network in (2). EDSR-S [2] mainly consists of 16 Resblock with 64 channels, having 1.5. parameters. The second one is DPDNN where denoiser network is U-net under model-guided framework. The last one is big network EDSR(2, consisting of 32 Resblock with 256 channels, having 43M parameters. The analysis of training cost can be found in our supplementary material.
4.2 Ablation Study
Table 2: Average PSNR and SSIM results for BI degradation on five datasets for investigating three
different loss. The best performance is shown in bold. We record the results in 4 x 10 iterations.
Base Model
Scale
Loss
PSNR
23
SSIM
PSNK
SSIM
B
PSNR
BSDT00 24
SSIM
PSNR
Urhan 25
SSIM
PSNR
Mangalin
SSIM
Original
31.61
0.8862
28.22
0.7721
27.30
0.7271
25.23
0.7575
29.31
0.8907
EDSR-S[]
x:4
31.83
0.8895
28.33
0.7754
27.37
0.7297
25.49
0.7665
29.70
0.8959
Cene
31.90
0.8897
28.37
0.7755
27.40
0.7301
25.54
0.7671
29.77
0.8967
To further verify the effectiveness of sparse uncertainty estimation at stepl, we have conducted an
ablation study to compare the final PSNR/SSIM results of LUDI. with LEU or with Lesu at stepl. In our ablation study, we have used x4 bicubic down-sampling degradation on five frequently-used benchmark datasets with EDSR-S backbone [2] As shown in Tab, 2 both LEu+LUDL and LUDL loss have achieved better performance than original loss. Besides, CUSU+ loss obtains better results than CCU+L due to more accurate uncertainty estimation shown in Fig. 3 (c) and (f).
4.3 Analysis of Different Weighted Loss
There are many different weighted loss guided by different weight maps, such Error_map. Gra- dieni_man _ma which can also reveal the challenging pixels. We have conducted experiments with a weighted loss function where the weight a pixel-wise gradient Error_map. The PSNR results of five benchmark datasets for investigating the influence of different weighted loss functions can be summarized in Tab. 3
The HR gradient_man and LR gradien candien_mppaaa denote calculating gradient map from high-resolution (ground truth) images and low-resolution images respectively. The calculation of gradient can be formulated as
((i.j) 1(i+1,j) (((++,,))-((ii))) H(I,) [(i,j((i))1-)))) G(i = (i( Hli,))ll, (1I)
7

---

Table 3: Average PSNR and A PSNR results with BI degradation on five datasets for investigating
the influence of different weighted loss functions. The best performance is shown bold.
Weighted loss
Sct5
A
Sct14
A
BSDI00
A
Urban 100
A
Manga109
A
Baseline
31.61
0.00
28.22
0.00
27.30
0.00
25.25
0.00
29.31
0.00
Uncertainty(Ours)
31.90
0.29
28.37
0.151
27.40
0.10
25.54
0.291
29.77
0.46
Firor map
31.77
0.16f
28.30
0.087
27.35
0.051
25.40
0.151
29.57
0.267
HR [i__grrdieni_ngg map
31.68
0.071
28.27
0.051
27.35
0.051
25.42
0.17f
29.45
0.147
LR gradient map
31.69
0.087
28.29
0.071
27.35
0.051
25.38
0.131
29.50
0.197
where I denotes pixels value and j.j denotes position of pixels. Note that we adjust the scaling functions of Error_map. HR [iR___rddiet__mm and LR___adggge [R__gaddieii_mee to get the best performance.
From the Tab. one can be observed that other weighted loss functions can indeed improve the PSNR results, but only to certain degrees. Comparing four different weight maps, our proposed uncertainty weighted loss function can bring the biggest improvement. Although the Error Error_map map can represent the variance of single pixel, the Error_map lacks semantic information or local information to capture more precise estimation of variance comparing uncertainty. With regard to the gradient map of HR or LR images, those gradient maps only well match the edges of images and have certain correlation to variance. Comparing the visual results of Error_ Error_map, HR HR_ gradient _ma and LR_ ARgrrddien _ma with uncertainty map, those maps only detect edges of images and fail reflecting complex texture details which are important to final reconstruction performance. Therefore, uncertainty-weighee. loss can is still valuable for achieving the best performance among other weighted maps.
4.4 Analysis of Different Scaling Functions
We have conducted experiments with several various monotonically increasing functions (including linear and non-linear) and the results can be summarized in Tab.
Table 4: Average PSNR and A PSNR results with BI degradation on five datasets for investigating the influence of different scaling functions. The best performance are shown in bold.
Scaling functions
Ser5
A
Sct14
A
BSDTO0
Urban
Manga (09
Bascline
31.61
0.00
28.22
0.00
27.30
0.00
25.25
0.00
29.31
0.00
31,90
0.297
28.37
0.15f
27.40
0.107
25.54
0.297
29.77
0.467
capls)
31,80
0.19t
28.34
0.12
27.40
0.107
25,53
0.281
29.66
0.351
crp(s) (172)
31.86
0.25t
28. 36
0.141
27.41
0.11
25.55
0.301
29.71
0.401
log(s) (og((og(s))
31.89
0.28
28.39
0.171
27.42
0.127
25.57
0.327
29.74
0.437
The best and second-best performances are shown in bold. Overall, four various monotonically increasing functions have achieved better results than the baseline method. The best two scaling functions are linear scaling and log scaling with a slight difference as shown the above table. Since the linear scaling function achieves comparable performance with low computational cost, we advocate this choice this paper.
(dI EDSR S GRAM (e) EDSK Cym. Ouns Figure 4: SISR visual quality comparisons of EDSR-S 2 with different loss function 'Img_004 004'
(a) HR
1D) HK
(c) EDSK.S
(6 Our Uncertainty
and 'Ing 016' from Urban100 (25) (bicubie-downsampling x x4). Best viewed in color.

---

(A) 095 095
(b) IIR
(d) DPENN
(6) DRDNN-GRAM
(o EDDDNN-LURE Ours
(1) Our Uncertainty
Figure 5: SISR visual quality comparisons of DPDNN B3 with different loss function on 'Img_09 095' from BSD100 [24] tbicubi-doonssamling x x4). Best viewed in color.
Table 5: Average PSNR and SSIM results for BI degradation on five benchmark datasets. The best performance is shown bold. Note that LUDI.-Our denotes adopting LESU stepl and LUDI. at step2 for simplicity.
Set5
23
Setl4
8
BSDT0U
124]
Urban PSNR
100
22
Manga (09) 226
Base Model
Scale
Loss
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
Original
37.66
0.8594
33.22
0.9146
31.95
0.8969
30.71
0,9205
37.79
0,9752
EDSR-S 2
x2
GRAM [2]
37.48
0.9589
32.99
0.9126
31.76
0.8946
30.11
0,9134
37.38
0,9739
-uus
37.95
0,9604
33.50
0.9165
32.13
0.8991
31.54
0,9304
38.38
0.9767
Original
37.75
0.9600
33.30
0.9150
52,09
0.8950
31.50
0.9220
DPDNN 3
x2
GRAM (2]
37.74
0.9597
33.27
0.9148
31.98
0.8973
30.97
0.9238
38.14 14
0.9758
Cure-Ours
38.00
0.9605
33.63
0.9176
32.16
0.8995
31.72
0.9331
38.55
0.9769
Original
38.11
0.9612
33.92
0.9195
32.32
0.9013
32,93
0.9351
39.10
0,9773
EDSR 2
x2
GRAM [21
37.87
0.9604
33.43
0.9164
32.08
0.8990
31.46 .44
0.9301
37,91
0.9765
Cuna-Ours
38.29
0.9615
34.14
0,9236
32.40
0,9027
32.99
0.9446
39.53
0.9787
Original
0.9231
0.8352
0.7996
32.52
0,9369
EDSR-S x
x3
GRAM 21
33.27
0.9178
29.60
0.8298
28.66 28.60
0.7936
26.52
0.8142
31.14
0,9258
-uuse-Ors
34.15
0,9251
30.15
0,8388
28.99
0.8021
27.72
0.8430
32.97
0.9406
OFiginal
33.93
0.9240
30.02
0.8360
29,00
0.8010
27.61
0.84211
DPDNN 3
x3
GRAM 211
33.92
0.9241
30.00
0.8362
28.86
0.8000
27.37
0.8353
32.41
0.9373
Cure-Ours
0.9267
30.31
0.8419
0.8047
33.27
0.9438
Original
34,65
0.9280
30.52
0.8462
29.25
0.8043
28.80
0.8653
34.17
0,9476
EDSR 2
x3
GRAM 21
34.34
0.9270
30.28
0.8412
29.07
0.8044
27.98
0.8789
33.32
0.9432
CUDL-Ours
34,83
0,9312
30.69
0.8497
29.28
0.8109
28.99
0.8697
34.63
0,9502
Original
31.61
0.8862
0.7721
27.30
0.7271
0.7575
29.31
0.8907
EDSR-S $
2
x4
GRAM 23
31.08
0.8787
27.89
0.7670
27.12
0.7229
24.81
0.7429
18
0.8762
-uus
31.90
0.8897
28.37
0.7755
27.40
0.7301
25.54
0.7671
29.77
0.8967
Original
31.72
0.8390
28. 28
0.7730
27.44
0.7290
25.53
0,7680
DPDNN 3
x4
GRAM (2]
31.89
0.8913
28.37
0.7772
27.41
0.7314
23
0,7708
29.70
0,9003
Cupe
32.20
0.8944
28.60
0.7819
27.56
0.7356
0.7862
0.9082
Original
32.46
0.8968
28.80
0,7876
27.71
0,7420
26.64
0.8033
31.02
0,9148
EDSR
GRAM [1]
32.32
0.8971
28.73
0.7858
27.66
0.7395
26.35
0.7955
30.73
0,9125
Cupu -Ours
32.59
0.8998
28.87
0.7889
27.78
0.7431
26.75
0.8054
31,24
0.9167
4.5 Results with BI Degradation Model
For bicubic downsampling (BI), we have compared proposed LUDL loss function with GRAM 21 and original loss functions such as MSE Li three different SISR networks. The average PSNR and SSIM results in Tab. (bb.5)ees are cited from corresponding papers or retrained from officially released code. It is casy to see that our proposed CUDI. loss function is superior to GRAM 21 and original
Table Average PSNR and SSIM results for BD degradation on five benchmark datasets. The best performance is shown in bold. Note that -Ours denotes adopting LESU at step1 and LUDL at step2 for simplicity.
Set5
23
Setl4
8
BSDTO0
24
Urban PSNR
TO0 25
Manga (09
26
Base Model
Scale
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SSIM
PSNR
SSIM
Original
31.70
0.8903
28.37
0.7778
27.37
0.7320
25.77
0.7789
29,83
0.9014
EDSR-S
GRAM 21
30.98
0.8791
27.85
0.7667
27.05
0.7225
24.79
0.7452
28.12
0.8773
Eupx-Ours
31.97
0.8927
28.45
0.7793
27.41
0.7321
25.95
0.7842
30.18
0,9053
Original
31.56
0.8923
28.38
0.7780
27.36
0.7311
25.82
0.7812
29.77
0.9033
DPDNN
GRAM 21
31.75
0.8913
28.33
0.7765
0.7302
25.62
0.7739
29.55
0.9003
Eupe-Ours
32,03
0.8949
28.60
0.7828
27.48
0.7355
26.21
0.7931
30.35
0,9097
Original
32.17
0.8975
28.65
0.7856
27.59
0.7400
26.56
0.8043
30.66
0.9133
FDSR
GRAM [2]]
32.13
0.8963
28.57
0.7822
27.49
0.7362
26.19
0.7916
30.48
0.9097
-Ours
32.37
0.8986
27.74
0.7867
27.62
0.7407
26.65
0.8065
30.81
0.9149

---

loss functions such as MSE or L1 in terms of PSNR and SSIM values. Note that the improvements achieved by our proposed method do not bring any additional computation cost during testing time. Comparing EDSR-S (1.5M parameters) with EDSR (43M parameters), our proposed LUDI. can bring lightweight networks with more greater performance improvements than big ones. The visual image comparison results are reported in Fig. and Fig. 5 As shown in Fig. 4 our proposed LUDL has recovered with fewer visible artifacts (e.g.. the circular pattern the roof and the lines on the glassy surface) than original loss and GRAM (21]. Fig. (f) depicts the uncertainty learned by LUDL, revealing the challenging pixels with poor reconstruction performance. From Fig. []] vertical center-lin line of window has been recover and (f), while DPDNN and OPDNN-GRAM [2] failed to discern shown (c) and (d) respectively. More visual comparisons can be found in supplementary material.
clear with precisely estimated uncertainty shown in (e)
4.6 Results with BD Degradation Model
For blur downsampling (BD), we have compared proposed LUDL loss function with GRAM (1 and original loss functions such as MSE or C1 on three different baseline networks. The average PSNR and SSIM results in Tab. are retrained from officially released code. It is easy
that propose LUDI loss function is superior to GRAM (21) and original loss functions such as MSE or L1 in terms of PSNR and SSIM values. The visual image comparison results of BD degradation
reported in Fig. and Fig. 7 Note that the BD degradation involves Gaussian blur, increasing difficulty in recovering structure patterns. From Fig- we can see that SR result (Fig. (e)) of 'Img 109" is the closest to that of the ground-truth. In another challenging image ('Img 078' from Urban10M (25)), our method can recover much more reliable textured details as shown Fig. (e): while all other methods have severe aliasing artifacts (i.e., distorted tile patterns). The visual quality improvement achieved by CUDL mainly due the fact that our proposed method makes full use of the captured uncertainty to train deep networks focusing on the challenging pixels with high uncertainty. More visual comparisons can be found our supplementary material.
(a) Img_109
[bJ HK
(e) DPONN
(d) DPENN-GRAM
(c) DPUNN--UDL -Ours
(f) Oar Uncertainty
Figure 6: SISR visual quality comparisons of DPDNN B with different loss function 'Img_109 109' from Manga109 (26 (blur-downsampling x4). Best viewed color.
(al Ing_078
[b] HK
(c) BUSK
(d) EUSR-GRAM
(e) EUSK-21mx--00
(i) Our Uncertainy
Figure 7: SISR visual quality comparisons EDSR 2 with different loss function on 'Img_07 from Urban (25] (blu--dowssampling x)). Best viewed in color.
Conclusion
In this paper, we propose new adaptive weighted loss LUDL for SISR to train SISR networks focusing on challenging pixels with high uncertainty (c.g., textured and edge pixels). Specifically, variance estimation is introduced into SISR so that the high-resolution images (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, modeling uncertainty under Bayesian framework allows us leverage sparsity prior for a more precise estimation of uncertainty. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance to visual quality. For the first time, we demonstrate that such uncertainty-driver loss can achieve better results than MSE or Li loss. Experimental results three popular SISR networks show that our proposed uncertainty-driven loss achieved better PSNR performance than traditional loss functions without any increased computation during testing.
10

---

Acknowledgement
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0101400 and the Natural Science Foundation of China under Grant 61991451, Grant 61632019, Grant 61621005, and Grant 61836008. Xin Li's work is partially supported by the NSF under grants I15-1951504 and DAC-1140855 the DoJ/NU under grant NIJ 2018-75-CX-0032, and the WV Higher Education Policy Commission Grant (HEPC.dsr. 18.5).
References
[1] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision, pages 184-199, 2014.
[2] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1132-1140, 2017.
[3] Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fanglang Wu, and Xiaotong Lu. Denoising prior driven deep neural network for image restoration. JEEE Transactions on Pattern Analysis and Machine Intelligence, 41(10):2305 2318, 2019.
[4] Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu, Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference Computer Vision (ECCV). pages 286-301, 2018.
[5] Tao Dai, Jianrui Cai, Yongbing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1 11065-11074, 2019.
[6] Qian Ning, Weisheng Dong, Guangming Shi, Leida Li, and Xin Li. Accurate and lightweight image super-resolution with model-guided deep unfolding network. IEEE Journal of Selected Topics in Signal Processing, 1522)220--22,, 2021. doi: 10.1 109/SSTSS2222003077561
[7] Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao. Zeshuai Deng, Yanwu Xu. and Mingkui Tan. Closed-loop matters: Dual regression networks for single image super-resolution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5407-5416, 2020.
[8] Roman Zeyde. Michael Elad, and Matan Protter. On single image scale-up using sparse-representationn. In Proceedings of the 7th international conference Curves and Surfaces, pages 711-730, 2010.
[9] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694 694-711. Springer, 2016.
[10] Mario AT Figueiredo. Adaptive sparseness using jeffreys prior. NIPS, pages 697-704, 2001.
[11] Qian Ning, Weisheng Dong, Fangfang Wu, Jinjian Wu, Jie Lin, and Guangming Shi. Spatiial-emporal gaussian scale mixture modeling for foreground estimation. Proceedings of the AAA/ Conference on Artificial Intelligence, 34(07):1 1791- 11798, Apr. 2020, doi: 10. (609/aaii.344007 .6551. URL https
V/ojs. wat.saaaa..rgnnneeeee
php/ AAAI wat.a//aallee/.oooo.ooo.ooa /651661
[12] Paul W Goldberg, Christopher KI Williams, and Christopher Bishop. Regression with input-dependent noise: A gaussian process treatment. Advances in neural information processing systems, 10:493-499 4999 1997.
[13] WA Wright. Bayesian approach to neural-netword modeling with input uncertainty. IEEE Transactions on Newral Networks, 10(6): 1999.
[14] Christopher M Bishop and Cazhaow S Qazaz. Regression with input-dependent noise: A bayesian treatment. Advances neural information processing systems, pages 347-353, 1997.
[15] Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for com- puter vision? In I. Guyon, U. V. Luxburg. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL httpps:://ppocceccc 2650d608 9 6:14 0c 5bw.88822b882268 bat.ceee.55552222555-222--00
ngs wew.eeeeddinnnununeneeeeeeee www.neuwr....paaeeeeeee pttp.//apppp///////111...//
[16] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- architecture for image sogmentation. IEEE transactions on pattern analysis and machine intelligence, 39 (12):2481224) 1211112245,,22 2017.
[17] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5710-
5719, 2020.
11

---

[18] Yingjie Gu, Zhong Jin, and Steve C Chiu. Active learning combining uncertainty and diversity for multi-class image classification. IET Computer Vision 93)):40-407, 2015.
[19] Yichun Shi and Anil K Jain. Probabilistic face embeddings. In Proceedings of the JEEE/CVE International Conference on Computer Vision, pages 6902 6911, 2019.
[20] Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does matter? Structural safety, 31(2): 105-112, 2009.
[21] Changwoo Lee and Ki-Seok Chung. Gram: Gradient rescaling attention model for data uncertainty estimation in single image super resolution. In 2019 18th IEEE International Conference Machine Learning And Applications (ICMLA), pages 13. IEEE, 2019.
[22] Aupendu Kar and Prabir Kumar Biswas. Fast bayesian uncertainty estimation and reduction of batch normalized single image super-resolution network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4957-4966, June 2021.
[23] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based nonnegative neighbor embedding. In British Machine Vision Conference 2012, pages 1-10, 2012.
[24] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume pages 416-423, 2001.
[25] Jia-Bin Bin Huang. Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-cxemplars. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5197-5206, 2015.
[26] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal09 dataset. Multimedia Tools and Applications, 76 (20::21111221838, 2017.
[27] Yingqian Wang, Xinyi Ying, Longguang Wang, Jungang Yang, Wei An, and Yulan Guo. Symmetric parallax attention for stereo image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pages 766-775, June 2021.
[28] Davis Gilton, Gregory Ongie, and Rebecca Willett. Deep equilibrium architectures for inverse problems in imaging. arXiv preprint arXiv:21022 102.07944, 2021.
[29] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops. pages 1 114-125, 2017.
[30] Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 13(4):600-61) 612, 2004.
[31] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
URLhttp: /ttp://///xivv/ org/ abt//bb//4440 412. 6980
12


Reviews:
[{"review_id": "mcExPWGa2ds", "reviewer": "Reviewer_fa9Z", "summary": "This paper proposes a novel uncertainty-driven loss for super-resolution. The authors demonstrate that high variance (uncertain) regions in an image should be prioritized for SR. To this end, they propose a 2-step framework, where the variance map is predicted first (step 1) and SR is learned with a loss using the predicted variance map. With the proposed uncertainty-driven loss (UDL), larger weights are assigned to uncertain regions. UDL loss is shown to improve three SR backbone networks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Differentiating how to handle uncertainties in low-level vision tasks (compared to high-level vision tasks) is meaningful: for low or high level vision, large variance regions should be prioritized or attenuated and epistemic or aleatoric uncertainty should be analyzed.\n* Assigning larger weights to larger variance regions lets the network to focus on challenging areas.\n* 2-step training strategy is a smart way to prevent the variance map from degenerating to zero.\n* Mathematical modelings provide insights (epistemic uncertainty modeling, Bayesian modeling, sparsity prior, etc).\n* The proposed UDL loss is simple and can be used on any SR model.\n* Theoretical background and analyses made throughout the paper is thorough and insightful.\n* Larger performance improvements can be achieved on lightweight models, which would be useful in practical scenarios. This is reasonable as big CNN models would already be able to handle the uncertain regions to a certain extent.\n\nCons:\n* Although adding the proposed loss would not impact computational complexity during inference, computational cost for training the framework (e.g. training time) may be increased. Providing computational complexity evaluation for the training stage would be useful.\n* A linear scaling is adopted in Eq. (10). Has this been empirically verified? What about other types of monotonically increasing functions? (e.g. exponential, log, exponentiation, etc)\n* Limitations of the proposed method are not discussed. A potential limitation would be longer training time due to the 2-step training.\n* Societal impacts are not discussed.\n\nOther comments:\n* An observation $-$ in Table 3, improvement seems to be larger for datasets with more high-frequency regions (Urban 100 and Manga 109) for the lightweight model EDSR-S, showing that using the proposed uncertainty model helps for challenging images.\n* Marking best results in Table 1 and 2 would be helpful for the readers.\n* Typos: (i) Table 3 and 4, GARM -> GRAM, (ii) Table 4 caption, DB -> BD\n* Line spacing in Conclusion seems a bit tight.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impacts have not been addressed. Please refer to comments above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "fM7lqvRokJZ", "reviewer": "Reviewer_g7Ce", "summary": "This paper uses the estimated uncertainty map as a weight map to the loss function and shows consistent performance improvement across different super-resolution (SR) models and datasets. The main contribution of this paper is adopting Bayesian uncertainty in the SR domain and effectively uses this to design a new uncertainty-driven loss function.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of this paper is limited. The estimated uncertainty-driven loss and experiments are technically valid. However, there are few concerns.\n\n1. Uncertainty maps are generally higher in the edge regions. So weighted loss in stage 2 gives more emphasis on the edges. Hence, it leads to good performance. Therefore, the comparison is required with the weighted loss function where the weight is a pixel-wise gradient or edge map instead of uncertainty. Then only the conclusion can be drawn that the improvement is due to uncertainty-weighted guidance.\n\n2. Good explanations with examples are necessary about how uncertainty is influencing better texture recovery. The subjective analysis between the models of the first stage and second stage can give a clear picture of the impact of uncertainty.\n\n3. What if we use the error map of the first stage model as a weighted loss function. How does it impact the performance?\n\n4. The motivation behind the uncertainty-weighted loss function is unclear. Either clear theoretical analysis or experimental analysis (as mentioned above) must be provided.\n\n# Post Rebuttal\nThis paper has given a very good rebuttal and addressed most of my concerns. Therefore, I am improving the rating of the paper.\nImproved the rating from 4: Ok but not good enough - rejection to 6: Marginally above the acceptance threshold.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should provide a comprehensive analysis with the edge-weighted loss functions as mentioned earlier to understand the impact of uncertainty weighted loss function. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "dMF2nqYvfyj", "reviewer": "Reviewer_4et4", "summary": "The paper is about single image super-resolution. The main motivation is that current existing losses are uniform for all pixels, despite the fact that more important information to recover is edges. The main proposed idea is to have adaptive losses that weigh the loss of each pixel differently, based on uncertainty. The paper introduces variance estimation characterizing the uncertainty on a pixel-by-pixel basis. The method simultaneously computes the targeted pixels and their corresponding uncertainty.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The proposed formulation of estimating the super-resolution pixels and their uncertainty simultaneously is interesting. In the super-resolution community, it seems to be novel. However, I am not particularly sure if it is novel in the larger deep learning communities.  While the paper discusses the novelty and contributions in the context of super-resolution, I am wondering if there is any novelty or contributions in the context of uncertainty computation. How is the proposed uncertainty computation compared with the existing methods?\n\n2. In the experimental results, while the comparisons are shown for super-resolution results, there is no comparison shown for the estimation of uncertainty. How is it compared to [22]?\n\n3. I am also wondering how good the uncertainty prediction is here. The paper should show the error map (the heatmap representing the distance between the estimated high-res and the ground-truth high-res) next to the estimated uncertainty map.\n\n4. In Fig.3, there is a module called “nearest upsampling”, however there is no discussion on this or its design. \n\n5. Line 179: “Linear scaling would be a natural option”. What is the theoretical reason for this? How can we decide if it has to be linear or non-linear? How do we know that the computed uncertainty has to be linear as a weight? Is this linear scaling a general rule  beyond super-res?\n\n6. The uncertainty in this paper is based on $\\theta$ in Eq.(4), which originated from the assumption of a Laplace distribution of the L1 error function.The paper also mentions that it focuses on epistemic/model uncertainty. I am wondering how we can justify that $\\theta$ has something to do with epistemic uncertainty, why not aleatoric/data uncertainty?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nNo limitations are mentioned or discussed.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a method for single image super-resolution based with emphasis on challenging situations such as textured and edge pixels with high uncertainty. The proposed uncertainty-driven loss improves results over other common losses (MSE and L1). All reviewers liked the paper. They asked some questions such as connection to other works, computational complexity at training and inference time, the reason why linear scaling would be a natural option, and the meaning of uncertainty in the proposed scheme. The authors provided a strong rebuttal and all reviewers at the end lean toward accepting the paper. I concur with them and recommend accept. Please make sure to include items suggested by reviewers in the final revision (e.g. comparison with other types of the weighted loss function).
Decision: Accept (Poster)