Uncertainty-Driven Loss for Single Image Super-Resolution

Abstract: In low-level vision such as single image super-resolution (SISR), traditional MSE or L1 loss function treats every pixel equally with the assumption that the importance of all pixels is the same. However, it has been long recognized that texture and edge areas carry more important visual information than smooth areas in photographic images. How to achieve such spatial adaptation in a principled manner has been an open problem in both traditional model-based and modern learning-based approaches toward SISR. In this paper, we propose a new adaptive weighted loss for SISR to train deep networks focusing on challenging situations such as textured and edge pixels with high uncertainty. Specifically, we introduce variance estimation characterizing the uncertainty on a pixel-by-pixel basis into SISR solutions so the targeted pixels in a high-resolution image (mean) and their corresponding uncertainty (variance) can be learned simultaneously. Moreover, uncertainty estimation allows us to leverage conventional wisdom such as sparsity prior for regularizing SISR solutions. Ultimately, pixels with large certainty (e.g., texture and edge pixels) will be prioritized for SISR according to their importance to visual quality. For the first time, we demonstrate that such uncertainty-driven loss can achieve better results than MSE or L1 loss for a wide range of network architectures. Experimental results on three popular SISR networks show that our proposed uncertainty-driven loss has achieved better PSNR performance than traditional loss functions without any increased computation during testing.


Reviews and author responses:

 Thank you for your careful review and constructive suggestions. We found them helpful and will include them in the final version of this paper.

 The rebuttal addresses all my concerns regarding this paper. I will improve my score. It will be good if this paper considers the following concerns in the final version of the paper.

1. Include the comparison with other types of the weighted loss function in the main paper. (It is also suggested by Reviewer fa9Z)

2. This paper claims in Line 59-60 that "For the first time, we propose to cast SISR into a Bayesian estimation framework". I believe this claim is not proper as [21] already introduced the bayesian uncertainty in the SISR framework.

3. [21], [22], [27], [31] have already been accepted in different venue. It is always good if this paper cites with a proper citation instead of arxiv pre-prints.

Thank you

This paper uses the estimated uncertainty map as a weight map to the loss function and shows consistent performance improvement across different super-resolution (SR) models and datasets. The main contribution of this paper is adopting Bayesian uncertainty in the SR domain and effectively uses this to design a new uncertainty-driven loss function.  The novelty of this paper is limited. The estimated uncertainty-driven loss and experiments are technically valid. However, there are few concerns.

1. Uncertainty maps are generally higher in the edge regions. So weighted loss in stage 2 gives more emphasis on the edges. Hence, it leads to good performance. Therefore, the comparison is required with the weighted loss function where the weight is a pixel-wise gradient or edge map instead of uncertainty. Then only the conclusion can be drawn that the improvement is due to uncertainty-weighted guidance.

2. Good explanations with examples are necessary about how uncertainty is influencing better texture recovery. The subjective analysis between the models of the first stage and second stage can give a clear picture of the impact of uncertainty.

3. What if we use the error map of the first stage model as a weighted loss function. How does it impact the performance?

4. The motivation behind the uncertainty-weighted loss function is unclear. Either clear theoretical analysis or experimental analysis (as mentioned above) must be provided.

# Post Rebuttal
This paper has given a very good rebuttal and addressed most of my concerns. Therefore, I am improving the rating of the paper.
Improved the rating from 4: Ok but not good enough - rejection to 6: Marginally above the acceptance threshold.
 The authors should provide a comprehensive analysis with the edge-weighted loss functions as mentioned earlier to understand the impact of uncertainty weighted loss function. 

 Thank you for providing additional experiments and addressing my comments, especially on training complexity and trying out other monotonically increasing functions. I think x1.8 increase in training time is reasonable, considering the 2-step training strategy. Also, the experiment results on different scaling functions provides a more complete analysis, backing up the authors' choice on using linear scaling.

The comparison to other types of weighted loss functions in the authors' reply to reviewer g7Ce would also be a nice addition to the revised version as the readers can compare to other simple baselines. It further shows meaningful improvements over the other methods when using the uncertainty-driven loss.

The paper provides many insights in leveraging uncertainties in SR. I'll keep my score of 7, enjoyed reading the paper!

 We thank the reviewer for the time and effort on evaluating this work.

**1." Uncertainty maps are generally higher in the edge regions. So weighted loss in stage 2 gives more emphasis on the edges. Hence, it leads to good performance. Therefore, the comparison is required with the weighted loss function, where the weight is a pixel-wise gradient or edge map instead of uncertainty. Then only the conclusion can be drawn that the improvement is due to uncertainty-weighted guidance. "**

**Reply:** Thanks for your constructive criticism. The physical meaning of uncertainty in our paper is the variances of recovered pixels, indicating the confidence of SR reconstruction. Despite error_map can be regraded as variance estimation of a single pixel, it is not accurate since the calculation of error_map is based on a single pixel.  Indeed, the pixel-wise gradient or edge map can show the edges of images, however, those maps also can't reflect the variance but only have a certain correlation to variance.  How to obtain an accurate estimation of the variance of recovered pixels is a challenging and unsolved problem. In this paper, we propose to estimate uncertainty (or called variance) by maximizing Laplace likelihood of pixels predictions from Bayes perspective.  Such approach gives a much more precise estimation of variance than other simple weighted maps such as *Error_map, HR_gradient_map and LR_gradient_map* . To better demonstrate the above claim, we have conducted experiments with a weighted loss function where the weight is a pixel-wise gradient or error map. The PSNR results of five benchmarks can be summarized as follows:

| Model  |  Weighted loss  |   Set5    |   ΔPSNR   |   Set14   |   ΔPSNR   |  BSD100   |   ΔPSNR   | Urban100  |   ΔPSNR   | Manga109  |   ΔPSNR   |
| :----: | :-------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| EDSR_S |    Baseline     |   31.61   |   0.00    |   28.22   |   0.00    |   27.30   |   0.00    |   25.25   |   0.00    |   29.31   |   0.00    |
| EDSR_S |   Uncertainty   | **31.90** | **0.29↑** | **28.37** | **0.15↑** | **27.40** | **0.10↑** | **25.54** | **0.29↑** | **29.77** | **0.46↑** |
| EDSR_S |    Error_map    |   31.77   |   0.16↑   |   28.30   |   0.08↑   |   27.35   |   0.05↑   |   25.40   |   0.15↑   |   29.57   |   0.26↑   |
| EDSR_S | HR_gradient_map |   31.68   |   0.07↑   |   28.27   |   0.05↑   |   27.34   |   0.04↑   |   25.36   |   0.11↑   |   29.47   |   0.16↑   |
| EDSR_S | LR_gradient_map |   31.69   |   0.08↑   |   28.29   |   0.07↑   |   27.35   |   0.05↑   |   25.38   |   0.13↑   |   29.50   |   0.19↑   |

The best performance is shown in **bold**. The "HR_gradient_map" and "LR_gradient_map" denote calculating gradient map from high-resolution (ground truth) images and low-resolution images respectively. The calculation of gradient can be formulated as 

           V(i,j)= I(i+1,j)-I(i,j),   H(i,j)= I(i,j+1)-I(i,j), G(i,j)=||(V(i,j),H(i,j)||_2,

where I denotes pixels value and i,j denotes position of pixels . Note that we adjust the scaling functions of *Error_map, HR_gradient_map and LR_gradient_map* to get the best performance. 

From the above table, one can be observed that other weighted loss functions can indeed improve the PSNR results, but only to certain degrees. Comparing four different weight maps, our proposed uncertainty weighted loss function can bring the biggest improvement. Although the error_map can represent the variance of a single pixel, the error_map lacks semantic information or local information to capture a more precise estimation of variance comparing uncertainty. With regard to the gradient map of HR or LR images, those gradient maps only well match the edges of images and have a certain correlation to variance.  Comparing the visual results of *Error_map, HR_gradient_map and LR_gradient_map* with *uncertainty map*, those maps only detect edges of images and fail reflecting complex texture details which are important to final reconstruction performance. Therefore, uncertainty-weighted loss can is still valuable for achieving the best performance among other weighted maps. To the best of our knowledge, there is no work except GRAM [20] in SISR domain that uses weighted loss to address the importance of different pixels, which is the biggest contribution and key novelty of our paper. 

Again, thank you for your constructive suggestions. We found them helpful and will include them in the next revised version.

**2."Good explanations with examples are necessary about how uncertainty is influencing better texture recovery. The subjective analysis between the models of the first stage and second stage can give a clear picture of the impact of uncertainty. "**

**Reply:** We have shown the visual comparison of our proposed uncertainty loss and in Fig.4, Fig.5, Fig.6 and Fig.7. Taking Fig.5 as an example, Fig.5.(c) shows a much more blur recovery effect than our proposed method shown in (e) (see the central line of the window) with the help of estimated uncertainty shown in (f). Comparing the first stage and second stage, there is much PSNR gap between them. In the first stage, the pixels with large uncertainty have less impact on the overall loss function. As shown in Tab. 1, the PSNR performance drops a lot comparing to the baseline method. In the second stage, the pixels with larger uncertainty have more impact on the overall loss function. As shown in Tab.2, the PSNR performance increases a lot comparing to the baseline method. From visual comparison, the recovered images seem more blur in step1, while the recovered images seem sharper in step2. Thanks for your suggestion, we will add subjective analysis between the models of the first stage and second stage to the present impact of uncertainty.

**3."What if we use the error map of the first stage model as a weighted loss function. How does it impact the performance?"**

**Reply:** Thanks for your suggestion, we have experimented that using the error map of the first stage model as a weighted loss function. Please see our reply to question 1.

**4."The motivation behind the uncertainty-weighted loss function is unclear. Either clear theoretical analysis or experimental analysis (as mentioned above) must be provided. "**

**Reply:** In this paper, we propose to estimate uncertainty (or called variance) by maximizing Laplace likelihood of pixels predictions from Bayes perspective. The physical meaning of uncertainty in our paper is the variances of recovered pixels, indicating the confidence of SR reconstruction. To this end, we propose a new adaptive weighted loss for SISR to train deep networks focusing on challenging situations with high variance. As a result, the networks reduce the variance of pixels with larger variance (such as edge and texture region) and increase the variance of pixels with small variance (such as edge and smooth region) slightly. Despite the error_map has achieved improvement, the error_map is not accurate since it is calculated on a single pixel. Therefore, our proposed method achieves much better performance than error_map. As you requested, we have provided experiments about different weighted losses, please see our reply to questions 1-3.

**Authors' Final Comments** We hope that the discussion we provided alleviates the reviewer's concerns. We are happy to answer any additional questions the reviewer has.

 We thank the reviewer for the time and effort on evaluating this work.

**1." Although adding the proposed loss would not impact computational complexity during inference, computational cost  for training the framework (e.g. training time) may be increased. Providing computational complexity evaluation for the  training stage would be useful."**

**Reply:** Thanks for your constructive suggestion. The training cost mainly depends on the original networks and iterative times. Let $n$ denotes the computational cost of forward inference of batch data. Then the computational cost of the backpropagation can be approximately equal to $n$. During our training experiments, the baseline method is trained with **4\*10^5** minibatch updates. Step1 training process costs **1.2\*10^5** minibatch updates to obtain a great uncertainty estimation. Step2 training process is trained with **4\*10^5** minibatch updates and the variance branch doesn't require backpropagation. The computational complexity evaluation can be summarized as follows:

|             | Forward cost | Backpropagation cost | Iterative times | Computation cost |
| :---------: | :----------: | :------------------: | :-------------: | :--------------: |
|  Baseline   |     $n$      |         $n$          |     4*10^5      |      8*10^5      |
| Ours(step1) |     $n$      |         $n$          |    1.2*10^5     |     2.4*10^5     |
| Ours(step2) |     $2n$     |         $n$          |     4*10^5      |     1.2*10^6     |

Overall, the computational cost of the training process of our proposed method is **1.8** times of traditional MSE or L1 loss. The computational complexity evaluation will be included in the next revised version.

**2."A linear scaling is adopted in Eq. (10). Has this been empirically verified? What about other types of monotonically increasing functions? (e.g. exponential, log, exponentiation, etc.)"**

**Reply:** Thanks for your constructive suggestions. We have conducted experiments with eleven various monotonically increasing functions, and the results can be summarized as follows:

| Model  | Scale functions |   Set5    |   ΔPSNR   |   Set14   |   ΔPSNR   |  BSD100   |   ΔPSNR   | Urban100  |   ΔPSNR   | Manga109  |   ΔPSNR   |
| :----: | :-------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| EDSR_S |    Baseline     |   31.61   |   0.00    |   28.22   |   0.00    |   27.30   |   0.00    |   25.25   |   0.00    |   29.31   |   0.00    |
| EDSR_S |     s-min(s)      | **31.90** | **0.29↑** | **28.37** | **0.15↑** |   27.40   |   0.10↑   |   25.54   |   0.29↑   | **29.77** | **0.46↑** |
| EDSR_S |     exp(s)      |   31.80   |   0.19↑   |   28.34   |   0.12↑   |   27.40   |   0.10↑   |   25.53   |   0.28↑   |   29.66   |   0.35↑   |
| EDSR_S |    exp(s)^2     |   31.68   |   0.07↑   |   28.27   |   0.05↑   |   27.35   |   0.05↑   |   25.42   |   0.17↑   |   29.45   |   0.14↑   |
| EDSR_S |  exp(s)^(1/2)   |   31.86   |   0.25↑   |   28.36   |   0.14↑   | **27.41** | **0.11↑** |   25.55   |   0.30↑   |   29.71   |   0.40↑   |
| EDSR_S |  exp(s)^(1/3)   |   31.86   |   0.25↑   |   28.35   |   0.13↑   |   27.40   |   0.10↑   |   25.52   |   0.27↑   |   29.72   |   0.41↑   |
| EDSR_S |  exp(s)^(1/4)   |   31.86   |   0.25↑   |   28.35   |   0.13↑   |   27.40   |   0.10↑   |   25.51   |   0.26↑   |   29.72   |   0.41↑   |
| EDSR_S |log(s)-min(log(s))| **31.89** | **0.28↑** | **28.39** | **0.17↑** | **27.42** | **0.12↑** | **25.57** | **0.32↑** | **29.74** | **0.43↑** |
| EDSR_S |       s^2       |   31.85   |   0.24↑   | **28.37** | **0.15↑** | **27.41** | **0.11↑** | **25.56** | **0.31↑** |   29.71   |   0.40↑   |
| EDSR_S |       s^3       |   31.80   |   0.19↑   |   28.33   |   0.11↑   |   27.40   |   0.10↑   |   25.53   |   0.28↑   |   29.65   |   0.34↑   |
| EDSR_S |     s^(1/2)     |   31.87   |   0.26↑   | **28.37** | **0.15↑** |   27.40   |   0.10↑   |   25.54   |   0.29↑   |   29.72   |   0.41↑   |
| EDSR_S |     s^(1/3)     |   31.88   |   0.27↑   |   28.36   |   0.14↑   |   27.4    |   0.10↑   |   25.52   |   0.27↑   |   29.73   |   0.42↑   |

The best and second-best performances are shown in **bold**. Overall, eleven various monotonically increasing functions have achieved better results than the baseline method. The best two scaling functions are linear scaling and log scaling with a slight difference as shown in the above table. Since the linear scaling function achieves a desirable performance, we advocate this choice. The analysis of various scaling functions will be added in the next revised version.

Again, thank you for your constructive suggestions. We found them helpful and will include them in the next revised version.

**3." Limitations of the proposed method are not discussed."**

**Reply:** The biggest limitation of our proposed method is longer training time as we require two-steps training. Thanks for your suggestion, we will add the above limitations in the next revised version. We leave turning two-steps training into one-step training or reducing training time in future work.

**4."Societal impacts are not discussed."**

**Reply:** Single image super-resolution (SISR) aims at reconstructing high-resolution (HR) images from their corresponding degraded low-resolution (LR) images, having many applications in various scientific and engineering fields (e.g., Nobel Prize in Chemistry was awarded to SR microscopy in 2014). Our work provides a new proposed uncertainty-driven loss that has achieved better PSNR performance than traditional loss functions without any increased computation during testing. A direct societal impact of this line of research will be SR imaging, which will continue to be at the frontier of developing next-generation smartphone cameras (e.g., Google Pixel vs. Apple iPhone).  

Thanks for your suggestion, we will add the above societal impacts in the next revised version.

**5."Other comments,..."**

**Reply:** Thank you for your other suggestions on further improving clarity. We found them helpful and will include them in the next revised version.

**Authors' Final Comments** We hope that the discussion we provided alleviates the reviewer's concerns. We are happy to answer any additional questions the reviewer has.

 We thank the reviewer for the time and effort on evaluating this work.

**1." if there is any novelty or contributions in the context of uncertainty computation. How is the proposed uncertainty computation compared with the existing methods?"**

**Reply:** First, we impose a useful sparse prior called **Jeffrey's prior** $p(w)=\frac{1}{w}$ on uncertainty  $\theta_i$, which is based on the observation that the uncertainty is sparse in view of the whole image as shown in Fig. 2. To the best of our knowledge, we are the first to impose prior on uncertainty $\theta_i$ for better and more accurate uncertainty estimation. Second, unlike other computer vision tasks that the pixels with larger variance are viewed as outliers with less impact on the overall loss function, we emphasize those pixels with large variance and develop a new uncertainty-driven loss to address the pixels with larger uncertainty. In summary, our approach toward uncertainty-driven loss represents a significant departure from the standard practice by exploiting the estimated uncertainty to improve the final reconstruction performance. 

**2." there is no comparison shown for the estimation of uncertainty. How is it compared to [22]?"**

**Reply:** The $L_{EU}$ Uncertainty shown in Fig.2(e) can be viewed as a baseline estimation of uncertainty (there is a little difference that we maximize Laplace likelihood to estimate uncertainty in SISR while [22] maximize Gaussian likelihood to estimate uncertainty in semantic segmentation). Comparing two uncertainty $L_{EU}$ and $L_{ESU}$  loss functions, it can be clearly observed that our uncertainty with sparse prior $L_{ESU}$ shows a more clear and accurate estimation. Tab. 1 and Tab. 2 also show that uncertainty with sparse prior leads to better performance in step1 as well as step2. Therefore, our proposed Estimating Sparse Uncertainty (ESU) performs better than the original uncertainty estimation in SISR domain.

**3." The paper should show the error map next to the estimated uncertainty map."**

**Reply:** Thanks for your suggestion, we will add the error map next to the estimated uncertainty map in the revised version.

**4." In Fig.3, there is a module called “nearest upsampling”, however there is no discussion on this or its design."**

**Reply:** The module called "nearest upsampling" is a simple interpolation operator to upsampling the feature maps to the resolution of high-res images. It is not designed by us, and we just use it due to its low computational cost.  Thanks for pointing it out, and we will add some description about "nearest upsampling" in the next revised version.

**5." Line 179: “Linear scaling would be a natural option”. What is the theoretical reason for this? How can we decide if it has to be linear or non-linear? How do we know that the computed uncertainty has to be linear as a weight? Is this linear scaling a general rule beyond super-res?"**

**Reply:** Thanks for your constructive suggestions. We have conducted experiments with eleven various monotonically increasing functions (including linear and non-linear) and the results can be summarized as follows:

| Model  | Scale functions |   Set5    |   ΔPSNR   |   Set14   |   ΔPSNR   |  BSD100   |   ΔPSNR   | Urban100  |   ΔPSNR   | Manga109  |   ΔPSNR   |
| :----: | :-------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |
| EDSR_S |    Baseline     |   31.61   |   0.00    |   28.22   |   0.00    |   27.30   |   0.00    |   25.25   |   0.00    |   29.31   |   0.00    |
| EDSR_S |     s-min(s)      | **31.90** | **0.29↑** | **28.37** | **0.15↑** |   27.40   |   0.10↑   |   25.54   |   0.29↑   | **29.77** | **0.46↑** |
| EDSR_S |     exp(s)      |   31.80   |   0.19↑   |   28.34   |   0.12↑   |   27.40   |   0.10↑   |   25.53   |   0.28↑   |   29.66   |   0.35↑   |
| EDSR_S |    exp(s)^2     |   31.68   |   0.07↑   |   28.27   |   0.05↑   |   27.35   |   0.05↑   |   25.42   |   0.17↑   |   29.45   |   0.14↑   |
| EDSR_S |  exp(s)^(1/2)   |   31.86   |   0.25↑   |   28.36   |   0.14↑   | **27.41** | **0.11↑** |   25.55   |   0.30↑   |   29.71   |   0.40↑   |
| EDSR_S |  exp(s)^(1/3)   |   31.86   |   0.25↑   |   28.35   |   0.13↑   |   27.40   |   0.10↑   |   25.52   |   0.27↑   |   29.72   |   0.41↑   |
| EDSR_S |  exp(s)^(1/4)   |   31.86   |   0.25↑   |   28.35   |   0.13↑   |   27.40   |   0.10↑   |   25.51   |   0.26↑   |   29.72   |   0.41↑   |
| EDSR_S |log(s)-min(log(s))| **31.89** | **0.28↑** | **28.39** | **0.17↑** | **27.42** | **0.12↑** | **25.57** | **0.32↑** | **29.74** | **0.43↑** |
| EDSR_S |       s^2       |   31.85   |   0.24↑   | **28.37** | **0.15↑** | **27.41** | **0.11↑** | **25.56** | **0.31↑** |   29.71   |   0.40↑   |
| EDSR_S |       s^3       |   31.80   |   0.19↑   |   28.33   |   0.11↑   |   27.40   |   0.10↑   |   25.53   |   0.28↑   |   29.65   |   0.34↑   |
| EDSR_S |     s^(1/2)     |   31.87   |   0.26↑   | **28.37** | **0.15↑** |   27.40   |   0.10↑   |   25.54   |   0.29↑   |   29.72   |   0.41↑   |
| EDSR_S |     s^(1/3)     |   31.88   |   0.27↑   |   28.36   |   0.14↑   |   27.4    |   0.10↑   |   25.52   |   0.27↑   |   29.73   |   0.42↑   |

The best and second-best performances are shown in **bold**. Overall, eleven various monotonically increasing functions have achieved better results than the baseline method. The best two scaling functions are linear scaling and log scaling with a slight difference as shown in the above table. Since the linear scaling function achieves a comparable performance with low computational cost, we advocate this choice. The analysis of various scaling functions will be added in the next revised version.

Again, thank you for your constructive suggestions. We found them helpful and will include them in the next revised version.

**6."The uncertainty in this paper is based on in Eq. (4), which originated from the assumption of a Laplace distribution of the L1 error function. The paper also mentions that it focuses on epistemic/model uncertainty. I am wondering how we can justify that has something to do with epistemic uncertainty, why not aleatoric/data uncertainty?"**

**Reply:** The uncertainty in our paper describes how much the model is uncertain about its predictions, indicating the difficulty of SR reconstruction. In [22], the epistemic/model uncertainty represents the uncertainty in model parameters and aleatoric/data uncertainty describes the noise in input data. We think there is a connection between model and data uncertainty, which can't be divided without any intersection. In this paper, we study the uncertainty that describes how much the model is uncertain about its predictions, which is related to not only model but also data. In this case, thanks for pointing it out, and we will change the corresponding terms in the next revised vision for better clarification. 

**Authors' Final Comments** We hope that the discussion we provided alleviates the reviewer's concerns. We are happy to answer any additional questions the reviewer has.

The paper is about single image super-resolution. The main motivation is that current existing losses are uniform for all pixels, despite the fact that more important information to recover is edges. The main proposed idea is to have adaptive losses that weigh the loss of each pixel differently, based on uncertainty. The paper introduces variance estimation characterizing the uncertainty on a pixel-by-pixel basis. The method simultaneously computes the targeted pixels and their corresponding uncertainty.  1. The proposed formulation of estimating the super-resolution pixels and their uncertainty simultaneously is interesting. In the super-resolution community, it seems to be novel. However, I am not particularly sure if it is novel in the larger deep learning communities.  While the paper discusses the novelty and contributions in the context of super-resolution, I am wondering if there is any novelty or contributions in the context of uncertainty computation. How is the proposed uncertainty computation compared with the existing methods?

2. In the experimental results, while the comparisons are shown for super-resolution results, there is no comparison shown for the estimation of uncertainty. How is it compared to [22]?

3. I am also wondering how good the uncertainty prediction is here. The paper should show the error map (the heatmap representing the distance between the estimated high-res and the ground-truth high-res) next to the estimated uncertainty map.

4. In Fig.3, there is a module called “nearest upsampling”, however there is no discussion on this or its design. 

5. Line 179: “Linear scaling would be a natural option”. What is the theoretical reason for this? How can we decide if it has to be linear or non-linear? How do we know that the computed uncertainty has to be linear as a weight? Is this linear scaling a general rule  beyond super-res?

6. The uncertainty in this paper is based on $\theta$ in Eq.(4), which originated from the assumption of a Laplace distribution of the L1 error function.The paper also mentions that it focuses on epistemic/model uncertainty. I am wondering how we can justify that $\theta$ has something to do with epistemic uncertainty, why not aleatoric/data uncertainty?
 
No limitations are mentioned or discussed.


This paper proposes a novel uncertainty-driven loss for super-resolution. The authors demonstrate that high variance (uncertain) regions in an image should be prioritized for SR. To this end, they propose a 2-step framework, where the variance map is predicted first (step 1) and SR is learned with a loss using the predicted variance map. With the proposed uncertainty-driven loss (UDL), larger weights are assigned to uncertain regions. UDL loss is shown to improve three SR backbone networks.  Pros:
* Differentiating how to handle uncertainties in low-level vision tasks (compared to high-level vision tasks) is meaningful: for low or high level vision, large variance regions should be prioritized or attenuated and epistemic or aleatoric uncertainty should be analyzed.
* Assigning larger weights to larger variance regions lets the network to focus on challenging areas.
* 2-step training strategy is a smart way to prevent the variance map from degenerating to zero.
* Mathematical modelings provide insights (epistemic uncertainty modeling, Bayesian modeling, sparsity prior, etc).
* The proposed UDL loss is simple and can be used on any SR model.
* Theoretical background and analyses made throughout the paper is thorough and insightful.
* Larger performance improvements can be achieved on lightweight models, which would be useful in practical scenarios. This is reasonable as big CNN models would already be able to handle the uncertain regions to a certain extent.

Cons:
* Although adding the proposed loss would not impact computational complexity during inference, computational cost for training the framework (e.g. training time) may be increased. Providing computational complexity evaluation for the training stage would be useful.
* A linear scaling is adopted in Eq. (10). Has this been empirically verified? What about other types of monotonically increasing functions? (e.g. exponential, log, exponentiation, etc)
* Limitations of the proposed method are not discussed. A potential limitation would be longer training time due to the 2-step training.
* Societal impacts are not discussed.

Other comments:
* An observation $-$ in Table 3, improvement seems to be larger for datasets with more high-frequency regions (Urban 100 and Manga 109) for the lightweight model EDSR-S, showing that using the proposed uncertainty model helps for challenging images.
* Marking best results in Table 1 and 2 would be helpful for the readers.
* Typos: (i) Table 3 and 4, GARM -> GRAM, (ii) Table 4 caption, DB -> BD
* Line spacing in Conclusion seems a bit tight. Limitations and societal impacts have not been addressed. Please refer to comments above.

Meta-review: This work presents a method for single image super-resolution based with emphasis on challenging situations such as textured and edge pixels with high uncertainty. The proposed uncertainty-driven loss improves results over other common losses (MSE and L1). All reviewers liked the paper. They asked some questions such as connection to other works, computational complexity at training and inference time, the reason why linear scaling would be a natural option, and the meaning of uncertainty in the proposed scheme. The authors provided a strong rebuttal and all reviewers at the end lean toward accepting the paper. I concur with them and recommend accept. Please make sure to include items suggested by reviewers in the final revision (e.g. comparison with other types of the weighted loss function).

Decision: accept