Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a new loss function for the training of single image super-resolution. The proposed method is motivated by the uncertainty of predicting different pixels. Firstly, the uncertainty of each pixel in super-resolution is estimated. This uncertainty (variance) was used during training that the pixels with higher uncertainty were weighted with higher weights. A two-step training scheme was designed. The experiments show good results compared with other uncertainty-driven methods, such as GRAM.
main_review: Major concerns:

1. The novelty of this paper is somehow limited. The author describes a probability-based framework to explain the uncertainty in super-resolution. However, this part of the statement is common. Describe super-resolution under Bayesian statistics introduce no novelty. The Jeffrey's prior part is interesting but the author did not explain too much. The whole method is similar to training SR with weighted data (hard samples have higher weights). The relationship between the proposed method and weighted training is not clear (there is no expariments and statements about it).

2. In the experiment section, the author only presents comparison between the proposed method and GARM. More experiments are expected. Can we weight the pixel using other methods? such as calculate the gradient, high gradient indicates dense textures. Or weight the loss for each pixel based on Wevelet or DCT? Is there any difference in using different methods to weight loss? Why is the proposed weighting method better?

3. More explaination and experiments about the Jeffery's prior and its function in the proposed method is expected.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a novel training scheme for end-to-end single image
super-resolution (SISR) with neural networks. The proposed scheme has two
steps. In the first step, a neural network is used to predict per-pixel
aleatoric uncertainty. In the second step, a second neural network is trained,
and the uncertainty estimated in the first step is used to weigh the per-pixel
restoration loss, so that uncertain pixels are more heavily penalized. Because
super-resolving textured regions and edges is typically more uncertain, the
authors claim that by making the model focus more on those regions, better
super-resolution results are obtained. Experimental results show consistent by
mild improvements over the baseline. In my opinion, the theoretical developments
need a stronger justification and their presentation needs to be improved.

main_review: Pros:
- Super-resolution is a one-to-many mapping so the outcome is inherently
  uncertain. It is good to see uncertainty being modeled in deep learning based
  SISR. This is a very recent trend that started with [21]. It also makes sense
  to me that the authors take into account the problem's heteorscedasticity
  since one expects uniform regions to exhibit less uncertainty in the output
  than texture or edges.
- Experimentally, the authors show  that their two-step training approach
  obtains a better perfoming model than the baseline (by 0.1 to 0.3db PSNR on
  average). Qualitative results in two examples show improvements over the
  baseline.


Cons:
- Since the goal of the work is to more strongly penalize textured and edge
  regions, a natural and simple baseline would be to optimize for the difference
  of image gradients (or other low-level fiters), or to weigh the penalization
  according to gradient magnitude. This would be much cheaper than having to
  recognize texture and edges with a secondary neural network.
- The theoretical developments in Section 3.1 have weak points that need to be
  clarified:
  - In lines 132, 137 authors say that they study only epistemic uncertainty,
    but it seems to me that what is being modeled is the aleatoric uncertainty:
    the formulation in (3) models heteroscedastic noise in the
    observations. Uncertainty of the model parameters (epistemic) does not
    appear in the presentation, to the best of my understanding.
  - In line 137 it is said that the observation noise in (3) is Gaussian but the
    likelihood equation (4) corresponds to Laplacian noise.
  - A Jeffrey's prior is introduced on one of the model's output (the estimated
    variance), by simply multiplying the likelihood function by one over the
    variance. I believe the probabilistic model intended here needs further
    clarification. Is the goal here to do a MAP formulation for this variance?
    Equation (7) is certainly confusing. The leftmost equation is invalid unless
    \theta_i is integrated over, and is independent of y_i. But then the
    probabilistic interpretation would not be clear, at least to me, since
    \theta_i is itself a function of y_i. Also, by multiplying the likelihood
    function by a factor it is no longer a probability over the x_i. I think all
    this deserves a better theoretical justification and an improved
    presentation.
  - There is no probabilistic justification for the second training step
    (weighing the reconstruction loss by the uncertainty).
- Although Figures 6 and 7 help to show the relative improvement of the proposed
  approach, it would help to include more examples and more state-of-the-art
  SISR methods in the comparison, so one can evaluate the validity of the
  approach as a general SISR method.

Minor:
- References [14] and [22] (Kendall and Gal 2017) are the same. This is a
  central reference to this work.



In my opinion, the weaknesses listed above outweight the submission's strengths.

limitations_and_societal_impact: No.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors proposed a new loss function guiding the SISR network focus more on the uncertain areas of the images. Specifically, the uncertainty map is obtained by the network trained with the proposed L_esu loss enforcing the sparsity in the map, and the final SISR network is initialized from the previous steps, and finetuned with the uncertainty-guided loss. The idea of using uncertainty guidance to train a SISR network will not increase the computational burden during inference, and the performance is demonstrated to be better than the baseline. 
main_review: Pros,

- Interesting idea of estimating the uncertainty map and adding the sparsity prior. 
- Better performance of using the guided training with the newly-proposed loss. 

Cons,

- It demonstrates the uncertainty map is mostly focusing on edges and texture regions of the image. I wonder whether the proposed method estimating the uncertainly map will be equivalent to high-pass filtering or edge detector for image preprocessing. What if there are high-level noises existing in the input images, will it highly influence the uncertainty map?
- Given a general-purpose loss function potentially good for other image restoration tasks, I wonder what it performs on more SOTA models, and whether it will perform consistently better than the baseline. Current evaluation is not enough with only three models and two tasks. Since it seems fairly easy to plug the training framework into any models, why not run more experiments to make the model more solid?

My main concern is there can be multiple similar alternatives for the uncertainty estimation, and the authors had better clarify the motivations and compare the current way with some simpler methods. Besides, the evaluated models are not enough to show the effectiveness of the model. Some potential generalization ability is not shown in the limited experiments. 

However, the simple loss function is intuitive and somehow brings some improvement on some specific models, I currently tend to accept the paper due to its strong intuition and clear presentation.

limitations_and_societal_impact: See the above comments.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a new loss for single image super-resolution by considering the uncertainty of each pixel within an image. Specifically, the authors claimed that pixels with larger uncertainty, such as texture and edge, should be more emphasized, and thus designed a loss function to achieve this goal. Experiments have been conducted to evaluated the effectiveness of the proposed method.
main_review: The idea of emphasizing pixels with larger uncertainty is new, and seems effective. Experiments have also showed clear improvements as compared with baselines.

However, I do not think technical part this work is clearly clarified. Specifically,
1. There are many priors can induce sparsity other than Jeffrey's prior, and thus it should be justify why adopt this prior. 

2. Also notice that, comparing Eq. (6) and Eq. (9), the difference only exists in the scalar multiplied to $s_i$. Therefore, it is natural to ask what the performance could be if we change this scalar to other values, which is also valid and can be derived similarly as Eq. (7) and Eq. (8), in the sense of generalize priors.

3. The final loss function in Eq. (10) is somewhat heuristic, since there are many ways to set $\hat{s}_i$ provided monotonically non-decreasing with respect to $s_i$.

Besides, many of the mathematics are not rigorous. For example,
1. In Eq. (3), the authors assume $\epsilon$ to be standard normal. However, according to equations after it, e.g., Eq. (4), $\epsilon$ should follow a Laplace distribution.

2. Eq. (7) is not correct, since $p(x_i|y_i,\theta_i)p(\theta_i)$ results in $p(x_i,\theta_i|y_i)$ according to conditional probability rule, but not $p(x_i|y_i)$.

3. Jeffrey's prior can only be written as $p(w)\propto\frac{1}{w}$, since it is not a valid distribution, and thus in Eq. (7) the equal sign after $p(x_i|y_i,\theta_i)p(\theta_i)$ should be replaced by "$\propto$".
limitations_and_societal_impact: The authors did not mention the limitations. As I can see, the major limitation of this method in practice is that it requires a two-step training, which increase the computational complexity.
review_title: null

Meta-review: This paper proposes a new weighted loss function for the training of single image super-resolution. The proposed scheme involves two steps. In the first step, the authors use a Bayesian estimation framework to predict mean and variance for each pixel in the high resolution image (not just a point estimate). Then the authors train a second neural network by weighting the reconstruction error according to the estimated variance. The intuition motivating this idea is that prediction errors on pixels with larger uncertainty (e.g. texture and edges) should be upweighted. Experimental evaluation shows that the new scheme is able to outperform the baseline.

The authors provided a constructive and high quality rebuttal, incorporating new experiments (including baselines and results using different monotonic functions to define the weighting scheme). However, all reviewers still consider the paper not ready to publish at NeurIPS.

Reviewers zyeM, oVep and H49w had concerns regarding the precision of the mathematical formulation. Specifically, several comments on the derivations in Section 3.1 and the lack of a probabilistic motivation for the second estimation step. This led Reviewer WXXg to reduce the score to 4, as she/he believes that the paper would benefit from a second revision. Reviewer H49w also expressed the same opinion.

Reviewer zyeM provided a score of 5 (updated from 4 after the rebuttal), although she/he did not change it in the system. While Reviewer zyeM considers the general direction of the work interesting (the investigation of weighting schemes for the SISR problem) and appreciated the detailed response provided by the authors, she/he still recommends rejecting the paper. The main reason is the lack of significant technical novelty in their approach. 

Reviewers H49w and zyeM, suggested incorporating simple baselines that could identify the textured regions in the image without the need of training a full system. The authors added in their rebuttal convincing baselines, showing that the proposed approach can lead to singnigicantly better results. 

All four reviewers recommend rejecting the work. The AC agrees with the concerns raised by the reviewers and finds no basis for overruling their recommendation. 

Decision: Accept (Poster)