Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Prior Networks (Malinin & Gales, 2018) use Dirichlet prior over categorical predictive distributions to distill ensembles for classification tasks. This paper extends Prior Networks to the regression setting by using a Normal-Wishart prior in order to attempt to match the predictive diversity. The authors define the model and loss terms including analytical derivation and evaluate their proposed approach with synthetic data, UCI datasets and monocular depth estimation. 

_Strengths_:
- The paper is well-written and clearly structured.
- Most design choices are justified.
- Simple idea (in a good way!) which seemed to work well, shown by the evaluation.

_Weaknesses_:
- Most of the work seems to be heavily based on Prior Networks (Malinin & Gales, 2018). Even Section 2.1 seems to be exactly like the Subsection in the paper about Prior Networks. This paper mainly focuses on an extension to the regression task. Therefore, the contribution / novelty of this paper is incremental. However, I still think the authors did a good job to present a general distillation method for regression task. Therefore, I would consider the novelty a minor weakness.
- I am on the fence about specifying the OOD dataset for learning with the loss in Eq. 8. I believe it is difficult to decide what kind of model to use for generating the OOD dataset, thus, the model choice can lead to large differences in performance. This is not really discussed. Further, the models trained have more data available for training, I believe it is not quite fair to compare against models which only have been trained on in-domain-data.
- There are no comparisons to other approaches for distillation of regression tasks. I understand, that this paper wants to show a viable general approach for regression distillation, however, this work is not the first one to do so and therefore should consider existing work.

_Overall assessment_: For me, this paper is borderline. The weaknesses, especially the OOD dataset used for training and the lack of comparisons in the evaluation are concerns. However, I like the idea and the execution so therefore, I would recommend a weak accept (6).

_Detailed comments and questions_:
- OOD data: I have seen that you have an ablation for the degree of regularization on the OOD dataset. However, what about different OOD data? Why choose KITTY and not a different dataset? Were there any large difference in performance?
- Table 3: I notice that NLL performance of distilled models are better than the actual ensemble, how can this be?
- OOD detection for monocular depth estimation: Did you also trained the comparing models with the OOD data, e.g. DD?
- Comparing models: Have you consider comparing your model to other ones, e.g. [1, 2]? This could improve your paper and approach to show that it also consider existing work on regression distillation.

_Post-rebuttal_:
I really appreciate the authors adding baseline models to the paper and missing citations. I do think this improves the overall paper by a lot. As mentioned already in my paper, I do believe this is a nice idea and executed well, even though novelty might be limited. I am keeping my score and recommending an accept.

[1] Chen, G., Choi, W., Yu, X., Han, T. and Chandraker, M., 2017. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems (pp. 742-751).
[2] Saputra, M.R.U., de Gusmao, P.P., Almalioglu, Y., Markham, A. and Trigoni, N., 2019. Distilling knowledge from a deep pose regressor network. In Proceedings of the IEEE International Conference on Computer Vision (pp. 263-272).
review_title: Official Blind Review #1

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper extends Prior networks models, previously introduced for classification, to regression problems.  Prior networks are neural networks whose main target is to "modelling uncertainty in classification tasks by emulating an ensemble using a single model".  Standard Prior networks models output the parameters of a Dirichlet probability distribution. This Dirichlet probability distribution then defines a distribution over categorical probability distributions over the different classes. This hierarchical approach allows to better capture uncertainty. The presented approach extends this framework to regression tasks. So, instead of returning the parameters of a Dirichlet distribution, it returns the parameters of a Normal-Wishart distribution, which then defines a probability distribution over Normal distributions, and, in turn, each Normal distribution defines a probability distribution over the value of the target variable.  


Pros:
* The presented approach is sound and addresses a relevant problem, which is modelling uncertainty for regression problems. 
* A method for distilling an ensemble model into a single model while maintaining accuracy is also proposed. 
* The proposed approach does not incur in computational and memory overheads like standard deep ensembles. 
* This work properly approaches technical difficulties (such as employing numerical stable precision parametrizations of the Normal-Wishart distribution) that arise in this kind of problems.

Cons: 
* The presented approach does not introduce any novel idea or insight. It's a relatively simple extension of a previously published method. 
* The empirical results do not show a clear advantage of the presented approach wrt previously published proposals. 
* The advantage of having a small computational and memory overhead is not properly evaluated with other proposals which also have a small  computational and memory overhead [1] (although this proposal has not been defined for regression problems, the adaptation to regression is as simple as the adaptation of the DeepEnsembles models employed in this work). 


I can not recommend the acceptation of this paper because I find the originality of the work quite limited. Although the extension of prior networks to regression task is mot really straightforward because of technical issues related to the problem of learning the parameters of a Normal-Wishart distribution. The general strategy to do that exactly matches the previous steps employed when introducing prior networks.  In consequence, this work does not provide any new relevant insight into the problem of modelling uncertainty and learning models with well-calibrated predictions. 


Minor comments:
- Eq (14): T parameter is not defined. Temperature? 	
- Typo at the end of Page 5: [-25,20] --> [-25,-20]
- ENSM is defined after Table 1. 
- Fix the following reference:
Andrey Malinin and Mark JF Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. 2019. 

Post-rebuttal:  I thank  the authors' effort for the improvement of the manuscript following the comments of the different reviewers. I think the overall quality of the paper has really improved. But, after many thoughts, I still think there is a limited novelty in this paper. I have increased my score to 5. But I can not recommend this paper for publication. 

  adding baseline models to the paper and missing citations. I do think this improves the overall paper by a lot. As mentioned already in my paper, I do believe this is a nice idea and executed well, even though novelty might be limited. I am keeping my score and recommending an accept.

[1]  Wen, Y., Tran, D., & Ba, J. (2020). BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning. arXiv preprint arXiv:2002.06715.
review_title: A simple extension of Prior networks models to regression. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses interpretable uncertainty quantification for data driven models. In particular, the authors focus on a sub-class of methods known as Prior Networks and attempt to extend these methods to regression tasks as existing approaches address classification only. The author contribution is thus clearly stated and positioned w.r.t. prior arts and tackle a non-trivial issue.

In the classification setting, the Dirichlet distribution is pretty much the universal model for the parameters of multinomial distributions. For regression, i.e. continuous r.v., there is no such universal solution and the authors chose to focus on outputs that have a normal distribution. The parameters of this latter are assumed to be normal-Wishart. Although, the proposed method is de facto non-applicable to other types of distributions, it can be argued that this already covers a majority of situations. 

The paper is rather well organized and seems technically sound. This said, a few mathematical details are missing and, most importantly, the experiments are not very convincing. These concerns, also with other minor remarks are detailed below, section by section.

sec 2.2

Maybe give the explicit definition of Z to clarify that is does not depend on network parameters.
The presence of the OOD loss term in (8) is a bit artificial as it boils down to regularizing because of the choice of beta. Is this choice systematic ?
In (9), how is p(y | mu, Lambda) computed ? Is it a T distribution ?

2.3

(12) lacks clarity : dataset is equal to an empirical distribution... Do you mean p hat is a sum of Dirac ?
What does phi represent ?

3
The acronym ENSM is not explained. I believe this corresponds to the deep ensemble. 
The Prior Networks achieve a form of disambiguation but the quality of it is a bit disappointing compared to ENSM. In particular, data uncertainty raises quickly for out-of-domain inputs. 

4

The presentation of the experimental protocol in 4 lacks clarity thereby impairing the interpretation of the results. The definition of the unconventional performance criteria [(Malinin et al. 2020] must be recalled (at least in an appendix). 
In addition, as honestly mentioned by the authors, these datasets may not offer sufficiently rich problems to provide interesting comparisons. Besides, the way that OOD data is generated does not seem to necessarily produce inputs that are not covered by the in-domain distribution. Perhaps, the authors could use a "bad GAN" to obtain such data points, I mean a GAN where the generator and the discriminator would co-operate instead of being adversaries. If it converges, the generator would produce synthetic inputs that are easy to discriminate, thus far from true inputs. 

5
While the dataset used in this section is more challenging, the experiment description is confusing. Again, performance criteria are not sufficiently explained and the general message becomes cryptic. Table 3 is overly complicated, I think RMSE is fairly enough to depict regression performances. Moreover, the definition of some columns are missing.
In Table 4, the performances of the methods seem quite unstable. For example, NWPN works fairly well for a given dataset configuration for one knowledge uncertainty criterion but fails miserably using another criterion on the same data.
On Fig 3, from what dataset are these image coming from ? Why are these or that object presumably "unknown" to the model ?
I think the whole section deserves some re-writing.

Final remark : there are a few English mistakes that should be wiped out. 


review_title: Clarifications needed

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of the Paper:

        This paper introduces regression prior networks. These are models that aim at capture predictive uncertainty, both epistemic and aleatoric, in the context of regression problems. Regression prior networks can also be used to compress an ensemble of predictors into a single model while keeping the benefits of the ensemble. That is, better predictive performance and uncertainty estimates. The method is validated on several problems from the UCI repository and compared with ensemble methods.

Specific details:

        I believe that this is a nice paper that illustrates an appealing method for uncertainty estimation in the context of neural networks. My main concern, however, is that it builds heavily on previous work. In particular, prior networks have already been proposed for classification and they have also been used to distill (compress) an ensemble. There is hence not much novelty here, only the extension to regression problems since, previously, only classification problems have been addressed. The use of prior networks for ensemble distillation is also not new. All this questions the novelty of the proposed approach.

        The extension to regression seems to follow very closely the work already carried out for classification. The only difference is that a Normal Wishart distribution is used instead of a Dirichlet distribution.

        The experiments carried out are extensive and consider different tasks involving prediction accuracy and out of distribution data detection. My main concern, however, is that no comparison is carried out with alternative methods to estimate prediction uncertainty such as those of Bayesian neural networks using variational inference or dropout. The authors should comment on the advantages of their method with respect to these techniques.

        The method proposed is also complicated and has several training parameters. The authors give specific values for them, but it is not clear the motivation for them or the sensitivity to their values.

        The paper is clearly written but heavily relies on previous work, making the reading difficult for someone who is not familiar with it. The paper is not self-contained.

        Summing up I believe that this could be an interesting contribution for the conference, suffering from a reduced amount of novelty.
review_title: An nice paper based on incremental work

Meta-review: This paper presents a useful contribution to the growing literature on uncertainty estimation with deep learning. The review process has significantly helped with strengthening this paper, specifically with the concerns about novelty and sufficient comparisons to existing work. I hope you will continue to improve this work for submission to a future venue.
Decision: Reject