Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a novel loss function using the standard cross-entropy loss along with a regularization term on logits for training the Dirichlet prior network. The benefit of using Dirichlet prior network is that it can distinguish the in-domain noisy data and completely out-of-domain data. For in-domain noisy data, the Dirichlet distribution should be sharp but in the middle of the simplex, while for the OOD data, the Dirichlet distribution should be flat. The Dirichlet prior network is proposed by Malinin & Gales (2018), the new method in this paper overcomes the challenge of training the network based on the KL divergence which cannot work well for dataset with large number of classes. The paper is well written and easy to follow. Here are some comments and questions: 

- In the proposed loss function, could you explain what is the reason that you choose to use sigmoid(z_c(x)) instead of \sum exp(z_c(x))? As you mentioned in the paper,  \sum exp(z_c(x)) suggests the sharpness of the distribution. Shouldn’t using \sum exp(z_c(x)) be more direct than the sigmoid(z_c(x))?

- The methods requires OOD dataset for training. The authors used the same OOD dataset for training and test. One concern is that what if the OOD dataset for test is not available at training. What is the alternative plan? How does that perform?

- In Table 1, for Gaussian in-domain dataset, why is \sum exp(z_c(x)) able to distinguish Gaussian in-domain from original in-domain images? If I understand correctly, Gusaain in-domain should have a sharp distribution which means the differential entropy (D. Ent) is small (as illustrated in Figure 2(d)), and the sum of the exponential of logits (\sum exp(z_c(x))) should be large. For the original in-domain, it should also have a sharp distribution with small differential entropy and large sum of the exponential of logits. Then I don’t understand why in the table, the AUROC and AUPR for the measure \sum exp(z_c(x) are very high values while that for the measure D. Ent are very low.

- Could you also compute the sum of the exponential of logits for the synthetic data, since it is the only metric that is evaluated in the real data experiments but not in the synthetic data?

- Could you clarify how you compute the differential entropy of the Dirichlet distribution given an input x? Did you use \alpha_c = exp(z_c(x))? 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes an improved DPN framework with a novel loss function, which uses the standard cross-entropy loss along with a regularization term to control the sharpness of the output Dirichlet distributions from the network.The proposed loss function aims to improve the training efficiency of the DPN framework for challenging classification tasks with large number of classes 

The proposed improved DPN is very incremental. It only adds a simple regularization term to the standard cross-entropy loss. The regularization term is the precision of the Dirichlet from the DPN. The technical novelty and contribution is not significant. 

The paper claims the proposed loss function allows distributional uncertainty to be modelled separately from data uncertainty and model uncertainty, and the proposed framework can improve efficiency. However these claims lack of sufficient support. 

Moreover, determining the source of uncertainty is just a mean of achieving better classification model, not the goal. The experimental results are not very convincing in improving classification over OOD examples due to the lack of comparison with state-of-the-art related works. Note many domain adaptation methods can handle OOD examples. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work studies the predictive uncertainty issue of deep learning models. In particular, this work focuses on the distributional uncertainty which is caused by distributional mismatch between training and test examples. The proposed method is developed based on the existing work called Dirichlet Prior Network (DPN). It aims to address the issue of DPN that its loss function is complicated and makes the optimization difficult. Instead, this paper proposes a new loss function for DPN, which consists of the commonly used cross-entropy loss term and a regularization term. Two loss functions are respectively defined over in-domain training examples and out-of-distribution (OOD) training examples. The final objective function is a weighted combination of the two loss functions. Experimental study is conducted on one synthetic dataset and two image datasets (CIFAR-10 and CIFAR-100) to demonstrate the properties of the proposed method and compare its performance with the relevant ones in the literature. The issue researched in this work is of significance because understanding the predictive uncertainty of a deep learning model has its both theoretical and practical value. The motivation, research issues and the proposed method are overall clearly presented. 

The current recommendation is Weak Reject because the experimental study is not convincing or comprehensive enough. 

1.	Although the goal of this work is to deal with the inefficiency issue of the objective function of existing DPN with the newly proposed one, this experimental study does not seem to conduct sufficient experiments to demonstrate the advantages (say, in terms of training efficiency & the capability in making the network scalable for more challenging dataset) of the proposed objective function over the existing one; 
2.	Table 1 compares the proposed method with ODIN. However, as indicated in this work, ODIN is trained with in-domain examples only. Is this comparison fair? Actually, ODIN's setting seems to be more practical and more challenging than the setting used by the propose methods. 
3.	The evaluation criteria shall be better explained at the beginning of the experiment, especially how they can be collectively used to verify that the proposed method can better distinguish distributional uncertainty from other uncertainty types.
4.	In addition, the experimental study can be clearer on the training and test splits. How many samples from CIFAR-10 and CIFAR-100 are used for training and test purpose, respectively? Also, since training examples are from CIFAR-10 and CIFAR-100 and the test examples are also from these two datasets, does this contradict with the motivation of “distributional mismatch between training and test examples” mentioned in the abstract?  
5.	The experimental study can have more comparison on challenging datasets with more classes since it is indicated that DPN has difficulty in dealing with a large number of classes. 

Minor:

1. Please define the \hat\theta in Eq.(5). Also, is the dirac delta estimation a good enough approximation here?
2. The \lambda_{out} < \lambda_{in} in Eq.(11) needs to be better explained. In particular, are the first terms in Eq.(10) and Eq.(11) comparable in terms of magnitude? Otherwise,  \lambda_{out} < \lambda_{in} may not make sense.
3. The novelty and significance of fine-tuning the proposed model with noisy OOD training images can be better justified.
review_title: Official Blind Review #3

Meta-review: In this work the authors build on the Dirichlet prior network of Malinin & Gales, replacing the loss function and adding a regularization term which improve training in the setting with a significant number of classes.   Improving uncertainty for deep learning is a challenging but very important problem.  The reviewers of this paper gave two weak rejects (one is of low confidence) and one weak accept.  They found the paper well written, easy to follow and well motivated but somewhat incremental and not entirely empirically justified.  None of the reviewers were willing to strongly champion the paper for acceptance.  Unfortunately as such the paper falls below the bar for acceptance.  It appears that the authors significantly added to the experiments in the discussion phase and hopefully that will make the paper much stronger for a future submission.
Decision: Reject