Dirichlet Wrapper to Quantify Classification Uncertainty in Black-Box Systems
Authors: José Mena Roldán
Oriol Pujol Vila
Jordi Vitrià Marca
Venue:  (2020)
Keywords: uncertainty
black-box classifiers
rejection
deep learning
NLP
CV

Abstract: Nowadays, machine learning models are becoming a utility in many sectors. AI companies deliver pre-trained encapsulated models as application programming interfaces (APIs) that developers can combine with third party components, their models, and proprietary data, to create complex data products. This complexity and the lack of control and knowledge of the internals of these external components might cause unavoidable effects, such as lack of transparency, difficulty in auditability, and the emergence of uncontrolled potential risks. These issues are especially critical when practitioners use these components as black-boxes in new datasets. In order to provide actionable insights in this type of scenarios, in this work we propose the use of a wrapping deep learning model to enrich the output of a classification black-box with a measure of uncertainty. Given a black-box classifier, we propose a probabilistic neural network that works in parallel to the black-box and uses a Dirichlet layer as the fusion layer with the black-box.  This Dirichlet layer yields a distribution on top of the multinomial output parameters of the classifier and enables the estimation of aleatoric uncertainty for any data sample.  
Based on the resulting uncertainty measure, we advocate for a rejection system that selects the more confident predictions, discarding those more uncertain, leading to an improvement in the trustability of the resulting system. We showcase the proposed technique and methodology in two practical scenarios, one for NLP and another for computer vision, where a simulated API based is applied to different domains. Results demonstrate the effectiveness of the uncertainty computed by the wrapper and its high correlation to wrong predictions and misclassifications.

Under review as a conference paper at ICLR 2020
DIRICHLET WRAPPER TO QUANTIFY CLASSIFICATION UNCERTAINTY IN BLACK-BOX SYSTEMS
Anonymous authors
Paper under double-blind review
ABSTRACT
Nowadays, machine learning models are becoming a utility in many sectors. Al companies deliver pre-trained encapsulated models as application programming interfaces (APIs) that developers can combine with third party components, their models, and proprietary data, to create complex data products. This complexity and the lack of control and knowledge of the internals of these external compo- nents might cause unavoidable effects, such as lack of transparency, difficulty auditability, and the emergence of uncontrolled potential risks. These issues are especially critical when practitioners use these components as black-boxes new datasets. In order to provide actionable insights in this type of scenarios, in this work we propose the use wrapping deep learning model to enrich the output of classification black-box with a measure of uncertainty. Given a black-box classifier, we propose a probabilistic neural network that works in parallel to the black-box and uses Dirichlet layer as the fusion layer with the black-box. This Dirichlet layer yields a distribution on top of the multinomial output parameters of the classifier and enables the estimation of alcatoric uncertainty for any data sample. Based on the resulting uncertainty measure, we advocate for a rejection system that selects the more confident predictions, discarding those more uncer- tain, leading to an improvement in the trustability of the resulting system. We showcase the proposed technique and methodology in two practical scenarios. one for NLP and another for computer vision, where simulated API based is applied to different domains. Results demonstrate the effectiveness of the uncer- tainty computed by the wrapper and its high correlation to wrong predictions and misclassifications
INTRODUCTION
The popularity of machine learning is giving birth to new business models based the productiza- tion and service of these models. the market there are many application programming interfaces (APIs) serving predictions in object recognition for images (Vision A language detection or timent analysis in natural language processing (Cloud Natural Language APC). to mention just a few. As this Machine Learnin--asa--erricce model starts to grow, becomes easier find these APIs as an integral component of more complex products.
The of pre-trained models gives rise these models are going to operate our intended application domain. In order to address this issue, there is vast literature on transfer learning that can be applied. However, when using third-party proprietary software or APIs, we may not have access to the internals or the possibility of fine- tuning the model to our domain. If we are to use the model as it is, one must at least understand when the model is going to work and when it is not, to have some confidence metric that tells about the expected performance of the methods when applied to our problem. However, this information is not always provided, especially in deep learning models. This effect can be worsened when these components are just one of the many different parts of a data product. This complexity leads us to the second problem: when different models might interact in complex pipelines, the construction of the appropriate confidence measures can be a challenging task.
different problems. First, we do not know how
https://cloou..ooggl
language/

---

Under review as a conference paper at ICLR 2020
In order to solve the previous issues, in this article, we propose deep learning wrapper algorithm that equips any black-box model with uncertainty prediction. Here a wrapper understood as a machine learning model that takes any other model and operates without accessing its internals. Because it does not have access to the internal states, parameters, or architecture of the model it is wrapping. the wrapper is model agnostic and be used on top of any other algorithm long as it satisfies some desideratum. In this article, we only require the black-box model to produce as output a distribution over the classes, soft requirement as any model with soft-max layer satisfies it. More specifically. the proposed wrapper uses a deep learning model and introduces a Dirichlet layer as the fusion layer with the black-box. This Dirichlet layer yields distribution on top of the multinomial output parameters. By sampling from this Dirichlet distribution, the wrapper enables the estimation of aleatoric uncertainty.
Uncertainty has been important topic in machine learning for many years (Koller & Friedman, 2009). With the emergence of deep learning, the reinterpretation of some existing mechanisms such as dropout, or the proposal of stochastic mechanisms such Montecarlo approaches, has broadened the use of these techniques for accounting for uncertainty in deep models (Gal 2016).
Uncertainty can be categorized into epistemic and aleatoric uncertainty. While the first accounts for the uncertainty that is associated to model parameters, the second corresponds to the uncertainty inherently present in the date
Uncertainty plays a key role when reporting a decision because it accounts for the reliability of the prediction and can help to show the limitations of the applicability of machine learning model. In this respect, we advocate for the use of selective prediction (aka rejection techniques) when the uncertainty metric large in order avoid potential harm avoid risks. Selective prediction is set of techniques based on abstaining from deciding according to some metric threshold. As previously commented, uncertainty is good candidate for rejection metric. In literature, we find examples of different rejection functions (Geifman & EI- Yaniv 2019) (De Stefano et al. 2000) and some of them use uncertainty measures (Geifman & El- Yaniv 2017) for rejection.
In the proposed scenario, where we are trying to characterize the uncertainty of a black-box non- mutable model, many of the ofate-of-the-arr techniques are not applicable. For example, some re- jectors have to be trained together with the classifier and need access to the internals of the model. In the same line, current models for uncertainty in deep learning need to have access to its internal states (Gal 2016).
The contributions of this article can be summarized as follows:
We propose wrapper algorithm that equips any other classification model that outputs distribution over predicted classes with Bayesian treatment without having knowledge or access its internals.
We use the wrapper to empirically estimate alcatoric uncertainty and show that the com- puted uncertainty can identify prone to err samples.
Finally, show that the computed uncertainty can be used by rejection techniques to in- crease the performance and robustness of the original black-box model in the target domain. We show improvements in transfer problems in natural language processing and computer vision problems.
In section 2, we introduce the method proposed for building an uncertainty wrapper around a black- box model. In section 3, we describe how to obtain an uncertainty score from the wrapper output. Section 4 introduces the concept of rejection and rejection performance metries. showcase the proposed method in four different scenarios for sentiment analysis in natural language processing and one for computer vision. The results obtained corroborate the importance of the rejection method and show the success of the proposed methodology. Finally, section 6 concludes the article.
section 5, we
'Observe that in this application only aleatoric uncertainty matters since we are dealing with tre-trained, non-mutable models. In this particular scenario, aleatoric uncertainty also serves as measure of the fitness of the model to the data.
2

---

Under review as a conference paper at ICLR 2020
2 BUILDING AN UNCERTAINTY WRAPPER
Our goal is to build a wrapper algorithm that takes another black-box model and operates on top of it. As such, there are several constraints to observe. First, we need to exclusively operate on the inputs and outputs of the black-box classifier. We are not allowed to use any intermediate internal value of the black-box model as we need to agnostic to it. Second, the input of the wrapper has to be compatible with the original distribution over the output classes
In the literature, other proposals suggest a deep learning model for estimating uncertainty. The problem with those approaches, like in (Kendall & Gal 2017) where they independent Gaussian random variables to model the pre-activation value of the logits, is that they do not conform to the constraints in our setting. First, having access the logits before the softmax breaks the black-box assumption; and, second, independent Gaussian distributions impose unnecessary assumptions and need of additional normalization steps. A more natural approach is to consider the output distribution coming from a Dirichlei probability distribution function.
2.1 DIRICHLET CONCENTRATION REPARAMETERIZATION
As commented, given a data set 'D composed of pairs (ris Wi), I. with yi E RC, being C the number of different classes, the wrapper output is assumed to come from Dirichlet probability density function:
ply X, w²) Dir(a),
where w" the parameters of the wrapper. We propose parameter two terms to relate the output of the black-box classifier, Ym. with the concentration parameter, a, in the Dirichlet distribution of the wrapper. To that effect, we recall some basic statistics of the Dirichlet distribution. Given a Dirichlet random variable x E IRC with concentration
a decomposition of the concentration
parameter RCC the expected value of the distribution is defined E(x) a Σ a..
Observe that the expected value has the same properties as a probability distribution and that the output of the black-box ym E RC is already a probability distribution. In this sense, we could directly use the output of the black box as the concentration parameter. However, each term of the concentration parameter not necessarily constrained to the interval [0, 1]. Let introduce new scalar parameter, 3 E IR that will model this difference, such that a B8mm
Observe that the value of 3 does not change the expected value of the output of the wrapper and coincides with the output of the black-box model, i.e. B(yw) Ym.
(a) B
(b) 3
(c) B 20
Figure 1: Dirichlet distribution in 3 dimensions for different B values given a prediction of [0.25, 0.25,0.50]
This decomposition has a simple interpretation: While the output of the black-box classifier stands for the mean, parameter 3 accounts for the spread of the distribution. The same or similar decompo- sition can be found in other works in different context(Malinin & Gales ( An example of the effect of varying this parameter in three dimensional Dirichlet distribution shown Figures (Ta) to Ic). Observe that the higher the value of 3, the more pointy the distribution is.
al. 2018
'I is worth noting that in the context of those works, there is degradation in performance when using Dirichlet. This does not happen in our case since the black-hox model is non-mutable.
3

---

Under review as a conference paper at ICLR 2020
Figure 2: Model used estimate the aleatoric uncertainty from the original black-box model
Dense solpks
Dense ndu
Derse rdu
Onge mode Unoatairy Werper
rec
This decoupling allows to effectively isolate the contribution of the black-box and the contribution that remains to be computed, i.e. the value of parameter 3. Figure 2 shows the integration of the wrapper (in light orange colour) with the black-box classifier (in light blue colour). Observe that the wrapper consists of two blocks: the Dirichlet reparameterization layer of the wrapper that decouples the influence of the black-box model from the rest (see the dashed line), and deep learning architecture which aims compute the scalar value of 1
2.2 INFERENCE THE DIRICHLET SETTING
SImilarly to (Kendall & Gal 2017 we approximate the expected value of the classification prob- abilities using Monte Carlo sampling from the learned Dirichlet distribution for each sample, U., ~ (ir( as [[j = M Σm= Dm,i-
This distribution is used define the loss function for our learning stage. Given of N training samples, we will use a regularized version of the cross-entropy loss function as follows:
N
C
N
C yo log
M Σ PrLm
11
((W) = N
Σ cml
Vi.o log ElDile + A||5||2 = NC
(m.in) + A1131122
M
Observe that we introduce the norm of the 3 value in the minimization function. This term is required since the unregularized cross-entropy forces the value of grow unbounded. By adding this term, we control its growth and govern the trade-off with a scalarization parameter X.
3 OBTAINING AN UNCERTAINTY SCORE FROM THE WRAPPER
The described Dirichlet layer effectively allows studying the variability of the parameters of the black-box output. This variability can be used to approximate a value for the heteroscedastic aleatoric uncertainty. In this work, we use Monte Carlo simulation sampling from the obtained Dirichlet function order to characterize the uncertainty GGal 2016).
Standard techniques for measuring uncertainty includes variation ratios or predictive entropy. Vari- ation ratios measures the variability of the predictions obtained from the sampling (Freeman) 1965) by computing the fraction of samples with the correct output. Alternatively, predictive entropy considers the average amount information contained in the predictive distribution. Those re- sults with low entropy values correspond to confident predictions, whereas high entropy leads to large uncertainty. Since the output the black-box model y" already describes a probability dis- tribution, one could compute its predictive entropy and obtain measure of its uncertainty with HI -- y
Σ.
log y
"The architecture used in this figure corresponds the one used in the experimental section.
4

---

Under review as a conference paper at ICLR 2020
However, the wrapper allows model the variability of the parameters of the black-box output distribution, we can compute a predictive entropy that takes into account the variability of the pre- dicted value. In this case, the sampled predictive entropy is defined H Ee Bly]e log E[ As we show in the experimental section, this latter approach captures better the uncertainty com- pared the predictive entropy of the original model.
USING UNCERTAINTY FOR REJECTION
Rejection is a mechanism that, given a particular metric related to the confidence the decision, allows discarding prediction if the metric value is below some threshold. In our proposal, we use the wrapper computed uncertainty as this rejection metric. In the context of our use cases, the hypothesis is that texts images with high uncertainty prone to be misclassified by the black- box model.
In order to the uncertainty score for evaluating the performance of the black-box in dataset, we first proceed to obtain the predictions applying the original model. Then for each pair of data and prediction, the predictions based on the uncertainty score, from more uncertain to more confident. From that ordering, we the rejection threshold that marks where start trusting the classification model.
obtain the associated uncertainty score using the wrapper. Next, we sort
Figure 3: Rejection performance metrics as proposed in Condessa et al. 2015
dessification
dasstfler
M
natrglected
(ADAN]
classiiiction rejection
dota
clociied
Miww
MiR
[M+
rejector
AnW
AnR
rejectionn quality (ACRR) [MI JAnRi
reected frorton
In order to evaluate the rejection metric, we split the dataset using two criteria: whether the method Rejects the data point or Not; and whether the point is Accurately classified, or Misclassified named as R, N, A M respectively. Using this terminology, we follow the guidelines in (Condessa et al. 2015 for rejection quality metrics. We have three quality metrics, illustrated in 3
Non-rejected Accuracy measures the ability of the classifier to classify non-rejected sam- ples accurately: NRA
N
INT
Classification Quality measures the ability the classifier with rejection to elassify non-
rejected samples accurately and to reject misclassified samples: CQ = JAΩ N
|
Rejection Quality measures the ability to concentrate all misclassified samples onto the JM || set of rejected samples:RQ = inn M|
good rejection point will show a trade-off between the three metrics, being able to divide the misclassified predictions from the right ones and preserve only those points that provide useful information. The higher the value displayed, the better that metric performs for rejection.
5 EXPERIMENTS AND RESULTS
This section describes the experiments run for validating the wrapper proposal and results obtained. The experiments include two different scenarios: a use case for sentiment analysis using natural language processing and, another, for image classification.
5

---

Under review as a conference paper at ICLR 2020
5.1 A NATURAL LANGUAGE PROCESSING SCENARIO
In order to validate the proposal, we use an NLP-based sentiment analysis system applied to product reviews. The goal of the system classify each review on whether it positive negative. The goal of the experiment is two-fold. First, we want to show how to apply the wrapper for given NLP task. Second, we demonstrate how the proposed method additionally captures the uncertainty caused by the change in domains. To this end, we include different combinations of training and prediction domains in the experiment. The details on the datasets used are the following:
Stanford Sentiment Treebank (Socher al. 2013). SST-2, binary version where the task is to classify a movie review in positive or negative. The dataset is split in 65,538 test samples, 872 for validation and 1,821 for testing.
Yelp challenge 201 the goal is classify reviews about Yelp venues where their users rated them using to 5 stars. To be able to reuse a classifier trained with the SST-2 problem, we transform the Yelp dataset from multiclass set to binary problem, grouping the ratings below three as a negative review, and as positive otherwise. The dataset is split in 186,189 test samples. 20,691 for validation and 22,991 for testing.
Amazon Multi-Domain Sentiment dataset contains product reviews taken from Ama- zon.com from many product types (domains) (Blitzer et al., 2007). As in Yelp, the dataset consists on ratings from 1 to 5 stars that we label as positive for those with values greater or equal to 3, and negative otherwise, split into training, validation and test datasets. We use of the domains available: music (109.733/1..193/52222 examples) and electronics (14,495/1,61) 1/6,,990 examples).
5.2 AN IMAGE CLASSIFICATION SCENARIO
In addition to the NLP use case presented above, we include here a use case for image classification. The task, in this case, is to classify images in one of the categories defined in the dataset. As in NLP, an image classifier trained using a source dataset, acting as the original API, is then applied to a new set of images belonging to a different dataset. Both datasets share almost the same output classes except for one. By predicting the uncertainty of the different class, we will show how the predicted uncertainty can also be used to detect out of sample images. The details on the datasets used for the vision use case are the following:
STL-10 (Coates et al. 2011). The STL-10 dataset is an image recognition dataset for de- veloping unsupervised feature learning, deep learning, self-taught learning algorithms. is inspired by the CIFAR-10 dataset but with some modifications. It includes 500 training images, 800 test images per class, belonging 10 classes: airplane, bird, car, cat, deer, dog, horse, monkey, ship, truck.
CIFAR1O (Krizhevsky) 2009). The CIFAR-10 dataset consists of 60000 32x32 colour im- ages in 10 classes(airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck), with 6000 images per class. There are 50000 training images and 10000 test images.
5.3 EXPERIMENT SET UP
On every experiment, we use two datasets: (i) a source dataset for training a model, that will be considered the black-box model from that moment and (ii) target dataset that corresponds to the domain we want to apply the black-box model and where to measure the uncertainty using the proposed wrapper. Specifically, the steps followed each case are:
Train the black-box. First, we train a classifier with the source dataset. In real scenarios, this step would not be necessary as we would be using a pre-trained model or third-party API.
Apply the black-box to the target domain. In this step, we use the black-box to obtain the predictions and evaluate the accuracy of the target dataset, and we can compute the predictive entropy based on the prediction outputs.
https::/wwww htt.:/eell..commaaaaettt//hhlll

---

Under review as a conference paper at ICLR 2020
Compute the uncertainty for the target domain using the wrapper model. Once have the predictions for the target domain, we proceed to train the uncertainty wrapper to ap- proximate the Dirichlet pdf for each input. By sampling the pdf, we compute the sampling predictive entropy of the average of the outputs to get the uncertainty score for each element in the target dataset.
Apply the rejection mechanism. Finally, we use the uncertainty score to sort the predictions from more to less uncertain, and we search for a rejection point that maximizes the three performance measures: non-rejected accuracy, and classification and rejection quality.
We run five different scenarios, including the training the black-box with the Yelp dataset and apply- ing it to SST-2 and vice-versa, training the black-box with the Amazon electronic products reviews dataset and applying to Amazon music products reviews, and rice-versa, and training with STL-10 and applying the black-box to CIFARIO. For each scenario, a selection of optimal training param- eters was carried out, including learning rates, batch sizes, number of units and number epochs. Details on the architectures used for the black-box are given in the Appendix A
5.4 RESULTS
In order to show the effect of the application of the uncertainty wrapper on each of the target do- mains, we compute the uncertainty score using the three different metrics described in section 4: the predictive entropy of the black-box output (baseline), the predictive entropy obtained after training the aleatoric wrapper (pred. entropy), and the variation ratios (var. ratios). Figures 4 to g show the results obtained on each combination for the rejection performance metrics for the three uncer- tainty scores analysed. From left to right, we find the values for non-rejected accuracy, classification quality and rejection quality. The higher the value in the plot, the better the result.
According to the results obtained, the proposed method shows better behaviour in all scenarios and metrics. As we remove more samples according to the uncertainty, the proposed method displays much better accuracy and quality than its counterparts. These results validate the hypothesis that the heteroscedastic aleatoric uncertainty computed by the wrapper effectively captures the confidence in the prediction and the samples prone to error. On the contrary, variation ratios are the worst perfor- mant method. Note that, although our proposal performs much better, its absolute gain depends on the scenario. In those domains where the black-box model performs worse, there is more to gain by using the wrapper. If we observe the classification quality (plot at the center of each figure) and the rejection quality, we can see that the proposed metric is also excellent at rejecting the misclassified points. A detailed table with numerical results for the same experiments is included in Appendix B
Results demonstrate how the usage of the uncertainty for rejecting uncertain predictions helps with the adaptation of pre-trained model to new domains of application. In some cases, the results obtained for the test dataset of the target domain by rejecting 10% of the less certain points overtake those obtained by the source dataset used for training the original model. As a curiosity, the use case where we trained a black-box model using the reviews of Amazon's is electronics products achieves better results when applied the test target dataset than to the original test dataset. Even in this case, where the applied classifier reaches an accuracy of more than 90 %, the proposed method increases it in almost 5 points. In Appendix [ we analyse how, for the case of images, the proposed method can detect out-of-sample images that belong to an unseen category.
6 CONCLUSIONSS AND FUTURE WORK
In this work, we introduced a deep learning wrapper technique that can endow any black-box model with uncertainty features. The wrapper uses reparameterization trick the Dirichlet distribution, and it can capture the distribution on the multinomial parameters of the output of the black-box classifier.
We use the predicted uncertainty to fuel a rejection method and show how this helps in assessing the fitness of model to domain or data set. By measuring the sampling uncertainty and using it for rejection, we can improve the accuracy results by 4%-8% by rejecting just 10% of the samples. Additionally, the method displays significant value rejection quality. These results tell us that

---

Under review as a conference paper at ICLR 2020
Non rejected accuracy
Classificaiion quality
Pejeciion quality
1.
54
boseline
var. rstoss
prod. an//a33
56
64
0.7
busefee
63
ralio
prod.
rejected points
rijected points
rejected points
Figure 4: Apply Yelp BB to SST-2
Notr rujudted aay
Clazdfiaatio quality
Majos quality
0.9
105
0.7
entropy
0.3
14
boadra na pred entroppy
03
luscita
20
Var.
pood estropy
20
43
m
100
0
20
61
60
100
25
20
25
(
103
rejected paints
rejucted points
rejucttd points
Figure 5: Apply SST-2 BB to Yelp
1.0
Non rejucted accuracy
Classfication quality
Pejlecton quality
basere
buselre
5.5
var. ratios
var. ratos
peed eetrary
peet. e0rr127
20
15
0.7
1.4
bunell
0.2
rejected paints
rejected potnts
rejected points
Figure 6: Apply electronics BB to Music
Non rejected ascuracy
Classfiiation quality
Besection quality
1.0
30
08
7%
pood entropp
20
23
basel ne vTr 131
Inselite
04
VTr. sstee
pred. entrops
pred entrang
20
60
60
100
43
60
80
300
20
60
60
100
rejected points
rejecttd potnts
rejected points
Figure 7: Apply music BB to electronics
Non rejected accurncy
Classiicttinn quality
Rejection quality
6.7
3.0
proj, und/upy
reton
0.5
2.5
prod. untrapg
var oxton
basiin rajio
06
0.5
gred. intrasy
0.7
40
40 rejected
repected points
Figure 8: Apply STL-10 BB CIFARIO
the predicted uncertainty focuses intricate, ambiguous, prone to error cases. We show results in NLP and computer vision domains with successful and encouraging results.
As future work, we are planning to keep exploring different architectures and strategies for the wrapper implementation and focus on other usual cases found in real-life implementations, such as how to deal with high dimensional and categorical outputs.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 440-447, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL http::::/www..alwwe.. org/
anthology/PO7- 1056
Wenhu Chen, Yilin Shen, Xin Wang, and William Wang. Enhancing the robustness of prior network in outt-of-distrbbuiin detection. CoRR, abs/1811.07300 2018. URL
00ss/111..0300
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised
feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudk (eds.), Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort I Luddeddaee, FI., USA, 11-13 Apr 2011. PMLR. Utk.hhttp://proceeddi ngs mm w.w.rppresssse
paress/vl55 http:///w//ottsssslslle.ess html
Filipe Condessa, Jelena Kovacevic, and José M. Bioucas-Dias. Performance measurss for classiffiaa tion systems with rejection. CoRR, abs/1500.002633, 2015. URL http::t:://xxxi..rr///b/./
1554402763
C. De Stefano, C. Sansone, and M. Vento. To reject not reject: That is the question-an answer in case of neural classifiers. Trans. Sys. Man Cyber Part C, 30(1):84-94, February 2000. ISSN 1094- 6977. doi: 1111009532.82755777
dx doi http:p/://.k.....ogg/000..1...1.
5326. 827457
L.C. Freeman, Elementary applied statistics: for students in behavioral science. Wiley, 1965. URL https://books. google. es www.oooosssdddaaaaaa.
Yarin Gal. Uncertainty in deep learning. PhD thesis, PhD thesis, University of Cambridge, 2016.
Yonatan Geifman and Ran El- Yaniv.... Selective classification for deep neural networks. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4878 4887. Curran Associates, Inc., 2017. URL http::/papers. wnw.snnipss...................s. 7073- htt-=:acccceeccaaaaaaaaaaoeoreee
Yonatan Geifman and Ran El-yaniv. Selectivenet: A deep neural network with an integrated re- ject option. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 2151-2159, 2019. URL http: //proceedings.. ml www.:p/wesss1.1... frneifmmanl 9a. html
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neurul information processing systems. pp. 5574-5584, 2017.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior net- works. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7047-7058. Curran Associates, Inc., 2018. URL http:://ppppeese wwwpaappprrsn.s.cccooaeee nips nww..cc/pap.........ccccpppppp
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiel Chen. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and seg- mentation. CoRR, abs/1801 04381, 2018. Utl. htwp://cce//arrc...oo/////////// 043811
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations, 2015.

---

Under review as a conference paper at ICLR 2020
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning. Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL htw.lhtpss:/c/
ac web web. wtt.://wwww..o.....aaaooooooo/oa
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00567,, 2015. URL http://arxe wtwp://.rr./aab/12....00
abs
A APPENDIX A
For the sake of reproducibility. this Appendix details the architectures used for training the black- box systems. Figure 9 describes the model used for training the black-box models in the two use
Figure 9: Models used train the black-boxes: a) the one used for text and b) describes that used for images
Proto
d-10
Prote
d+2
Sottrax
d-10
Saltnux
d+2
Nomeatztion
Banch
d-10
Dense
d+2
Denie
d-10
representation
Latent test
d:300
representation
Lateni test
Medi aiberv2
(maggene))
The
movie
great
Input image
d-3232d
Trained
Text
cases. As stated before, the only purpose of this model is to obtain black-box classifier for given source domain. The goal, in this case, is not to obtain the best classifier but to obtain a model which easy to train and offers good performance.
The main difference between the model for NLP and Image classification comes from the embedding component. In the case of NLP, opted for representing sentence the average value of the embedding of each word using pre- trained word2vec embeddings. In the case of images, we trained MobileNET v2 model (Sandler et al. 2018), initialized with imagenet weights, using input the STL-10 images, resized to 32x32x3 to accommodate them to the CIFAR10 10 dataset
B APPENDIX B
Table shows detail of the numerical results obtained during the experiments for the four com- binations tested. The first column, black-box source acc, describes the accuracy obtained for the source dataset after training the original classifier. Next, column black-box target acc describes the accuracy obtained when applying the black-box to the target dataset. The rest of the columns show the non-rejected accuracy and the classification and rejection quality after rejecting 10, 20 an 30% of the points, using the proposed predictive entropy as rejector.
7 we tried other embeddings such as ELMO, and Seq2seq for text, or VGG- 16 (imoonyan & Zisserman 2015) and ResNET50 (Szegedy al. 2015 for images, but stick to word2vec and MobileNet due to limitations on the computing resources.
10

---

Under review as a conference paper at ICLR 2020
Table 1: Accuracy obtained by training an standalone classifier, applying the API and the proposed
wrapper for each domain
Non-reject. acc.
Class. quality
Reject. quality
BB source acc.
BB target acc.
(10/20/30%
(10/20/30%
(10/20/30%
81.38 0.72%
18.8210.9114
0.66
Apply Velp BB to SST-2
89.18+0.088%
77.13+0.52%
85.83=0.88%
79.66=1.15%
4.33=0.44
90.08 0.94%
78.46 1.31%
0.39
86,34 0.18%
83.27 0.88%
5.981
Apply SST-2 BB to Yelp
83.306++0.18
82.106==0.88%
89 0.38%%
80.95-0.38%%
4.10110.7
92.08 0.33%
76.77-0.46%
3.21+0.10
35.04-0.435%
90.67=0.88%
10.7::1.65
Apply Electronics BB to Music
86.39=00222%
90.38:0.1333
96.45 0.35%
83.93.0.67%
4.82.10.35
97.26-0.31%
75.77=0.544%
3.25+00144
01.79-0.31% %
90.27-0.54%
19.19+2.9
Apply Music BB to Electronics
0.:00.
83.06=00.0%
94.90 0.85%
86.22
6.60
96.00=0.83%
79.91=0.98%
4.02=0025
42.53-0.04%%
46.22=00.0%%
2.56=0.05
Apply STL-10 to CIFARIO
53.53:10.22%
39.29-00088%
52.18 0..066
2.62+0.02
47.78=0.05%
56.55=0.08%
2.25=0.01
C APPENDIX C
This Appendix shows detailed results on the image case. Although the resulting quality obtained for the rejection mechanism the case images not as large as in texts, when comparing to predictive entropy of the original classifier, we observe that the proposed measure is still excellent for detecting out of sample images. The main difference between STL-110 and CIFARTO on one of the classes. Where in STL-10 class 6 held monkeys, in CIFARIO it corresponds to frogs. As one can expect, the black-box model trained with STL-10 will struggle on detecting frogs.
variation
Figure 10: Distribution of the predicted entropies for two of the CIFAR10 classes.
300
600
200
400
100
200
0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Figure 11: Entropies for frogs
Figure 12: Entropies for trucks
we can see distributions of the images that belong to the frogs class and images that belong to the trucks class. For the frogs class, that the values of uncertainty concentrated in the higher band of the diagram, whereas the case of trucks, we find many with lower uncertainty. This detail shows that the metric assigns significant uncertainty to out-of-sample class points.
In figure 11 and 12
11


Reviews:
[{"review_id": "rkekhHcaYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for learning a “wrapper” model which endows a multiclass predictor with an estimate of model uncertainty. The base model is treated as a black box which emits a categorical distribution, while the wrapper model estimates the parameters of a Dirichlet distribution. The wrapper is trained against silver labels from the base model, and the sampled predictive entropy is used to threshold predictions so as to withhold a decision on uncertain examples.\n\nI believe this paper should not be accepted, as the main contribution is not particularly novel and some experimental details (Section 3) are not well-motivated. \n\nIn particular, the idea of learning a Dirichlet prior is very similar to that proposed in (Malinin and Gales, NeurIPS 2018). The main contribution in this work seems to be the application to an existing black-box model, but this seems like a straightforward application of knowledge distillation, another well-established technique (e.g. Hinton et al. 2015, though oddly, this paper doesn’t mention this).\n\nThe details of the modeling set-up (section 2.2 and 3) are also not entirely clear. It seems from the preceding discussion that the main goal of the wrapper model is to estimate \\beta, but it is not clear how the sampling procedure (2.2) allows for this. It seems that the value of the “sampled predictive entropy” (no equation number, but see end of Section 3) is just that of the mean predicted distribution, and it’s not clear why this should be different (assuming the wrapper model converges) than the predictive entropy of the base model.\n\nThis paper could be improved by a more thorough comparison to the literature, and by a clearer motivation for the training procedure used. Additionally, it would help to present a summary table of results, and to frame the metrics in terms of standard classification metrics such as precision and recall where applicable.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgWFpePqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes to train a wrapper to assess the confidence of a black box classifier decision on new samples. The resulting uncertainty prediction is used to reject decision with low confidence, such that the accuracy of the prediction on retained samples remains high. The idea, although a bit incremental, is potentially useful to practitioners, as argued in the introduction, and some empirical result tend to suggest that the method can be useful. However, I am not convinced the approach and its implementation are appropriate. \nMain comments:\nWriting: the model seems strongly based on references such as Kendall & Gal (2017), however lack of introduction of notations and explanations prevent the paper to be self contained. For example: \n(1)\tHere is a list of quantities for which I could not find a definition: y_m, \\omega, W. \n(2)\t\\hat{y} appears with several indexing styles (up to three indices), sometime bold and sometimes not: the meaning of the indexing (which I could not find) is difficult to infer from the text\nObjective:\nI was unable to understand the rational behind the objective to be minimized, introduced in page 4. This is introduced as a cross-entropy loss, however, taking the expression of the Dirichlet distribution, it is not obvious to me how to reach this very simple expression. Is it possible that the authors simply mimic the expression of Kendall and Gal (2017, eq. (5)), that was designed for the Gaussian case, for which the cross-entropy expression is correct?\nAs a consequence, I do not see how this objective is supposed to learn properly the correct beta parameter.\n\nResults:\nFigures 4-8 shows convincing evidence that the procedure improves with respect to a baseline that consist (as far as I understood) in ranking decision based on the entropy of the output probability vector of the classifier. However, given that I am unsure about what the proposed optimization does, it remains unclear to me whether these results reflect a true achievement. For example, one can argue that the chosen baseline is unlikely to be a good estimate of the entropy of the decision due to the fluctuations of the output probability for the unlikely classes. Those low probability values are the classical source of variance and bias in entropy estimates, and a classifier is not designed to get these low probabilities right (as they are low anyways). As a consequence, an already better baseline might be achieved for a given reference class by cropping the probability vector to keep only classes that non-negligible probability over the training set (here they can be many alternative approaches to test). \nA trivial explanation for the proposed approached to work better than the currently chosen baseline is that the noise introduced by the sampling from the Dirichlet distribution leads to larger probabilities for the cases where the probability given by the classifier is small, which would reduce the variance of the entropy estimator based on the formula of page 5 (top). Overall, extensively checking many simpler baselines (which do not require training!) is a first step to see if the achieved result is not easy to get.   \nMinor comments:\n(1)\tPlease check for typos. \n(2)\tPlease avoid remove the multiple parenthesis for successive citations (use a single “\\citep”).\n(3)\tIn Fig. 1, subtitles are inconsistent.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byev-9Zwcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by real-world challenges in applying pre-trained models, the authors propose a model for selective prediction (prediction with an option for abstention) that wraps an existing black-box classification model. The resulting model output is a Dirichlet distribution with mean equal to the categorical distribution produced by the black-box and concentration parameter specified by a separate auxiliary model. This additional model is trained to minimize negative log-likelihood of observations under categorical distributions sampled from the aforementioned Dirichlet along with an L1 regularization term on the concentration parameter. To infer the model’s level of uncertainty, the authors propose computing the entropy of the average of sampled categorical distributions.\n\nThe authors evaluated this model on several pairs of sentiment-analysis NLP tasks and one pair of image datasets where a base model is trained on the source dataset, and the auxiliary model is trained on the second, target dataset. Using metrics proposed in (Condessa et al. 2015), the results show positive results at nearly all thresholds compared to a simple entropy baseline.\n\nThe paper addresses an important practical challenge in machine learning, but a confusing problem-framing and lack of robust baselines make me skeptical that it is suitable for publication at ICLR 2020.\n\nA primary concern with this work is its framing of the problem as one of measuring aleatoric (irreducible) uncertainty, but the motivation in transfer learning and interdependence in production ML systems requires models that can characterize epistemic (reducible) uncertainty. A black box model that yields distributions over classes expresses aleatoric uncertainty via that distribution, and uncertainty due to a shifted data distribution is epistemic as additional data from the new domain would reduce it.\n\nMore problematic is the study’s lack of robust baselines. The authors only present the predictive entropy baseline, but numerous methods exist for out-of-distribution detection and selective classification. Though the specific case of selective classification from a blackbox base model is perhaps more niche, other methods from related problems can either be adapted accordingly or used as upper / lower bounds on what we can expect for this problem. \n\nSome simple baselines I would expect to see include:\n* Training a new classification model entirely on the new domain.\n* Training an auxiliary classifier to predict if the base model will be correct (similar to SelectiveNet but without a shared network body). Ideally this model should also have access to the base-model’s prediction as input.\n* “Confidence score” (i.e. the probability assigned to the base-model’s predicted class) -- this is a common baseline for OOD detection.\n\nAdditional questions / concerns:\n* Do I understand correctly that when drawing samples for the entropy calculation, $E[\\hat{y}]$ will equal the black-box model’s prediction in the limit of sample size? If so, this looks like an inefficient and complicated proxy for measuring the concentration of categorical entropies produced by the Dirichlet.\n* Since the paper is focused on scenarios where one is taking advantage of a pre-trained model, one might wonder if the wrapping scheme is indeed less expensive than a new model trained from scratch on the new domain (e.g. with comparable capacity to the proposed wrapping model). Authors should include experiments / baselines to assess this.\n* In section 2, the paper asserts that assuming access to logits breaks the blackbox assumption, but these are computable from softmax values (up to constant factors).\n* Is the beta regularization term is theoretically required to prevent unbounded growth or is it simply an empirically practical necessity?\n* In the image transfer experiments, STL-10 has relatively very few labeled examples (hundreds per class), so why was this only used as the source domain? Realistic transfer learning generally entails an expensive model trained on a source domain with plentiful data transferred to a target domain with scarce data.\n* Authors should mention that STL-10 images represent a distribution shift from CIFAR-10 as the two were not generated identically. The paper currently only highlights differences in dataset size.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgRMcbStS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to use a dirichlet prior over the multinomial distribution outputted by blackbox DL models, to quantify uncertainty in predictions. The main contribution is to learn the parameters of the prior and use it as a wrapper over the black box, to adjudicate whether to retain or reject a particular prediction made by the model. The dirichlet parameters are learnt in conjunction with the model parameters as a fine tuning step in transfer learning tasks. Experiments on NLP and CV domains and multiple datasets demonstrate the efficacy of the method. \n\nThe paper is well written, and easy to understand. The main motivation of the paper seems to be to learn what samples to drop, but the authors do not address what can be done about the dropped samples (i.e what happens if we end up having to drop 80% of the samples?) . Adding the dirichlet prior to quantify uncertainty has also been studied in the context of VAEs before, (and LDA back in the day) so conceptually there's limited novelty. Nonetheless, the method seems to provide impressive results on multiple datasets, and I think this is interesting enough to warrant an accept. \n\nMINOR:\n1. Figure 1: the values of \\beta is the same in all subfigures\n2. Sec 2.2 line 1: SImilarly --> Similar\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes adding a Dirichlet distribution as a wrapper on top of a black box classifier in order to better capture uncertainty in the predictions.  This paper received four reviews in total with scores (1,1,1,6).  The reviewer who gave the weak accept found the paper well written, easy to follow and intuitive.  The other reviewers, however, were primarily concerned about the empirical evaluation of the method.  They found the baselines too weak and weren't convinced that the method would work well in practice.  The reviewers also cited a lack of comparison to existing literature for their scores.  One reviewer noted that while the method addresses aleatoric uncertainty, it doesn't provide any mechanism for epistemic uncertainty, which would be necessary for the applications motivating the work.  

The authors did not provide a response and thus there was no discussion. 
Decision: Reject