Dirichlet Wrapper to Quantify Classification Uncertainty in Black-Box Systems

1 INTRODUCTION . The popularity of machine learning is giving birth to new business models based on the productization and service of these models . In the market there are many application programming interfaces ( APIs ) serving predictions in object recognition for images ( Vision AI1 ) , language detection or sentiment analysis in natural language processing ( Cloud Natural Language API2 ) , to mention just a few . As this Machine Learning-as-a-Service model starts to grow , it becomes easier to find these APIs as an integral component of more complex products . The use of pre-trained models gives rise to two different problems . First , we do not know how these models are going to operate in our intended application domain . In order to address this issue , there is a vast literature on transfer learning that can be applied . However , when using third-party proprietary software or APIs , we may not have access to the internals or the possibility of finetuning the model to our domain . If we are to use the model as it is , one must at least understand when the model is going to work and when it is not , to have some confidence metric that tells about the expected performance of the methods when applied to our problem . However , this information is not always provided , especially in deep learning models . This effect can be worsened when these components are just one of the many different parts of a data product . This complexity leads us to the second problem : when different models might interact in complex pipelines , the construction of the appropriate confidence measures can be a challenging task . 1https : //cloud.google.com/vision/ 2https : //cloud.google.com/natural-language/ In order to solve the previous issues , in this article , we propose a deep learning wrapper algorithm that equips any black-box model with uncertainty prediction . Here a wrapper is understood as a machine learning model that takes any other model and operates without accessing its internals . Because it does not have access to the internal states , parameters , or architecture of the model it is wrapping , the wrapper is model agnostic and can be used on top of any other algorithm as long as it satisfies some desideratum . In this article , we only require the black-box model to produce as output a distribution over the classes , a soft requirement as any model with a soft-max layer satisfies it . More specifically , the proposed wrapper uses a deep learning model and introduces a Dirichlet layer as the fusion layer with the black-box . This Dirichlet layer yields a distribution on top of the multinomial output parameters . By sampling from this Dirichlet distribution , the wrapper enables the estimation of aleatoric uncertainty . Uncertainty has been an important topic in machine learning for many years ( Koller & Friedman , 2009 ) . With the emergence of deep learning , the reinterpretation of some existing mechanisms such as dropout , or the proposal of stochastic mechanisms such as Montecarlo approaches , has broadened the use of these techniques for accounting for uncertainty in deep models ( Gal , 2016 ) . Uncertainty can be categorized into epistemic and aleatoric uncertainty . While the first accounts for the uncertainty that is associated to model parameters , the second corresponds to the uncertainty inherently present in the data3 . Uncertainty plays a key role when reporting a decision because it accounts for the reliability of the prediction and can help to show the limitations of the applicability of a machine learning model . In this respect , we advocate for the use of selective prediction ( aka rejection techniques ) when the uncertainty metric is large in order to avoid potential harm or avoid risks . Selective prediction is a set of techniques based on abstaining from deciding according to some metric threshold . As previously commented , uncertainty is a good candidate for a rejection metric . In literature , we find examples of different rejection functions ( Geifman & El-Yaniv , 2019 ) ( De Stefano et al. , 2000 ) and some of them use uncertainty measures ( Geifman & El-Yaniv , 2017 ) for rejection . In the proposed scenario , where we are trying to characterize the uncertainty of a black-box nonmutable model , many of the state-of-the-art techniques are not applicable . For example , some rejectors have to be trained together with the classifier and need access to the internals of the model . In the same line , current models for uncertainty in deep learning need to have access to its internal states ( Gal , 2016 ) . The contributions of this article can be summarized as follows : • We propose a wrapper algorithm that equips any other classification model that outputs a distribution over predicted classes with Bayesian treatment without having knowledge or access to its internals . • We use the wrapper to empirically estimate aleatoric uncertainty and show that the computed uncertainty can identify prone to err samples . • Finally , we show that the computed uncertainty can be used by rejection techniques to increase the performance and robustness of the original black-box model in the target domain . We show improvements in transfer problems in natural language processing and computer vision problems . In section 2 , we introduce the method proposed for building an uncertainty wrapper around a blackbox model . In section 3 , we describe how to obtain an uncertainty score from the wrapper output . Section 4 introduces the concept of rejection and rejection performance metrics . In section 5 , we showcase the proposed method in four different scenarios for sentiment analysis in natural language processing and one for computer vision . The results obtained corroborate the importance of the rejection method and show the success of the proposed methodology . Finally , section 6 concludes the article . 3Observe that in this application only aleatoric uncertainty matters since we are dealing with pre-trained , non-mutable models . In this particular scenario , aleatoric uncertainty also serves as a measure of the fitness of the model to the data . 2 BUILDING AN UNCERTAINTY WRAPPER . Our goal is to build a wrapper algorithm that takes another black-box model and operates on top of it . As such , there are several constraints to observe . First , we need to exclusively operate on the inputs and outputs of the black-box classifier . We are not allowed to use any intermediate or internal value of the black-box model as we need to be agnostic to it . Second , the input of the wrapper has to be compatible with the original distribution over the output classes . In the literature , other proposals suggest a deep learning model for estimating uncertainty . The problem with those approaches , like in ( Kendall & Gal , 2017 ) where they use independent Gaussian random variables to model the pre-activation value of the logits , is that they do not conform to the constraints in our setting . First , having access to the logits before the softmax breaks the black-box assumption ; and , second , independent Gaussian distributions impose unnecessary assumptions and need of additional normalization steps . A more natural approach is to consider the output distribution coming from a Dirichlet probability distribution function . 2.1 DIRICHLET CONCENTRATION REPARAMETERIZATION . As commented , given a data set D composed of pairs ( xi , yi ) , i = 1 . . . N , with yi ∈ RC , being C the number of different classes , the wrapper output is assumed to come from a Dirichlet probability density function : p ( yw|X , w∗ ) ∼ Dir ( α ) , ( 1 ) wherew∗ are the parameters of the wrapper . We propose to use a decomposition of the concentration parameter in two terms to relate the output of the black-box classifier , ym , with the concentration parameter , α , in the Dirichlet distribution of the wrapper . To that effect , we recall some basic statistics of the Dirichlet distribution . Given a Dirichlet random variable x ∈ RC with concentration parameter α ∈ RC , the expected value of the distribution is defined as E ( x ) = α/ C∑ i=1 αi . Observe that the expected value has the same properties as a probability distribution and that the output of the black-box ym ∈ RC is already a probability distribution . In this sense , we could directly use the output of the black box as the concentration parameter . However , each term of the concentration parameter is not necessarily constrained to the interval [ 0 , 1 ] . Let us introduce a new scalar parameter , β ∈ R that will model this difference , such that α = βym . Observe that the value of β does not change the expected value of the output of the wrapper and coincides with the output of the black-box model , i.e . E ( yw ) = ym . This decomposition has a simple interpretation : While the output of the black-box classifier stands for the mean , parameter β accounts for the spread of the distribution . The same or similar decomposition can be found in other works in a different context ( Malinin & Gales , 2018 ) ( Chen et al. , 2018 ) 4 . An example of the effect of varying this parameter in a three dimensional Dirichlet distribution is shown in Figures ( 1a ) to ( 1c ) . Observe that the higher the value of β , the more pointy the distribution is . 4It is worth noting that in the context of those works , there is a degradation in performance when using Dirichlet . This does not happen in our case since the black-box model is non-mutable . This decoupling allows to effectively isolate the contribution of the black-box and the contribution that remains to be computed , i.e . the value of parameter β . Figure 2 shows the integration of the wrapper ( in light orange colour ) with the black-box classifier ( in light blue colour ) . Observe that the wrapper consists of two blocks : the Dirichlet reparameterization layer of the wrapper that decouples the influence of the black-box model from the rest ( see the dashed line ) , and a deep learning architecture which aims to compute the scalar value of β5 . 2.2 INFERENCE IN THE DIRICHLET SETTING . SImilarly to ( Kendall & Gal , 2017 ) , we approximate the expected value of the classification probabilities using Monte Carlo sampling from the learned Dirichlet distribution for each sample , ŷ. , i ∼ Dir ( αi ) as E [ ŷi ] = 1M ∑M m=1 ŷm , i . This distribution is used to define the loss function for our learning stage . Given a set of N training samples , we will use a regularized version of the cross-entropy loss function as follows : L ( W ) = − 1 N N∑ i=1 1 C C∑ c=1 yi , c logE [ ŷi ] c + λ‖β‖2 = − 1 N 1 C N∑ i=1 C∑ c=1 yi , c log ( 1 M M∑ m=1 ŷm , i , c ) + λ‖β‖2 . Observe that we introduce the norm of the β value in the minimization function . This term is required since the unregularized cross-entropy forces the value of β to grow unbounded . By adding this term , we control its growth and govern the trade-off with a scalarization parameter λ . 3 OBTAINING AN UNCERTAINTY SCORE FROM THE WRAPPER . The described Dirichlet layer effectively allows studying the variability of the parameters of the black-box output . This variability can be used to approximate a value for the heteroscedastic aleatoric uncertainty . In this work , we use Monte Carlo simulation sampling from the obtained Dirichlet function in order to characterize the uncertainty ( Gal , 2016 ) . Standard techniques for measuring uncertainty includes variation ratios or predictive entropy . Variation ratios measures the variability of the predictions obtained from the sampling ( Freeman , 1965 ) by computing the fraction of samples with the correct output . Alternatively , predictive entropy considers the average amount of information contained in the predictive distribution . Those results with low entropy values correspond to confident predictions , whereas high entropy leads to large uncertainty . Since the output of the black-box model ym already describes a probability distribution , one could compute its predictive entropy and obtain a measure of its uncertainty with H = − ∑ c y m c log y m c 5The architecture used in this figure corresponds to the one used in the experimental section . However , as the wrapper allows us to model the variability of the parameters of the black-box output distribution , we can compute a predictive entropy that takes into account the variability of the predicted value . In this case , the sampled predictive entropy is defined as H = − ∑ c E [ ŷ ] c logE [ ŷ ] c. As we show in the experimental section , this latter approach captures better the uncertainty compared to the predictive entropy of the original model .

Summary: This paper presents a method for learning a “wrapper” model which endows a multiclass predictor with an estimate of model uncertainty. The base model is treated as a black box which emits a categorical distribution, while the wrapper model estimates the parameters of a Dirichlet distribution. The wrapper is trained against silver labels from the base model, and the sampled predictive entropy is used to threshold predictions so as to withhold a decision on uncertain examples.