Dirichlet Wrapper to Quantify Classification Uncertainty in Black-Box Systems

1 INTRODUCTION . The popularity of machine learning is giving birth to new business models based on the productization and service of these models . In the market there are many application programming interfaces ( APIs ) serving predictions in object recognition for images ( Vision AI1 ) , language detection or sentiment analysis in natural language processing ( Cloud Natural Language API2 ) , to mention just a few . As this Machine Learning-as-a-Service model starts to grow , it becomes easier to find these APIs as an integral component of more complex products . The use of pre-trained models gives rise to two different problems . First , we do not know how these models are going to operate in our intended application domain . In order to address this issue , there is a vast literature on transfer learning that can be applied . However , when using third-party proprietary software or APIs , we may not have access to the internals or the possibility of finetuning the model to our domain . If we are to use the model as it is , one must at least understand when the model is going to work and when it is not , to have some confidence metric that tells about the expected performance of the methods when applied to our problem . However , this information is not always provided , especially in deep learning models . This effect can be worsened when these components are just one of the many different parts of a data product . This complexity leads us to the second problem : when different models might interact in complex pipelines , the construction of the appropriate confidence measures can be a challenging task . 1https : //cloud.google.com/vision/ 2https : //cloud.google.com/natural-language/ In order to solve the previous issues , in this article , we propose a deep learning wrapper algorithm that equips any black-box model with uncertainty prediction . Here a wrapper is understood as a machine learning model that takes any other model and operates without accessing its internals . Because it does not have access to the internal states , parameters , or architecture of the model it is wrapping , the wrapper is model agnostic and can be used on top of any other algorithm as long as it satisfies some desideratum . In this article , we only require the black-box model to produce as output a distribution over the classes , a soft requirement as any model with a soft-max layer satisfies it . More specifically , the proposed wrapper uses a deep learning model and introduces a Dirichlet layer as the fusion layer with the black-box . This Dirichlet layer yields a distribution on top of the multinomial output parameters . By sampling from this Dirichlet distribution , the wrapper enables the estimation of aleatoric uncertainty . Uncertainty has been an important topic in machine learning for many years ( Koller & Friedman , 2009 ) . With the emergence of deep learning , the reinterpretation of some existing mechanisms such as dropout , or the proposal of stochastic mechanisms such as Montecarlo approaches , has broadened the use of these techniques for accounting for uncertainty in deep models ( Gal , 2016 ) . Uncertainty can be categorized into epistemic and aleatoric uncertainty . While the first accounts for the uncertainty that is associated to model parameters , the second corresponds to the uncertainty inherently present in the data3 . Uncertainty plays a key role when reporting a decision because it accounts for the reliability of the prediction and can help to show the limitations of the applicability of a machine learning model . In this respect , we advocate for the use of selective prediction ( aka rejection techniques ) when the uncertainty metric is large in order to avoid potential harm or avoid risks . Selective prediction is a set of techniques based on abstaining from deciding according to some metric threshold . As previously commented , uncertainty is a good candidate for a rejection metric . In literature , we find examples of different rejection functions ( Geifman & El-Yaniv , 2019 ) ( De Stefano et al. , 2000 ) and some of them use uncertainty measures ( Geifman & El-Yaniv , 2017 ) for rejection . In the proposed scenario , where we are trying to characterize the uncertainty of a black-box nonmutable model , many of the state-of-the-art techniques are not applicable . For example , some rejectors have to be trained together with the classifier and need access to the internals of the model . In the same line , current models for uncertainty in deep learning need to have access to its internal states ( Gal , 2016 ) . The contributions of this article can be summarized as follows : • We propose a wrapper algorithm that equips any other classification model that outputs a distribution over predicted classes with Bayesian treatment without having knowledge or access to its internals . • We use the wrapper to empirically estimate aleatoric uncertainty and show that the computed uncertainty can identify prone to err samples . • Finally , we show that the computed uncertainty can be used by rejection techniques to increase the performance and robustness of the original black-box model in the target domain . We show improvements in transfer problems in natural language processing and computer vision problems . In section 2 , we introduce the method proposed for building an uncertainty wrapper around a blackbox model . In section 3 , we describe how to obtain an uncertainty score from the wrapper output . Section 4 introduces the concept of rejection and rejection performance metrics . In section 5 , we showcase the proposed method in four different scenarios for sentiment analysis in natural language processing and one for computer vision . The results obtained corroborate the importance of the rejection method and show the success of the proposed methodology . Finally , section 6 concludes the article . 3Observe that in this application only aleatoric uncertainty matters since we are dealing with pre-trained , non-mutable models . In this particular scenario , aleatoric uncertainty also serves as a measure of the fitness of the model to the data . 2 BUILDING AN UNCERTAINTY WRAPPER . Our goal is to build a wrapper algorithm that takes another black-box model and operates on top of it . As such , there are several constraints to observe . First , we need to exclusively operate on the inputs and outputs of the black-box classifier . We are not allowed to use any intermediate or internal value of the black-box model as we need to be agnostic to it . Second , the input of the wrapper has to be compatible with the original distribution over the output classes . In the literature , other proposals suggest a deep learning model for estimating uncertainty . The problem with those approaches , like in ( Kendall & Gal , 2017 ) where they use independent Gaussian random variables to model the pre-activation value of the logits , is that they do not conform to the constraints in our setting . First , having access to the logits before the softmax breaks the black-box assumption ; and , second , independent Gaussian distributions impose unnecessary assumptions and need of additional normalization steps . A more natural approach is to consider the output distribution coming from a Dirichlet probability distribution function . 2.1 DIRICHLET CONCENTRATION REPARAMETERIZATION . As commented , given a data set D composed of pairs ( xi , yi ) , i = 1 . . . N , with yi ∈ RC , being C the number of different classes , the wrapper output is assumed to come from a Dirichlet probability density function : p ( yw|X , w∗ ) ∼ Dir ( α ) , ( 1 ) wherew∗ are the parameters of the wrapper . We propose to use a decomposition of the concentration parameter in two terms to relate the output of the black-box classifier , ym , with the concentration parameter , α , in the Dirichlet distribution of the wrapper . To that effect , we recall some basic statistics of the Dirichlet distribution . Given a Dirichlet random variable x ∈ RC with concentration parameter α ∈ RC , the expected value of the distribution is defined as E ( x ) = α/ C∑ i=1 αi . Observe that the expected value has the same properties as a probability distribution and that the output of the black-box ym ∈ RC is already a probability distribution . In this sense , we could directly use the output of the black box as the concentration parameter . However , each term of the concentration parameter is not necessarily constrained to the interval [ 0 , 1 ] . Let us introduce a new scalar parameter , β ∈ R that will model this difference , such that α = βym . Observe that the value of β does not change the expected value of the output of the wrapper and coincides with the output of the black-box model , i.e . E ( yw ) = ym . This decomposition has a simple interpretation : While the output of the black-box classifier stands for the mean , parameter β accounts for the spread of the distribution . The same or similar decomposition can be found in other works in a different context ( Malinin & Gales , 2018 ) ( Chen et al. , 2018 ) 4 . An example of the effect of varying this parameter in a three dimensional Dirichlet distribution is shown in Figures ( 1a ) to ( 1c ) . Observe that the higher the value of β , the more pointy the distribution is . 4It is worth noting that in the context of those works , there is a degradation in performance when using Dirichlet . This does not happen in our case since the black-box model is non-mutable . This decoupling allows to effectively isolate the contribution of the black-box and the contribution that remains to be computed , i.e . the value of parameter β . Figure 2 shows the integration of the wrapper ( in light orange colour ) with the black-box classifier ( in light blue colour ) . Observe that the wrapper consists of two blocks : the Dirichlet reparameterization layer of the wrapper that decouples the influence of the black-box model from the rest ( see the dashed line ) , and a deep learning architecture which aims to compute the scalar value of β5 . 2.2 INFERENCE IN THE DIRICHLET SETTING . SImilarly to ( Kendall & Gal , 2017 ) , we approximate the expected value of the classification probabilities using Monte Carlo sampling from the learned Dirichlet distribution for each sample , ŷ. , i ∼ Dir ( αi ) as E [ ŷi ] = 1M ∑M m=1 ŷm , i . This distribution is used to define the loss function for our learning stage . Given a set of N training samples , we will use a regularized version of the cross-entropy loss function as follows : L ( W ) = − 1 N N∑ i=1 1 C C∑ c=1 yi , c logE [ ŷi ] c + λ‖β‖2 = − 1 N 1 C N∑ i=1 C∑ c=1 yi , c log ( 1 M M∑ m=1 ŷm , i , c ) + λ‖β‖2 . Observe that we introduce the norm of the β value in the minimization function . This term is required since the unregularized cross-entropy forces the value of β to grow unbounded . By adding this term , we control its growth and govern the trade-off with a scalarization parameter λ . 3 OBTAINING AN UNCERTAINTY SCORE FROM THE WRAPPER . The described Dirichlet layer effectively allows studying the variability of the parameters of the black-box output . This variability can be used to approximate a value for the heteroscedastic aleatoric uncertainty . In this work , we use Monte Carlo simulation sampling from the obtained Dirichlet function in order to characterize the uncertainty ( Gal , 2016 ) . Standard techniques for measuring uncertainty includes variation ratios or predictive entropy . Variation ratios measures the variability of the predictions obtained from the sampling ( Freeman , 1965 ) by computing the fraction of samples with the correct output . Alternatively , predictive entropy considers the average amount of information contained in the predictive distribution . Those results with low entropy values correspond to confident predictions , whereas high entropy leads to large uncertainty . Since the output of the black-box model ym already describes a probability distribution , one could compute its predictive entropy and obtain a measure of its uncertainty with H = − ∑ c y m c log y m c 5The architecture used in this figure corresponds to the one used in the experimental section . However , as the wrapper allows us to model the variability of the parameters of the black-box output distribution , we can compute a predictive entropy that takes into account the variability of the predicted value . In this case , the sampled predictive entropy is defined as H = − ∑ c E [ ŷ ] c logE [ ŷ ] c. As we show in the experimental section , this latter approach captures better the uncertainty compared to the predictive entropy of the original model .

Summary: Motivated by real-world challenges in applying pre-trained models, the authors propose a model for selective prediction (prediction with an option for abstention) that wraps an existing black-box classification model. The resulting model output is a Dirichlet distribution with mean equal to the categorical distribution produced by the black-box and concentration parameter specified by a separate auxiliary model. This additional model is trained to minimize negative log-likelihood of observations under categorical distributions sampled from the aforementioned Dirichlet along with an L1 regularization term on the concentration parameter. To infer the model’s level of uncertainty, the authors propose computing the entropy of the average of sampled categorical distributions.