Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations

1 Introduction . Increasing concentration of big data and computing resources has resulted in widespread adoption of machine learning as a service ( MLaaS ) . The best-performing NLP , speech , image and video recognition tools are now provided as network services . MLaaS comes with few accuracy specifications or service level agreements , perhaps only leaderboard numbers from benchmarks that may not be closely related to most clients ’ deployment data distributions . The client , therefore , finds it difficult to choose the best provider without extensive pilot trials [ 1 ] . Different clients may need to deploy the service on very different data distributions , with possibly widely different accuracy . In such circumstances , we propose that a service provider , or a service standardization agency , publish the accuracy of the classifier , not as one or few aggregate numbers , but as a surface defined on a space of input instance attributes that capture the variability of consumer expectations . Indoor/outdoor , day/night , urban/rural may be attributes of input images for visual object recognition tasks . Speaker age , gender , ethnicity/accent may be attributes of input audio for speech recognition tasks . We call a combination of attributes in their Cartesian space an arm ( borrowing from bandit terminology ) 2 . The labeled instances used by the service provider may not represent or cover well the space of attributes of interest to subscribers . Labeled data may be proprietary and inaccessible to prospective consumers and standardization agencies . Whoever estimates the accuracy surface , therefore , needs to ∗vihari @ cse.iitb.ac.in 2Figure 1 shows an example of diverse accuracy over arms . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . actively select instances from an unlabeled pool for labeling , presumably within a restricted budget , to adequately cover the attribute space . Several recent studies have highlighted the variability in accuracy across data sub-populations [ 2 , 3 ] , specifically in the context of fairness [ 4 , 5 , 6 ] , and also proposed active estimation techniques of sub-population accuracy [ 7 , 8 ] . We solve a more general problem where the space of arms ( subpopulation ) defined by the Cartesian space of attributes grows combinatorially . This inevitably leads to extreme sparsity of labeled instances for many arms . A central challenge is how to smooth the estimate across related arms while faithfully representing the uncertainty for active exploration . We present Attributed Accuracy Assay ( AAA ) — a practical system that estimates accuracy , together with the uncertainty of the estimate , as a function of the attribute space . AAA uses these estimates to drive the sampling policy for each attribute combination . Gaussian Process ( GP ) regression is a natural choice to obtain smooth probabilistic accuracy estimates over arm attributes . However , a straightforward GP model fails to address the challenge of heteroscedasticity that we face with uneven and sparse supervision across arms . We model arm-specific service accuracy as drawn from a Beta density that is characterized by mean and scale parameters , which are sampled from two GPs that are informed by suitable trained kernels over the attribute space . We propose two further enhancements to the training of this model . First , we recognize an over-smoothing problem with GP ’ s estimation of the Beta scale parameters , and propose a Dirichlet likelihood to supervise the relative values of scale across arms . Second , we recognize that arms with very low support interfere with learning the kernel parameters of the GPs . We mitigate this by pooling observations across related arms . With these fixes , AAA achieves the best estimation performance among competitive alternatives . Another practical challenge in our setting is that some attributes of instances are not known exactly . For example , attributes , such as camera shutter speed or speaker gender , may be explicitly provided as meta information attached with instances . But other attributes , such as indoor/outdoor , or speaker age , may have to be estimated noisily via another ( attribute ) classifier , because accurate human-based acquisition of attributes would be burdensome . AAA also tackles uncertain attribute inference . Its attribute classifiers are trained on a small amount of labeled data and their error rates are modeled in a probabilistic framework . We report on extensive experiments using several real data sets . Comparison with several estimators based on Bernoulli arm parameters , Beta densities per arm , and even simpler forms of GPs on the arm Beta distributions , shows that AAA is superior at quickly cutting down arm accuracy uncertainty . Summarizing , our contributions are : • We motivate and define the problem of accuracy surface estimation over a large space of attribute combinations . • Our proposed estimator AAA fits a Beta density for every attribute combination ( arm ) , with its parameters smoothed via two GPs to capture heteroscedastic uncertainty of each arm ’ s accuracy under limited data settings . • We propose two important components included in AAA : 1 ) a Dirichlet regularization to control over-smoothing of the Beta scale parameters , and 2 ) pooled observations to reduce over-fitting of a GP-associated kernel to sparse arms . • We show significant gains in terms of both estimation quality and the efficiency of exploration on four real classification models compared to existing methods . AAA obtains an average 80 % reduction in macro averaged square error over the existing methods . 2 Problem Setup . Our goal is to evaluate a given machine learning service model S used by a diverse set of consumers . The service S : X 7→ Y could be any predictive model that , for an input instance x ∈ X , assigns an output label ŷ ∈ Y , where Y is a discrete label space . Let y ( x ) denote the true label of x and Agree ( y , ŷ ) denote the match between the two labels . For scalar classification , Agree ( y , ŷ ) is in { 0,1 } . For structured outputs , e.g. , sequences , we could use measures like BLEU scores in [ 0,1 ] . Classifiers are routinely evaluated on their expected accuracy on a data distribution P ( X , Y ) : ρ = EP ( x , y ) [ Agree ( y , S ( x ) ] ( 1 ) We propose to go beyond this single measure and define accuracy as a surface over a space of attributes of the input instances . Let A denote a list of K attributes that capture the variability of consumer expectation on which the service S will be deployed . For instance , visual object recognition is affected by the background scene , and facial recognition is affected by demographic attributes . We use A ( x ) ∈ A to denote the vector of values of attributes of input x and A to denote the Cartesian product of the domains of all attributes . An attribute could be discrete , e.g. , the ethnicity of a speaker ; Boolean , e.g. , whether a scene is outdoors/indoors ; or continuous , e.g. , the age of the speaker in speech recognition . Some of the attributes of x , for example the camera settings of an image , may be known exactly , and others may only be available as a distribution Mk ( ak|x ) for an attribute ak ∈ A , obtained from a pre-trained probabilistic classifier . Generalizing from a single global expected accuracy ( 1 ) , we define the accuracy surface ρ : A → [ 0 , 1 ] of a service S at each attribute combination a ∈ A , given a data distribution P ( X , Y ) , as ρ ( a ) = EP ( x , y|A ( x ) =a ) [ Agree ( y , S ( x ) ] ( 2 ) Our goal is to provide an estimate of ρ ( a ) given two kind of data sampled from P ( X , Y ) : a small labeled sample D , and a large unlabeled sample U . In addition , we are given a budget of B instances for which we can seek labels y from a human by selecting them from U . Applying Mk to all of U is , however , free of cost . We aim to design a probabilistic estimator for ρ ( a ) , which we denote as P ( ρ|a ) where ρ ∈ [ 0 , 1 ] and a ∈ A . This is distinct from active learning , which selects instances to train the learner toward greater accuracy , and also active accuracy estimation [ 7 ] , which does not involve a surface over as . We also show that standard tools to regress from a to ρ are worse than our proposal . We measure the quality of our estimate as the square error between the gold accuracy ρ ( a ) and the mean of the estimated accuracy distribution P ( ρ|a ) . Our estimator distribution naturally gives an idea of the posterior variance of accuracy estimate of each attribute combination , which we use for uncertainty-based exploration . 3 Proposed Estimator . We will first review recent work that leads to candidate solutions to our problem , discuss their limitations , and finally present our solution . Initially , to keep the treatment simple , we assume A ( x ) and gold y ( hence c = Agree ( S ( x ) , y ) , the service correctness bit ) is known for all instances . Later in this section , we remove these assumptions . The simplest option is to ignore any relationship between arms , and , for each arm a , fit a suitable density over ρ ( a ) . When this density is sampled , we get a number in [ 0 , 1 ] , which is like a coin head probability used to sample correctness bits c. For representing uncertainty of accuracy values ( which are ratios between two counts ) , the Beta distribution B ( · , · ) is a natural choice . We call this baseline method Beta-I . The variance of the estimated Beta density can be used for actively sampling arms . Ji et al . [ 7 ] describe a related scenario , stressing on active sampling . However , this approach can not share observations or smooth the estimated density at a sparsely-populated arm with information from similar arms . In our real-life scenario , we expect accuracy surface smoother and the number of arms to be large enough that many arms will get very few , if any , instances . The second baseline method , which we call BernGP , is to view the ( a , c ) instances inD as a standard classification data set with the binary c values as class label and a as input features . Given the limited data , we can use the well-known GP classification approach [ 9 ] for fitting smooth values ρ as a function of a . Suppose the arms a can be embedded to V ( a ) in a suitable space induced by some similarity kernel . In this embedding space , we expect the accuracy of S to vary smoothly . Given a kernel K1 ( a , a′ ) to guide the extent of sharing of information across arms , a standard form of this GP would be P ( c|a ) = Bernoulli ( c ; sigmoid ( fa ) ) ; f ∼ GP ( 0 , K1 ) . ( 3 ) The GP can give estimates of uncertainty of ρ ( a ) , which may be used for active sampling of arms . As we will demonstrate , such GP-imposed estimate of uncertainty of ρ ( a ) is inadequate , because it loses sight of the number of supporting observations at each arm , which could be very diverse . This is because the standard GP assumption of homoscedasticity , that is , identical noise around each arm is violated when observations per arm differ significantly . We therefore need a mechanism to separately account for the uncertainty at each arm , even the unexplored ones , to guide the strategy for actively collecting more labeled data .

Summary:         This paper describes a method to estimate the predictive performance of a black-box classifier on different tasks depending on the subsets of attributes considered. Some level of smoothness is assumed depending on the attributes considered for prediction. This smoothness is captured using a Gaussian process. The model also uses a beta distribution as the generative model for the probability of correctly predicting a label for a data instance. Further improvements are introduced in the model using a Dirichlet distribution. The proposed method is evaluated on several tasks and compare with several base-lines that are variants of the proposed method. An active learning strategy that queries the points with the highest associated variance is suggested.