Year: 2021

Abstract:
ML prediction APIs from providers like Amazon and Google have made it simple to use ML in applications. A challenge for users is that such APIs continuously change over time as the providers update models, and changes can happen silently without users knowing. It is thus important to monitor when and how much the ML APIs’ performance shifts. To provide detailed change assessment, we model ML API shifts as confusion matrix differences, and propose a principled algorithmic framework, MASA, to provably assess these shifts efficiently given a sample budget constraint. MASA employs an upper-confidence bound based approach to adaptively determine on which data point to query the ML API to estimate shifts. Empirically, we observe significant ML API shifts from 2020 to 2021 among 12 out of 36 applications using commercial APIs from Google, Microsoft, Amazon, and other providers. These real-world shifts include both improvements and reductions in accuracy. Extensive experiments show that MASA can estimate such API shifts more accurately than standard approaches given the same budget.

heading: 1 INTRODUCTION
text: Machine learning (ML) prediction APIs have made it dramatically easier to build ML applications. For example, one can use Microsoft text API (Mic, a) to determine the polarity of a text review written by a customer, or Google speech API (Goo, b) to recognize users’ spoken commands received by a smart home device. These APIs have been gaining in popularity (MLa; Chen et al., 2020), as they eliminate the need for ML users to train their own models.
Monitoring and assessing the performance of these third-party ML APIs over time, however, is under-explored. ML API providers continuously collect new data or change their model architectures (Qi et al., 2020) to update their services, which could silently help or harm downstream applications’ performance. For example, as shown in Figure 1 (a) and (b), we observe a 7% overall accuracy drop of IBM speech API on the AUDIOMNST dataset in March 2021 compared March 2020. In our systematic study of 36 API and dataset combinations, there are 12 cases where the API’s performance changed by more than 1% on the same dataset from 2020 to 2021 (sometimes for the worse). Such performance shifts are of serious concern not only because of potential disruptions to downstream tasks, but also because consistency is often required for audits and oversight. Therefore, it is important to precisely assess shifts in an API’s predictions over time. Moreover, in this assessment, it is often more informative to quantify how the entire confusion matrix of the API has changed rather than just the overall accuracy. In the IBM case in Figure 1, it is interesting that a major culprit of the drop in performance is the 2021 model mistaking “four” for “five”. In other settings, changes in the confusion matrix could still cause issues even if the overall accuracy stays the same.
In this paper, we formalize the problem of assessing API shifts as estimating changes in the confusion matrix on the same dataset in a sample-efficient manner (i.e., with few calls to the API itself to minimize dollar cost). The straightforward approach is to compare the API’s prediction on randomly sampled data. However, this can require a large number of API calls to estimate the confusion matrix, which is expensive given that each call costs money. To help address this challenge, we propose MASA, a principled algorithm for ML API shift assessments. MASA efficiently estimates shifts in the API’s confusion matrix by clustering the dataset and adaptively sampling data from different clusters to query the API. MASA automates its sampling rate from different data clusters based on the uncertainty in the confusion matrix estimation. For example, MASA may query the ML API on
more samples with the true label “four” than “one”, if it is less sure about the estimated performance change on the former. Employing an upper-confidence-bound approach to estimate the uncertainties, MASA enjoys a low computation and space cost as well as a fast estimation error rate guarantee.
MASA’s adaptive sampling substantially improves the quality of estimation for API shifts. In extensive experiments on real world ML APIs, MASA’s assessment error is often an order of magnitude smaller than that of standard uniform sampling with same sample size (e.g., Figure 1 (c)). To reach the same tolerable target estimation error, MASA can reduce the required sample size by more than 50%, sometimes up to 90%.
Contributions. In short, our main contributions include:
1. We demonstrate that commercial Ml APIs can experience significant performance shifts over time, and formulate ML API shift assessments via confusion matrix difference estimation as an important practical problem. We will release the dataset consisting of 1,224,278 samples annotated by ML APIs in different date to stimulate more research on ML API shifts.
2. We propose MASA, an algorithm to assess the ML API performance shifts efficiently. MASA adaptively determines querying the ML API on which data points to minimize the shift estimation error under a sample size constraint. We show that MASA enjoys a low computation cost and performance guarantees.
3. We evaluate MASA on real world APIs from Google, Microsoft, Amazon and other providers for tasks including speech recognition, sentiment analysis, and facial emotion recognition. MASA leads to estimation errors an order of magnitude smaller than standard uniform sampling using the same sample size, or over 90% fewer samples to reach the same tolerable estimation error.
Related Work. Distribution shifts in ML deployments: Performance shifts in ML systems have been observed in applications like disease diagnosis (Lipton et al., 2018), facial recognition (Wang et al., 2020), and Inference of molecular structure (Koh et al., 2020). Most of them are attributed to distribution shifts, i.e., the distribution of the test and training datasets are different. Distribution shifts are usually modeled as covariate shifts (Shimodaira, 2000; Sugiyama et al., 2007; QuiñoneroCandela et al., 2009) or label shifts (Lipton et al., 2018; Saerens et al., 2002; Azizzadenesheli et al., 2019; Zhao et al., 2021). API shifts are orthogonal to distribution shifts: instead of attributing the performance shifts to data distribution changes, API shifts concern with ML APIs changes which changes its predictions on the same dataset. The methods for detecting distribution drifts typically rely on changes in data feature statistics and can not detect changes in the API on the same data. To the best of our knowledge, MASA is the first work to systematically investigate ML API shifts.
Deploying and monitoring ML APIs: Several issues in deployed ML APIs have been studied. For example, Buolamwini & Gebru (2018) showed that strong biases toward minority can exist in
commercial APIs and Ribeiro et al. (2020) revealed that several bugs in commercial APIs can be detected using checklists. Kang et al. (2020) extend program assertions to monitor and improve deployed ML models. Chen et al. (2020) consider the trade-offs between accuracy performance and cost via exploiting multiple APIs. On the other hand, the proposed MASA focuses on estimating (silent) API performance changes cheaply and accurately, which has not been studied before.
Stratified sampling and multi-arm bandits: Stratified sampling has proved to be useful in various domains, such as approximate query processing (Chaudhuri et al., 2007), population mean estimation (Carpentier & Munos, 2011; Carpentier et al., 2015), and complex integration problems (Leprêtre et al., 2017). A common approach is to model stratified sampling as a multi-arm bandit (MAB) problem: view each data partition as an arm, and set the regret as the variance of the obtained estimator. Plenty of algorithms exist for solving standard MAB problems, such as the epsilon greedy algorithm (Slivkins, 2019), upper-confidence-bound approach (Auer et al., 2002), and Thompson sampling method (Russo et al., 2018). While those algorithms aim at playing the best arm as often as possible, stratified sampling’s goal is often to estimate the average of all arms, and thus needs to allocate the number of calls for all arms aptly based on their variance. In contrast to estimating the population average in standard stratified sampling, our goal is to simultaneously estimate a matrix whose entries can be correlated. Thus, we have an objective (uncertainty score) that is different from theirs (variance), which also leads to an optimal allocation different from that of standard stratified sampling. This difference requires additional statistical bounds to prove convergence, which we provide in our paper. It also results in a different upper-confidence-bound term and convergence rate compared to standard bandit algorithms.

heading: 2 THE API SHIFT PROBLEM
text: Empirical assessment of ML API shifts. We start by making an interesting observation: Commercial ML APIs’ performance can change substantial over time on the same datasets. We investigated twelve standard datasets across three different tasks, namely, YELP (Dat, c), IMDB (Maas et al.), WAIMAI (Dat, b), SHOP (Dat, a) for sentiment analysis, FER+ (Goodfellow et al., 2015), RAFDB (Li et al.), EXPW (Zhang et al.), AFNET (Mollahosseini et al., 2019) for facial emotion recognition, and DIGIT (Dat, d), AMNIST (Becker et al., 2018), CMD (Warden, 2018), FLUENT (Lugosch et al.), for speech recognition. For each dataset, we evaluated three commercial ML APIs’ accuracy in April 2020 and April 2021. Figure 2 summarizes the overall accuracy changes.
There are several interesting empirical findings. First, API performance changes are quite common. In fact, as shown in Figure 2, API performance changes exceeding 1% occurred in about 33% of all (36) considered ML API-dataset combinations. Since the data distribution remains fixed, such a change is due to ML APIs’ updates. Second, the API updates can either help or hurt the accuracy performance depending on the datasets. For example, as shown in Figure 2 (a), the Amazon sentiment analysis API’s accuracy increases on YELP, WAIMAI, and SHOP, but decreases on IMDB. In addition, the update of Microsoft facial emotion recognition API only affects performance on the FER+ dataset, as
shown in Figure 2 (b). Another interesting finding is that the magnitude of the performance change can be quite different. In fact, most of the accuracy differences are between 1–3%, but on DIGIT dataset, Google’s accuracy change is more than 20%.
Fine-grained assessment of API shift as changes in the confusion matrix. Based on feedback from practitioners, accuracy change alone is insufficient, and attribution to per class change is often much more informative (Tsipras et al., 2020; Hou et al., 2019; Luque et al., 2019). Thus, a natural idea is to quantify an ML API’s performance by its confusion matrix. We assess the change of the confusion matrix over time as a measure of API shift.
Formally, consider an ML service for a classification task with L labels. For a data point x from some domain X , let ŷ(x) ∈ [L] denote its predicted label on x, and y(x) be the true label. For example, for sentiment analysis, x is a text paragraph, and the task is to predict if the polarity of x is positive or negative. Here L = 2, and ŷ(x) = 1 implies positive predicted label while y(x) = 2 indicates negative true label. The confusion matrix is denoted by C ∈ RL×L where C i,j , Pr[y(x) = i, ŷ(x) = j]. Given a confusion matrix of the ML API measured previously (say, a few months ago),Co, the ML API shift is defined as ∆C , C −CO. Using confusion matrix difference to quantify the ML API shift is informative. E.g, the overall accuracy change is simply the trace of ∆C . It also explains which label gets harder or easier for the updated API. Still consider, e.g., sentiment analysis. Given a 2% overall accuracy change, ∆C1,2 = 1% and ∆C2,1 = −3% implies that the change is due to predicting less (-3%) negative texts as positive, by sacrificing the accuracy on positive texts slightly (1%). This suggests that the API could have been updated with more negative training texts.

heading: 3 MASA: ML API SHIFT ASSESSMENT
text: Now we present MASA, an algorithmic framework efficiently to assess ML API shifts. Suppose the old confusion matrixCo and a large labeled dataset D are available. Given a query budget N , our goal is to generate ∆Ĉ , an estimation of the API shifts as accurately as possible by querying the ML API ŷ(·) on N samples drawn from D. MASA achieves its goal via an adaptive sampling approach (Figure 3). It first divides the given dataset D into several partitions (clusters). Then it adaptively decides which sample to query the ML API in an iterative manner: at each iteration, it selects one data partition based on some uncertainty measure (defined below), and queries the ML API on one sample randomly drawn from this partition. The API’s prediction is obtained to update the uncertainty measure as well as the estimated shift ∆Ĉ . This process is repeated until the ML API has been queried N times or if a stopping rule is reached. We explain each step in detail next.

heading: 3.1 DATA PARTITIONING
text: A key intuition in MASA is that not all samples are equally informative for estimating API shifts. Consider, for example, a vision API makes perfect predictions on “dog” images, and guesses randomly
on ”cat“ pictures. The “dog” images are less informative, as even a small sample of “dog” queries would tell that there is essentially no confusion for this class. Intuitively, within a sample budget, an estimator with more samples from “cat” pictures should be more accurate overall than that from “dog”. Generally, harder images tend to be more informative.
Thus, it is a natural idea to partition all data points based on factors that may correlate with their informativeness, and sample from those partitions separately. In MASA, we use partitions Di,k that each contain the points with true label i and difficulty level k. The difficulty level is an integer indicating how hard it is to predict the data point’s label. It needs not be perfect, and can be simply the discretized prediction confidence generated by some simple ML models. A total of L labels and K distinct difficulty labels lead to a total of LK partitions. If the uncertainty or variability of the ML API’s prediction on each partition is different, then drawing a different number of samples from each partition may improve the shift assessment performance compared to standard uniform sampling. We verify this empirically in our evaluation (Section 4).

heading: 3.2 BUDGET ALLOCATION PROBLEM
text: Given the data partition, two questions arise: (i) how many samples should be drawn from each partition, and (ii) how to estimate the ML API shifts given available samples. The second question is relatively straightforward. Note that the API shifts satisfy
∆C i,j = Pr[y(x) = i, ŷ(x) = j]−Coi,j = L∑ i=1 K∑ k=1 Pr[y(x) = i, ŷ(x) = j, x ∈ Di,k]−Coi,j
= K∑ k=1 Pr[ŷ(x) = j, x ∈ Di,k]−Coi,j = K∑ k=1 Pr[x ∈ Di,k] Pr[ŷ(x) = j|x ∈ Di,k]−Coi,j
where the first equation is by definition, the second is due to total probability rule, the third uses the fact that x ∈ Di,k implies y(x) = i, and the last equation applies conditional probability. Here, Pr[x ∈ Di,k] is simply ratio of size of partition Di,k and entire dataset D, known a prior. To assess ∆C i,j , we only need to estimate Pr[ŷ(x) = j|x ∈ Di,k], the predicted label distribution on partition Di,k. It can be estimated simply via the frequency of predicting label j among all available samples drawn from Di,k.
Now we consider the sample allocation problem. For ease of notation, we denote Pr[xi ∈ Di,j ] by pi,k, Pr[ŷ(x) = j|x ∈ Di,k] and its estimation by µi,k,j and µ̂i,k,j , respectively. Then for deterministic sample allocations, the squared Frobenius norm error can be written as
E [ ‖∆C −∆Ĉ‖2F ] = ∑ i,j E ( ∆C i,j −∆Ĉ i,j )2 = ∑ i,j E (∑ k pi,k[µi,k,j − µ̂i,k,j ] )2 = ∑ i,j,k p2i,kE ( [µi,k,j − µ̂i,k,j ]
)2 Thus we use the loss L(A, N) , ∑ i,j,k p 2 i,kE ( [µi,k,j − µ̂i,k,j ] )2 to measure the performance of any sample budget allocation algorithm A using N samples. For any fixed N , our goal is to find a sample budget allocation algorithm A to minimize the loss L(A,N). Notably, we can generalize it for other scenarios by replacing (∆C −∆Ĉ) withW (∆C −∆Ĉ), where is element-wise multiplication andW is an L×L weight matrix. Different choices ofW can penalize the error of each entry in ∆Ĉ differently and serve for different purposes. For example, if misclassifying label 1 as label 2 is the only focus, then we can simply setW 1,2 = 1 andW i,j = 0,∀(i, j) 6= (1, 2). If we use the identity matrix as the weightW , then it becomes equivalent to estimating the overall accuracy by minimizing the variance. To minimize the loss for the general scenarios, Algorithm 1 (which is explained in the rest of this section) is still applicable by simply multiplying the API selection formula (line 3 in Algorithm 1) with its corresponding weights.

heading: 3.3 UNCERTAINTY SCORE AND OPTIMAL ALLOCATION
text: The optimal sample allocation is directly connected to how informative each data partition is. To see this, let us first introduce the notation of uncertainty score for each data partition.
Algorithm 1 MASA’s ML API shift assessment algorithm. Input :ML API ŷ(·), query budget N , partitions Di,k, p ∈ RL×K ,Co ∈ RL×L, and a > 0 Output :Estimated ML API Shift ∆Ĉ ∈ RL×L̂
1 SetN = 0L×K , µ̂ = 0L×K×L, σ̂ = 0L×K ,H = 0L×K×L . Initialization 2 for n← 1 to N do
3 (i∗, k∗)← (i, k), ifN i,k < 2arg maxi,k pi,kN i,k (σ̂i,k + 4√ aN i,k) , o/w . Determine data partition 4 Sample xn from Di∗,k∗ and query the ML API to obtain ŷ(xn) 5 N i∗,k∗ ←N i∗,k∗ + 1 . Update sample size 6 µ̂i∗,k∗,j ← µ̂i∗,k∗,j + 1ŷ(xn)=j−µ̂i∗,k∗,j N i∗,k∗ ,∀j ∈ [L] . Update predicted label distribution
7 σ̂2i∗,k∗ ←  1 2H i∗,k∗,ŷ(xn), ifN i∗,k∗ < 2
σ̂2i∗,k∗ + 1−
Hi∗,k∗,ŷ(xn) Ni∗,k∗−1
−σ̂2i∗,k∗ N i∗,k∗ , o/w . Update uncertainty score
8 H i∗,k∗,ŷ(xn) ←H i∗,k∗,ŷ(xn) + 1 . Update label frequency 9 end
10 Return ∆Ĉ ∈ RL×L where ∆Ĉ i,j = ∑K k=1 pi,kµ̂i,k,j −Coi,j ,∀i, j . Confusion estimation
Definition 1. σ2i,k , (1− ∑L j=1 Pr 2[ŷ(x) = j|x ∈ Di,k]) denotes the uncertainty score of Di,k.
The uncertainty score quantifies how informative each Di,k is by subtracting from 1 the sum of the square of each label’s probability mass. The uncertainty score is related to collision entropy (discussed in Appendix A), and determines the optimal allocation as follows. Lemma 1. Let A∗ be the sample allocation algorithm that achieves the smallest expected squared Frobenius norm error. Then the number of samples drawn from Di,k by A∗ is
N ∗i,k = pi,kσi,k∑ i,k pi,kσi,k N
Lemma 1 shows that the optimal budget allocation depends on the uncertainty score, but in practice, we do not know the uncertainty score before drawing samples and querying the ML API. Thus, a natural question is how to estimate the uncertainty score σ2i,k. Suppose n samples, x1, x2, · · · , xn, are drawn from partition Di,k. Then we can estimate σ2i,k by
σ̂2i,k , 1− 1
n(n− 1) n∑ s=1 n∑ t:t=1,t6=s 1ŷ(xs)=ŷ(xt) (3.1)

heading: 3.4 AN UNCERTAINTY-AWARE ADAPTIVE SAMPLING ALGORITHM
text: Now we have a chicken-and-egg problem: estimating the uncertainty scores is needed to find the optimal sample allocation, but sampling from all partitions is needed to estimate their uncertainty scores. To overcome this issue, we adopt an iterative sampling approach, as shown in Algorithm 1. At each iteration, it alternates between (i) uncertainty score-based new sample selection (line 3 - 4) and (ii) uncertainty score and predicted label distribution update using the new sample (line 5 - 8). After querying the ML API N times, the API shifts are obtained by (iii) fusing the estimated predicted label distribution on each partition (line 10). We give the details as follows.
Uncertainty score and predicted label distribution update. After obtaining the predicted label for a sample from partition Di∗,k∗ , we need to update (i) the number of samples already drawn from this partition, denoted by N i∗,k∗ , (ii) the estimated predicted label distribution, denoted by µ̂i∗,k∗,j ,∀j, and (iii) the estimated uncertainty score, σ̂2i∗,k∗ . For N i∗,k∗ and µ̂i∗,k∗,j (line 5-6), we use standard incremental update approach (Cotton, 1975), which requires constant space and computational cost per iteration. For σ̂2i∗,k∗ , we additionally maintain the number of label j being predicted among all samples drawn from Di∗,k∗ , denoted byH i∗,k∗,j (line 8). This enables a fast incremental update of σ̂2i∗,k∗ (line 7).
Uncertainty score-based new sample selection. To determine on which partition to select a new sample, we use an upper-confidence-bound approach on the weighted uncertainty score (second case in line 3), after ensuring two samples have been drawn from each partition (first case in line 3). Two samples are needed for an initial estimation of each partition’s uncertainty score. Here, we use a parameter a > 0 to balance between exploiting knowledge of uncertainty score (σ̂2i,k) and exploring
more partitions ( 4 √
1 N i,k ).
We quantify the performance of MASA v.s. the optimal allocation algorithm A∗ as follows. Theorem 2. If a > 2 logL+ logK + 94 logN and N > 4LK, then we have
L(MASA, N)− L(A∗, N) ≤ O(N− 54 log 1 4 N)
Roughly speaking, Theorem 2 shows that the loss gap between the API shift estimated by MASA and the (unreachable) optimal allocation algorithm ceases in the rate of N−5/4. Note that the loss of the optimal allocation decays in the rate of N−1. Thus, as N gets larger and larger, the relative gap becomes more and more negligible.

heading: 4 EXPERIMENTS
text: We apply MASA to estimate the shifts of several real world ML services for various tasks. Our goal is three-fold: (i) understand if and why MASA assess the API shifts efficiently, (ii) examine how much sample cost MASA can reduce compared to standard sampling, and (iii) exploit the trade-offs between estimation accuracy and query cost achieved by MASA. We also study how the hyperparameters affect MASA’s performance, left to Appendix C.
Tasks, ML APIs, and datasets. As shown in Section 2, we have observed 12 of 36 cases where there is a >1% overall accuracy performance change of an ML API. Thus, we focus on evaluating MASA’s performance on those 12 cases. Except for case study, all experiments were averaged over 1500 runs. In all tasks, we created partitions using difficulty levels induced by a cheap open source model from GitHub. More details are in Appendix C.
Sentiment analysis: a case study on Amazon API. We start by a case study on Amazon API on a sentiment analysis dataset, YELP to understand MASA’s performance. We adopt MASA with sample budget 2000. The dataset is divided into 4 partitions D+,l, D+,h, D−,l, D−,h, depending on whether the true label is positive (+) or negative (-), and quality score produced by the 2020 version is lower (l) or higher (h) than the median.
We first note that the API shift gives an interesting explanation to Amazon API’s accuracy change. In fact, as shown in Figure 4 (a-c), the accuracy increase is mostly because more texts (2.7%) with negative altitudes are correctly classified. One possible explanation is that the API has been retrained on a dataset with more negative texts. Next, we observe that MASA produces accurate estimation of the API shift by comparing Figure 4 (c) and (d). This is primarily due to (i) that the data partitioning separates more uncertain data from less uncertain ones, and (ii) that adaptive sampling learns the uncertainty level effectively. Figure 4 (e-f) shows that while the partitions’ size is similar, their uncertainty scores are different. For example, higher quality score implies a much smaller uncertainty for positive texts (D+,h and D+,l in Figure 4(f)). As shown in Figure 4 (g), MASA indeed learns to utilize such imbalanced uncertainty: its sampling allocation for each partition is quite close to the optimal allocation (dark star point). Finally, it is worth noting that MASA outperforms standard uniform sampling notably, as shown in Figure 4(h). This is because uniform sampling does not exploit the uncertainty of each partition.
Budget savings achieved by MASA. In many applications, it suffices to obtain an estimated API shift close to the true shift, e.g., within a 1% Frobenius norm error. Thus, a natural question arises: to reach the same tolerable estimation error, how much sampling cost can MASA reduce compared to standard sampling approaches?
To answer this question, we compare MASA with two natural approaches: (i) uniform sampling and (ii) stratified sampling by drawing same number of samples for each true label. For each approach,
we measure the number of samples needed to reach 1% Frobenius norm error with probability 95%, via an upper bound on the estimated Frobenius error. The details are left to Appendix C. As shown in Table 1, MASA usually requires more than 70% fewer samples to reach such tolerable Frobenius norm error than the uniform and stratified sampling, primarily due to its shift estimation is more accurate. Uniform and stratified sampling required the similar number of samples because the upper bounds on their estimated Frobenius error are similar.
Trade-offs between estimation error and query budget . Next we examine the trade-offs between API shift estimation error and sample size achieved by MASA, shown in Figure 5. We first note that, across all 12 observed API shifts, MASA consistently outperforms standard uniform sampling for any fixed sample size. In fact, the achieved estimation error of MASA is usually an order of magnitude smaller than that of uniform sampling. This verifies that MASA can provide
more accurate assessments of API shifts in diverse applications. Second, some API shifts are easier to estimate than others. For example, for Google API shift on AMNIST, using 1000 samples already gives an expected squared Frobenius norm error lower than 10−4, while it usually requires 2000 samples for other shifts. This is probably because the skew in its uncertainties among different partitions is more severe than other shifts.

heading: 5 CONCLUSION
text: In this paper, we identify and formulate the problem of characterizing ML API shifts. Our systematic empirical study has shown that API model updates are frequent, and that some updates can reduce performance substantially. Quantifying such shifts is an important but understudied problem that can greatly affect the reliability of applications using ML-as-a-service. To capture fine-grained change assessment, we model the ML API shifts as confusion matrix differences. Adaptive sampling methods are typically designed for estimating a single scalar (e.g., adaptive population mean) or training a model (e.g., active learning), and thus not directly applicable for API shift estimation. Uniform sampling is natural but requires a large number of samples. Thus, we propose an algorithmic framework, MASA, to adaptively assess the API shifts using as few samples as possible. Our work focuses on estimating changes in the confusion matrix because the confusion matrix is often what is used by practitioners to assess API performance. We acknowledge that confusion matrices are most applicable for classification tasks, and other measures need to be used for more complex APIs (e.g. OCR, NLP). While this is a limitation, classification with a small to moderate number of classes of interest is a common use case for ML APIs, and this is an important starting point since it has not been studied before. We will release our dataset of 1,224,278 samples annotated by commercial APIs in different dates as the first dataset and resource for studying ML API performance shifts.

Review summaries:
This paper considers the problem of estimating the change in the performance of commercial ML APIs (ML as a service) as the models are updated over time (experiments are for 2020 vs 2021). It formalizes the problem as estimating the change in the confusion matrix over time. The main theoretical contribution is an adaptive sampling method to more efficiently estimate this shift. Interesting empirical results on various ML APIs are provided in the paper, showing the relevance of the problem and the effectiveness of the proposed method.

Authors show that ML models behind publicly available APIs change and these changes cause result changes for input datasets. Authors track the changes through confusion matrix differences. They propose an efficient algorithm they call MASA to evaluate the changes in results with reduced number of queries. Their algorithm achieves better estimates given the same budget than uniform sampling.