FrugalMCT: Efficient Online ML API Selection for Multi-Label Classification Tasks

1 INTRODUCTION . Many machine learning users are starting to adopt machine learning as a service ( MLaaS ) APIs to obtain high-quality predictions . One of the most common tasks these APIs target is multi-label classification . For example , one can use Google ’ s computer vision API ( Goo ) to tag an image with a wide range of possible labels for $ 0.0015 , or Microsoft ’ s API ( Mic ) for $ 0.0010 . Another example is to extract all text strings from an image for $ 0.005 via iFLYTEK ’ s API ( Ifl ) or $ 0.021 via Tencent ’ s API ( Ten ) . In practice , these APIs also provide different performance on different types of input data ( e.g. , English vs Chinese text ) . The heterogeneity in APIs ’ performance and prices makes it hard for users to decide which API , or combination of APIs , to use for their own datasets and budgets . Recent work ( Chen et al. , 2020 ) proposed FrugalML , an algorithmic framework that adaptively decides which APIs to call for a data point to optimize accuracy and cost . Their approach learns a fast decision rule for each possible output label that can significantly improve cost-performance over the individual APIs . However , FrugalML requires a large amount of training data and involves solving a non-convex optimization problem with complexity exponential in the number of distinct labels . This prevents it from being used for tasks with large number of labels , such as multi-label classification . Furthermore , FrugalML ignores correlation between different APIs ’ predictions , potentially limiting its accuracy . For example , APIs A and B may output { person , car } and { car , bike } separately for an image whose true keywords are { person , car , bike } . FrugalML would select one of the two label sets , but combining them results in the true label set and thus higher accuracy . Thus , this paper aims to solve these significant limitations and address the question : how do we design efficient ML API selection strategies for multi-label classification tasks to maximize accuracy within a budget ? We propose FrugalMCT , a principled framework that learns the strengths and weaknesses of different combinations of multi-label classification APIs , and efficiently selects the optimal combinations of APIs to call for different data items and budget constraints . As shown in Fig . 1 ( a ) , FrugalMCT directly estimates the accuracy of each API combination on a particular input based on the features and predicted labels of that input . Then it uses a fast service selector based on the estimated accuracy to balance accuracy and budget . For example , we might first call API A on an input . If A returns person and teddy bear and the accuracy predictor gives relatively high estimated accuracy ( Fig . 1 ( c ) ) , then we stop and report { person , teddy bear } as the label set . If A returns person and tennis racket , and we predict that combining it with API B ’ s output gives a much higher accuracy , then we invoke API B and combine their prediction to obtain { person , sports ball , tennis racket } ( Fig . 1 ( d ) ) . Contributions . FrugalMCT is an end-to-end approach that integrates the selection of APIs and the combination of their outputs for individual user queries . It leverages our key finding that current commercial APIs have complementary strengths and weaknesses , and that we can reliably predict which APIs are likely to work well for a new query based on easy-to-generate metadata about its input . FrugalMCT then executes an efficient online algorithm to determine which combination of APIs to call for different user queries . We show that the online algorithm enjoys an accuracy provably close to the offline method as well as a small computational cost . All components in FrugalMCT are trainable , making it easy to customize for different applications . To our knowledge , FrugalMCT is the first work on how to effectively select and combine multi-label ML APIs . Empirically , FrugalMCT produces substantially better prediction performance than individual APIs and than FrugalML adapted for multi-label tasks ( Fig . 1 ( b ) ) . Extensive experiments with real commercial APIs on several tasks , including multi-label image classifications , scene text recognition , and named entity recognition , show that FrugalMCT typically provides over 60 % ( as high as 98 % ) cost reduction when aiming to match the best commercial API ’ s performance . Also , when targeting the same cost as the best commercial API , FrugalMCT can improve performance up to 8 % . We will release our dataset of 295,212 samples annotated by commercial multi-label APIs as the largest dataset and resource for studying multi-label ML prediction APIs . Related work . MLaaS : With the growing importance and adoption of MLaaS APIs ( Ama ; Ten ; Goo ; IBM ; Mic ) , existing research has largely focused on evaluating individual API for their performance ( Yao et al. , 2017 ) , robustness ( Hosseini et al. , 2017 ) , biases ( Koenecke et al. , 2020 ) and applications ( Buolamwini & Gebru , 2018 ; Hosseini et al. , 2019 ; Reis et al. , 2018 ) . Recent work on FrugalML ( Chen et al. , 2020 ) studies API calling strategies for single label classification . While their approach ’ s computational complexity is exponential in the number of labels , FrugalMCT ’ s complexity does not depend on the number of labels , making it suitable for multi-label prediction APIs . In addition , FrugalML selects only one API per user query , while FrugalMCT considers the combination of multiple APIs ’ output for each input data . This improves the overall accuracy ( as shown in Sec 4 ) , but also creates unique optimization challenges that we solve . Ensembles for multi-label classification : Ensemble learning is a natural approach to combine different predictors ’ output . Several ensemble methods have been developed , such as using pruned sets ( Read et al. , 2008 ) , classifier chains ( Read et al. , 2011 ) , and random subsets ( Tsoumakas & Vlahavas , 2007 ) , with applications in image annotations ( Xu et al. , 2011 ) , document classification ( Chen et al. , 2017 ) , and speech categorization ( Liu et al. , 2019 ) . Moyano et al . ( 2018 ) provide a detailed survey of this area . Almost all of these ensemble methods require joint training of the base classifiers , but MLaaS APIs are black box to the users . Also , while ensemble methods focus only on improving accuracy , FrugalMCT explicitly considers the cost of each API and enforces a budget constraint . Model cascades : A series of works ( Viola & Jones , 2001a ; b ; Sun et al. , 2013 ; Cai et al. , 2015 ; Wang et al. , 2011 ; Xu et al. , 2014 ; Chen et al. , 2018 ; Kumar et al. , 2018 ; Chen et al. , 2018 ) explores cascades ( a sequence of models ) to balance the quality and runtime of inference . Model cascades use a single predicted quality score to avoid calling computationally expensive models , but FrugalMCT ’ strategies utilize both quality scores and predicted label sets to select an expensive add-on service . AutoML for multi-label classification : AutoML ( Thornton et al. , 2013 ) automates the customization of ML pipelines , including the selection , combination , and parametrization of the learning algorithms . There is a rich literature of AutoML techniques for standard single label tasks , and fewer methods on multi-label predictions ( Wever et al. , 2021 ) ( e.g . genetic algorithms ( de Sá et al. , 2017 ) and a neural network-based search scheme ( Pakrashi & Namee , 2019 ) ) . Applying AutoML to use multiple ML APIs is underexplored , and FrugalMCT can be viewed as the first AutoML approch designed for automating the selection of multiple mutlti-label ML APIs . While most AutoML systems exclusively focus on prediction performance , FrugalMCT optimizes accuracy and cost jointly , which is desirable for cost-sensitive API users . Multiple choice knapsack and integer programming : Many resource allocation problems can be modeled as multiple choice knapsack problem ( MCKP ) ( Pamela H. Vance & Toth ) , 1993 ) , such as keyword bidding ( Zhou & Naroditskiy , 2008 ) and quality of service control ( Lee et al. , 1999 ) . While NP-hard ( Sinha & Zoltners , 1979 ) , various approximations have been proposed for MCKP , such as branch and bound ( Pamela H. Vance & Toth ) , 1993 ) , convex hull relaxation ( Akbar et al. , 2006 ) and bi-objective transformation ( Bednarczuk et al. , 2018 ) . Inherently an integer linear programming ( ILP ) problem , MCKP can also be tackled by ILP solvers , motivated by online adwords searching ( Devanur & Hayes , 2009 ) , resource allocation ( Devanur & Hayes , 2019 ) and general linear programming ( Li et al. , 2020 ) . The service selector of FrugalMCT can be viewed as a MCKP with the same item cost vector per item group , which we leverage to obtain a customized fast and online solver . 2 PRELIMINARIES . Notation . We denote matrices and vectors in bold , and scalars , sets , and functions in standard script . Given a matrix A ∈ Rn×m , we let Ai , j denote its entry at location ( i , j ) . 1 ( · ) represents the indicator function . Multi-label classification Tasks . Throughout this paper , we focus on multi-label classification tasks : assigning a label set Y ⊆ Y to any data point x ∈ X . In contrast to basic supervised learning , in multi-label learning each data point is associated with a set of labels instead of a single label . Many MLaaS APIs target such tasks . Consider , for example , image tagging , where X is a set of images and Y is the set of all tags . Example label sets could be { person , car } or { bag , train , sky } . MLaaS Market . Consider a MLaaS market consisting of K different ML services for some multilabel task . For a data point x , the kth service returns to the user a set of labels with their quality scores , denoted by Yk ( x ) ⊆ Y × [ 0 , 1 ] . For example , one API for multi-label image classification might produce Yk ( x ) = { ( person , 0.8 ) , ( car , 0.7 ) } , indicating the label person with confidence 0.8 and car with confidence 0.7 . Let the vector c ∈ RK denote the unit cost of all services . For example , ck = 0.01 means that users need to pay $ 0.01 every time they call the kth service . 3 FRUGALMCT FRAMEWORK . In this section , we present FrugalMCT , a framework to adaptively select ML APIs for multi-label classification tasks within a budget . All proofs are left to the appendix . We generalize the scheme in Figure 1 ( a ) to K ML services . As shown in Figure 2 , FrugalMCT contains three main components : an accuracy estimator , a service selector , and a label combiner . Given a data point x , it first calls some base service , denoted by base , which is one of the K APIs , and obtains Ybase ( x ) . Often , base is a cheap or free service ; we discuss how to choose base in Section 3.4 . Next , an accuracy predictor produces a vector â ( x ) ∈ [ 0 , 1 ] K , whose kth value estimates the accuracy of the label set produced by the label combiner using base ’ s and kth API ’ s outputs . The service selector s ( · ) : X 7→ [ K ] then decides if and which add-on service needs to be invoked . Finally , a label combiner generates a label set by combining the predictions from the base and add-on APIs . Take Figure 1 ( d ) as an example . The image is first passed to the GitHub model , which produces { ( person , 0.46 ) , ( tennis racket,0.18 ) } , by which the accuracy predictor predicts the accuracy of the label set generated by combining each API ’ s output with GitHub model ’ s . The service selector then decides to further invoke Everypixel , which gives { ( person , 0.46 ) , ( sports ball , 0.52 ) } . Finally , the label combiner uses both APIs ’ output for the final prediction . FrugalMCT allows users to customize the accuracy predictor and the label combiner , depending on the applications . For example , for the image tagging problem , one might use image features ( e.g. , brightness and contrast ) to build the accuracy predictor , while word embeddings can be more useful for named entity recognition . In the following sections , we explain the key of accuracy predictor , API selector and the label combiner in more detail .

Summary: Given several mutli-label machine learning APIs, this paper study how to select those APIs under a budget constrant while striving to improve the overall accuracy. The author first formulate the budget API select problem as an integer linear programming problem, then relax the integer contraint and solving the relax problem in dual. The advantage of such modeling is to have a fast decision function for online deployment of their API selection systems. The experiment results are promising and consists of several ablation studies.