FrugalMCT: Efficient Online ML API Selection for Multi-Label Classification Tasks

1 INTRODUCTION . Many machine learning users are starting to adopt machine learning as a service ( MLaaS ) APIs to obtain high-quality predictions . One of the most common tasks these APIs target is multi-label classification . For example , one can use Google ’ s computer vision API ( Goo ) to tag an image with a wide range of possible labels for $ 0.0015 , or Microsoft ’ s API ( Mic ) for $ 0.0010 . Another example is to extract all text strings from an image for $ 0.005 via iFLYTEK ’ s API ( Ifl ) or $ 0.021 via Tencent ’ s API ( Ten ) . In practice , these APIs also provide different performance on different types of input data ( e.g. , English vs Chinese text ) . The heterogeneity in APIs ’ performance and prices makes it hard for users to decide which API , or combination of APIs , to use for their own datasets and budgets . Recent work ( Chen et al. , 2020 ) proposed FrugalML , an algorithmic framework that adaptively decides which APIs to call for a data point to optimize accuracy and cost . Their approach learns a fast decision rule for each possible output label that can significantly improve cost-performance over the individual APIs . However , FrugalML requires a large amount of training data and involves solving a non-convex optimization problem with complexity exponential in the number of distinct labels . This prevents it from being used for tasks with large number of labels , such as multi-label classification . Furthermore , FrugalML ignores correlation between different APIs ’ predictions , potentially limiting its accuracy . For example , APIs A and B may output { person , car } and { car , bike } separately for an image whose true keywords are { person , car , bike } . FrugalML would select one of the two label sets , but combining them results in the true label set and thus higher accuracy . Thus , this paper aims to solve these significant limitations and address the question : how do we design efficient ML API selection strategies for multi-label classification tasks to maximize accuracy within a budget ? We propose FrugalMCT , a principled framework that learns the strengths and weaknesses of different combinations of multi-label classification APIs , and efficiently selects the optimal combinations of APIs to call for different data items and budget constraints . As shown in Fig . 1 ( a ) , FrugalMCT directly estimates the accuracy of each API combination on a particular input based on the features and predicted labels of that input . Then it uses a fast service selector based on the estimated accuracy to balance accuracy and budget . For example , we might first call API A on an input . If A returns person and teddy bear and the accuracy predictor gives relatively high estimated accuracy ( Fig . 1 ( c ) ) , then we stop and report { person , teddy bear } as the label set . If A returns person and tennis racket , and we predict that combining it with API B ’ s output gives a much higher accuracy , then we invoke API B and combine their prediction to obtain { person , sports ball , tennis racket } ( Fig . 1 ( d ) ) . Contributions . FrugalMCT is an end-to-end approach that integrates the selection of APIs and the combination of their outputs for individual user queries . It leverages our key finding that current commercial APIs have complementary strengths and weaknesses , and that we can reliably predict which APIs are likely to work well for a new query based on easy-to-generate metadata about its input . FrugalMCT then executes an efficient online algorithm to determine which combination of APIs to call for different user queries . We show that the online algorithm enjoys an accuracy provably close to the offline method as well as a small computational cost . All components in FrugalMCT are trainable , making it easy to customize for different applications . To our knowledge , FrugalMCT is the first work on how to effectively select and combine multi-label ML APIs . Empirically , FrugalMCT produces substantially better prediction performance than individual APIs and than FrugalML adapted for multi-label tasks ( Fig . 1 ( b ) ) . Extensive experiments with real commercial APIs on several tasks , including multi-label image classifications , scene text recognition , and named entity recognition , show that FrugalMCT typically provides over 60 % ( as high as 98 % ) cost reduction when aiming to match the best commercial API ’ s performance . Also , when targeting the same cost as the best commercial API , FrugalMCT can improve performance up to 8 % . We will release our dataset of 295,212 samples annotated by commercial multi-label APIs as the largest dataset and resource for studying multi-label ML prediction APIs . Related work . MLaaS : With the growing importance and adoption of MLaaS APIs ( Ama ; Ten ; Goo ; IBM ; Mic ) , existing research has largely focused on evaluating individual API for their performance ( Yao et al. , 2017 ) , robustness ( Hosseini et al. , 2017 ) , biases ( Koenecke et al. , 2020 ) and applications ( Buolamwini & Gebru , 2018 ; Hosseini et al. , 2019 ; Reis et al. , 2018 ) . Recent work on FrugalML ( Chen et al. , 2020 ) studies API calling strategies for single label classification . While their approach ’ s computational complexity is exponential in the number of labels , FrugalMCT ’ s complexity does not depend on the number of labels , making it suitable for multi-label prediction APIs . In addition , FrugalML selects only one API per user query , while FrugalMCT considers the combination of multiple APIs ’ output for each input data . This improves the overall accuracy ( as shown in Sec 4 ) , but also creates unique optimization challenges that we solve . Ensembles for multi-label classification : Ensemble learning is a natural approach to combine different predictors ’ output . Several ensemble methods have been developed , such as using pruned sets ( Read et al. , 2008 ) , classifier chains ( Read et al. , 2011 ) , and random subsets ( Tsoumakas & Vlahavas , 2007 ) , with applications in image annotations ( Xu et al. , 2011 ) , document classification ( Chen et al. , 2017 ) , and speech categorization ( Liu et al. , 2019 ) . Moyano et al . ( 2018 ) provide a detailed survey of this area . Almost all of these ensemble methods require joint training of the base classifiers , but MLaaS APIs are black box to the users . Also , while ensemble methods focus only on improving accuracy , FrugalMCT explicitly considers the cost of each API and enforces a budget constraint . Model cascades : A series of works ( Viola & Jones , 2001a ; b ; Sun et al. , 2013 ; Cai et al. , 2015 ; Wang et al. , 2011 ; Xu et al. , 2014 ; Chen et al. , 2018 ; Kumar et al. , 2018 ; Chen et al. , 2018 ) explores cascades ( a sequence of models ) to balance the quality and runtime of inference . Model cascades use a single predicted quality score to avoid calling computationally expensive models , but FrugalMCT ’ strategies utilize both quality scores and predicted label sets to select an expensive add-on service . AutoML for multi-label classification : AutoML ( Thornton et al. , 2013 ) automates the customization of ML pipelines , including the selection , combination , and parametrization of the learning algorithms . There is a rich literature of AutoML techniques for standard single label tasks , and fewer methods on multi-label predictions ( Wever et al. , 2021 ) ( e.g . genetic algorithms ( de Sá et al. , 2017 ) and a neural network-based search scheme ( Pakrashi & Namee , 2019 ) ) . Applying AutoML to use multiple ML APIs is underexplored , and FrugalMCT can be viewed as the first AutoML approch designed for automating the selection of multiple mutlti-label ML APIs . While most AutoML systems exclusively focus on prediction performance , FrugalMCT optimizes accuracy and cost jointly , which is desirable for cost-sensitive API users . Multiple choice knapsack and integer programming : Many resource allocation problems can be modeled as multiple choice knapsack problem ( MCKP ) ( Pamela H. Vance & Toth ) , 1993 ) , such as keyword bidding ( Zhou & Naroditskiy , 2008 ) and quality of service control ( Lee et al. , 1999 ) . While NP-hard ( Sinha & Zoltners , 1979 ) , various approximations have been proposed for MCKP , such as branch and bound ( Pamela H. Vance & Toth ) , 1993 ) , convex hull relaxation ( Akbar et al. , 2006 ) and bi-objective transformation ( Bednarczuk et al. , 2018 ) . Inherently an integer linear programming ( ILP ) problem , MCKP can also be tackled by ILP solvers , motivated by online adwords searching ( Devanur & Hayes , 2009 ) , resource allocation ( Devanur & Hayes , 2019 ) and general linear programming ( Li et al. , 2020 ) . The service selector of FrugalMCT can be viewed as a MCKP with the same item cost vector per item group , which we leverage to obtain a customized fast and online solver . 2 PRELIMINARIES . Notation . We denote matrices and vectors in bold , and scalars , sets , and functions in standard script . Given a matrix A ∈ Rn×m , we let Ai , j denote its entry at location ( i , j ) . 1 ( · ) represents the indicator function . Multi-label classification Tasks . Throughout this paper , we focus on multi-label classification tasks : assigning a label set Y ⊆ Y to any data point x ∈ X . In contrast to basic supervised learning , in multi-label learning each data point is associated with a set of labels instead of a single label . Many MLaaS APIs target such tasks . Consider , for example , image tagging , where X is a set of images and Y is the set of all tags . Example label sets could be { person , car } or { bag , train , sky } . MLaaS Market . Consider a MLaaS market consisting of K different ML services for some multilabel task . For a data point x , the kth service returns to the user a set of labels with their quality scores , denoted by Yk ( x ) ⊆ Y × [ 0 , 1 ] . For example , one API for multi-label image classification might produce Yk ( x ) = { ( person , 0.8 ) , ( car , 0.7 ) } , indicating the label person with confidence 0.8 and car with confidence 0.7 . Let the vector c ∈ RK denote the unit cost of all services . For example , ck = 0.01 means that users need to pay $ 0.01 every time they call the kth service . 3 FRUGALMCT FRAMEWORK . In this section , we present FrugalMCT , a framework to adaptively select ML APIs for multi-label classification tasks within a budget . All proofs are left to the appendix . We generalize the scheme in Figure 1 ( a ) to K ML services . As shown in Figure 2 , FrugalMCT contains three main components : an accuracy estimator , a service selector , and a label combiner . Given a data point x , it first calls some base service , denoted by base , which is one of the K APIs , and obtains Ybase ( x ) . Often , base is a cheap or free service ; we discuss how to choose base in Section 3.4 . Next , an accuracy predictor produces a vector â ( x ) ∈ [ 0 , 1 ] K , whose kth value estimates the accuracy of the label set produced by the label combiner using base ’ s and kth API ’ s outputs . The service selector s ( · ) : X 7→ [ K ] then decides if and which add-on service needs to be invoked . Finally , a label combiner generates a label set by combining the predictions from the base and add-on APIs . Take Figure 1 ( d ) as an example . The image is first passed to the GitHub model , which produces { ( person , 0.46 ) , ( tennis racket,0.18 ) } , by which the accuracy predictor predicts the accuracy of the label set generated by combining each API ’ s output with GitHub model ’ s . The service selector then decides to further invoke Everypixel , which gives { ( person , 0.46 ) , ( sports ball , 0.52 ) } . Finally , the label combiner uses both APIs ’ output for the final prediction . FrugalMCT allows users to customize the accuracy predictor and the label combiner , depending on the applications . For example , for the image tagging problem , one might use image features ( e.g. , brightness and contrast ) to build the accuracy predictor , while word embeddings can be more useful for named entity recognition . In the following sections , we explain the key of accuracy predictor , API selector and the label combiner in more detail .

Summary: This paper addresses the practical task to use the combination of ML APIs for multi-label classification. Different from the related work FrugalML which ignores the correlation between ML APIs, the proposed FrugalMCT allows selecting and combining the different ML APIs based on a budget. Sufficient theoretical and empirical analyses are provided to demonstrate the effective of FrugalMCT.