Opportunistic Learning: Budgeted Cost-Sensitive Learning from Data Streams

1 INTRODUCTION . In traditional machine learning settings , it is usually assumed that a training dataset is freely available and the objective is to train models that generalize well . In this paradigm , the feature set is fixed , and we are dealing with complete feature vectors accompanied by class labels that are provided for training . However , in many real-world scenarios , there are certain costs for acquiring features as well as budgets limiting the total expenditure . Here , the notation of cost is more general than financial cost and it also refers to other concepts such as computational cost , privacy impacts , energy consumption , patient discomfort in medical tests , and so forth ( Krishnapuram et al. , 2011 ) . Take the example of the disease diagnosis based on medical tests . Creating a complete feature vector from all the relevant information is synonymous with conducting many tests such as MRI scan , blood test , etc . which would not be practical . On the other hand , a physician approaches the problem by asking a set of basic easy-to-acquire features , and then incrementally prescribes other tests based on the current known information ( i.e. , context ) until a reliable diagnosis can be made . Furthermore , in many real-world use-cases , due to the volume of data or necessity of prompt decisions , learning and prediction should take place in an online and stream-based fashion . In the medical diagnosis example , it is consistent with the fact that the latency of diagnosis is vital ( e.g. , urgency of specific cases and diagnosis ) , and it is often impossible to defer the decisions . Here , by online we mean processing samples one at a time as they are being received . Various approaches were suggested in the literature for cost-sensitive feature acquisition . To begin with , traditional feature selection methods suggested to limit the set of features being used for training ( Greiner et al. , 2002 ; Ji & Carin , 2007 ) . For instance , L1 regularization for linear classifiers results in A version of the source code and the health dataset preproccessing code for this paper is available at : https : //github.com/mkachuee/Opportunistic models that effectively use a subset of features ( Efron et al. , 2004 ) . Note that these methods focus on finding a fixed subset of features to be used ( i.e. , feature selection ) , while a more optimal solution would be making feature acquisition decisions based on the sample at hand and at the prediction-time . More recently , probabilistic methods were suggested that measure the value of each feature based on the current evidence ( Chen et al. , 2015 ) . However , these methods are usually applicable to Bayesian networks or similar probabilistic models and make limiting assumptions such as having binary features and binary classes ( Chen et al. , 2014 ) . Furthermore , these probabilistic methods are computationally expensive and intractable in large scale problems ( Chen et al. , 2015 ) . Motivated by the success of discriminative learning , cascade and tree based classifiers suggested as an intuitive way to incorporate feature costs ( Karayev et al. , 2012 ; Chen et al. , 2012 ; Xu et al. , 2012 ; 2014 ) . Nevertheless , these methods are basically limited to the modeling capability of tree classifiers and are limited to fixed predetermined structures . A recent work by Nan & Saligrama ( 2017 ) suggested a gating method that employs adaptive linear or tree-based classifiers , alternating between low-cost models for easy-to-handle instances and higher-cost models to handle more complicated cases . While this method outperforms many of the previous work on the tree-based and cascade cost-sensitive classifiers , the low-cost model being used is limited to simple linear classifiers or pruned random forests . As an alternative approach , sensitivity analysis of trained predictors is suggested to measure the importance of each feature given a context ( Early et al. , 2016a ; Kachuee et al. , 2017 ; 2018 ) . These approaches either require an exhaustive measurement of sensitivities or rely on approximations of sensitivity . These methods are easy to use as they work without any significant modification to the predictor models being trained . However , theoretically , finding the global sensitivity is a difficult and computationally expensive problem . Therefore , frequently , approximate or local sensitivities are being used in these methods which may cause not optimal solutions . Another approach that is suggested in the literature is modeling the feature acquisition problem as a learning problem in the imitation learning ( He et al. , 2012 ) or reinforcement learning ( He et al. , 2016 ; Shim et al. , 2017 ; Janisch et al. , 2017 ) domain . These approaches are promising in terms of performance and scalability . However , the value functions used in these methods are usually not intuitive and require tuning hyper-parameters to balance the cost vs. accuracy trade-off . More specifically , they often rely on one or more hyper-parameters to adjust the average cost at which these models operate . On the other hand , in many real-world scenarios it is desirable to adjust the trade-off at the prediction-time rather than the training-time . For instance , it might be desirable to spend more for a certain instance or continue the feature acquisition until a desired level of prediction confidence is achieved . This paper presents a novel method based on deep Q-networks for cost-sensitive feature acquisition . The proposed solution employs uncertainty analysis in neural network classifiers as a measure for finding the value of each feature given a context . Specifically , we use variations in the certainty of predictions as a reward function to measure the value per unit of the cost given the current context . In contrast to the recent feature acquisition methods that use reinforcement learning ideas ( He et al. , 2016 ; Shim et al. , 2017 ; Janisch et al. , 2017 ) , the suggested reward function does not require any hyper-parameter tuning to balance cost versus performance trade-off . Here , features are acquired incrementally , while maintaining a certain budget or a stopping criterion . Moreover , in contrast to many other work in the literature that assume an initial complete dataset ( He et al. , 2012 ; Kusner et al. , 2014 ; Chen et al. , 2015 ; Early et al. , 2016b ; Nan & Saligrama , 2017 ) , the proposed solution is stream-based and online which learns and optimizes acquisition costs during the training and the prediction . This might be beneficial as , in many real-world use cases , it might be prohibitively expensive to collect all features for all training data . Furthermore , this paper suggests a method for sharing the representations between the class predictor and action-value models that increases the training efficiency . 2 PRELIMINARIES . 2.1 PROBLEM SETTINGS . In this paper , we consider the general scenario of having a stream of samples as input ( Si ) . Each sample Si corresponds to a data point of a certain class in Rd , where there is a cost for acquiring each feature ( cj ; 1 ≤ j ≤ d ) . For each sample , initially , we do not know the value of any feature . Subsequently , at each time step t , we only have access to a partial realization of the feature vector denoted by xti that consists of features that are acquired so far . There is a maximum feature acquisition budget ( B ) that is available for each sample . Note that the acquisition may also be terminated before reaching the maximum budget , based on any other termination condition such as reaching a certain prediction confidence . Furthermore , for each Si , there is a ground truth target label ỹi . It is also worth noting that we consider the online stream processing task in which acquiring features is only possible for the current sample being processed . In other words , any decision should take place in an online fashion . In this setting , the goal of an Opportunistic Learning ( OL ) solution is to make accurate predictions for each sample by acquiring as many features as necessary . At the same time , learning should take place by updating the model while maintaining the budgets . Please note that , in this setup , we are assuming that the feature acquisition algorithm is processing a stream of input samples and there are no distinct training or test samples . However , we assume that ground truth labels are only available to us after the prediction and for a subset of samples . More formally , we define a mask vector kti ∈ { 0 , 1 } d where each element of k indicates if the corresponding feature is available in xti . Using this notation , the total feature acquisition cost at each time step can be represented as Cttotal , i = ( k t i − k0i ) T c . ( 1 ) Furthermore , we define the feature query operator ( q ) as xt+1i = q ( x t i , j ) , where k t+1 i , j − k t i , j = 1 . ( 2 ) In Section 3 , we use these primitive operations and notations for presenting the suggested solution . 2.2 PREDICTION CERTAINTY . As prediction certainty is used extensively throughout this paper , we devote this section to certainty measurement . The softmax output layer in neural networks are traditionally used as a measure of prediction certainty . However , interpreting softmax values as probabilities is an ad hoc approach prone to errors and inaccurate certainty estimates ( Szegedy et al. , 2013 ) . In order to mitigate this issue , we follow the idea of Bayesian neural networks and Monte Carlo dropout ( MC dropout ) ( Williams , 1997 ; Gal & Ghahramani , 2016 ) . Here we consider the distribution of model parameters at each layer l in an L layer neural network as : ω̂l ∼ p ( ωl ) , where 1 ≤ l ≤ L , ( 3 ) where ω̂l is a realization of layer parameters from the probability distribution of p ( ωl ) . In this setting , a probability estimate conditioned on the input and stochastic model parameters is represented as : p ( y|x , ω̂ ) = softmax ( f ω̂D ( x ) ) , ( 4 ) where f ω̂D is the output activation of a neural network with parameters ω̂ trained on dataset D. In order to find the uncertainty of final predictions with respect to inputs , we integrate equation 4 with respect to ω : p ( y|x , D ) = ∫ p ( y|x , ω ) p ( ω|D ) dω . ( 5 ) Finally , MC dropout suggests interpreting the dropout forward path evaluations as Monte Carlo samples ( ωt ) from the ω distribution and approximating the prediction probability as : p ( y|x , D ) = 1 T T∑ t=1 p ( y|x , ωt ) . ( 6 ) With reasonable dropout probability and number of samples , the MC dropout estimate can be considered as an accurate estimate of the prediction uncertainty . Readers are referred to Gal & Ghahramani ( 2016 ) for a more detailed discussion . In this paper , we denote the certainty of prediction for a given sample ( Cert ( xti ) ) as a vector providing the probability of the sample belonging to each class in equation 6 .

Summary: The paper presents a RL approach for sequential feature acquisition in a budgeted learning setting, where each feature comes at some cost and the goal is to find a good trade-off between accuracy and cost. Starting with zero feature, the model sequentially acquires new features to update its prediction and stops when the budget is exhausted. The feature selection policy is learned by deep Q-learning. The authors have shown improvements over several prior approaches in terms of accuracy-cost trade-off on three datasets, including a real-world health dataset with real feature costs.