Leveraging Simple Model Predictions for Enhancing its Performance

1 INTRODUCTION . Simple models such as decision trees or rule lists or shallow neural networks still find use in multiple settings where a ) ( global ) interpretability is needed , b ) small data sizes are available , or c ) memory/computational constraints are prevalent ( Dhurandhar et al. , 2018b ) . In such settings compact or understandable models are often preferred over high performing complex models , where the combination of a human with an interpretable model can have better on-field performance than simply using the best performing black box model ( Varshney et al. , 2018 ) . For example , a manufacturing engineer with an interpretable model may be able to obtain precise knowledge of how an out-of-spec product was produced and can potentially go back to fix the process as opposed to having little-to-no knowledge of how the decision was reached . Posthoc local explainability methods ( Ribeiro et al. , 2016 ; Bach et al. , 2015 ; Dhurandhar et al. , 2018a ) can help delve into the local behavior of black box models , however , besides the explanations being only local , there is no guarantee that they are in fact true ( Rudin , 2018 ) . There is also a growing concern of the carbon footprint left behind in training complex deep models ( Strubell et al. , 2019 ) , which for some popular architectures is more than that left behind by a car over its entire lifetime . In this paper , we propose a method , SRatio , which reweights the training set to improve simple models given access to a highly accurate complex model such as a deep neural network , boosted trees , or some other predictive model . Given the applications we are interested in , such as interpretability or deployment of models in resource limited settings , we assume the complexity of the simple models to be predetermined or fixed ( viz . decision tree of height ≤ 5 ) . We can not grow arbitrary size ensembles such as in boosting or bagging ( Freund & Schapire , 1997 ) . Our method applies potentially to any complex-simple model combination which is not the case for some state-of-the-art methods in this space such as Knowledge Distillation ( Geoffrey Hinton , 2015 ) or Profweight ( Dhurandhar et al. , 2018b ) , where the complex model is assumed to be a deep neural network . In addition , we generalize and formalize the concept of probes presented in ( Dhurandhar et al. , 2018b ) and provide examples of what they would correspond to for classifiers other than neural networks . Our method also uses the a priori low performing simple model ’ s confidences to enhance its performance . We believe this to be conceptually novel compared to existing methods which seem to only leverage the complex model ( viz . its predictions/confidences ) . The benefit is seen in experiments where we outperform other competitors in a majority of the cases and are tied with one or more methods for best performance in the remaining cases . In fact , in a couple of cases we even approach the complex model ’ s performance , i.e . a single tree is made to be as accurate as 100 boosted trees . Moreover , we motivate our approach by contrasting it with covariate shift and show that our weighting scheme where we now minimize the weighted loss of the simple model is equivalent to minimizing an upper bound on the loss of the complex model . 2 RELATED WORK . Knowledge Distillation ( Geoffrey Hinton , 2015 ; Tan et al. , 2017 ; Lopez-Paz et al. , 2016 ) is one of the most popular approaches for building `` simpler '' neural networks . It typically involves minimizing the cross-entropy loss of a simpler network based on calibrated confidences ( Guo et al. , 2017 ) of a more complex network . The simpler networks are usually not that simple in that they are typically of the same ( or similar ) depth but thinned down ( Romero et al. , 2015 ) . This is generally insufficient to meet tight resource constraints ( Reagen et al. , 2016 ) . Moreover , the thinning down was shown for convolutional neural networks but it is unclear how one would do the same for modern architectures such as ResNets . The weighting of training inputs approach on the other hand can be more easily applied to different architectures . It also has another advantage in that it can be readily applied to models optimizing losses other than cross-entropy ( viz . hinge loss , squared loss ) with some interpretation of which inputs are more ( or less ) important . Some other strategies to improve simple models ( Buciluǎ et al. , 2006 ; Ba & Caurana , 2013 ; Bastani et al. , 2017 ) are also conceptually similar to Distillation , where the actual outputs are replaced by predictions from the complex model . In Dehghani et al . ( 2017 ) , authors use soft targets and their uncertainty estimates to inform a student model on a larger dataset with more noisy labels . Uncertainty estimates are obtained from Gaussian Process Regression done on a dataset that has less noisy labels . In Furlanello et al . ( 2018 ) , authors train a student neural network that is identically parameterized to the original one by fitting to soft scores rescaled by temperature . In our problem , the complexity of the student is very different from that of the teacher and we do compare with distillation-like schemes . Frosst & Hinton ( 2017 ) define a new class of decision trees called soft decision trees to enable it to fit soft targets ( classic distillation ) of a neural network . Our methods use existing training algorithms for well-known simple models . Ren et al . ( 2018 ) advocate reweighting samples as a way to make deep learning robust by tuning the weights on a validation set through gradient descent . Our problem is about using knowledge from a pre-trained complex model to improve a simple model through weighting samples . The most relevant work to our current endeavor is ProfWeight ( Dhurandhar et al. , 2018b ) , where they too weight the training inputs . Their method however , requires the complex model to be a neural network and thus does not apply to settings where we have a different complex model . Moreover , their method , like Distillation , takes into account only the complex model ’ s assessment of an example ’ s difficulty . Curriculum learning ( CL ) ( Bengio et al. , 2009 ) and boosting ( Freund & Schapire , 1997 ) are two other approaches which rely on weighting samples , however , their motivation and setup are significantly different . In both CL and boosting the complexity of the improved learner can increase as they do not have to respect constraints such as interpretability ( Dhurandhar et al. , 2017 ; Montavon et al. , 2017 ) or limited memory/power ( Reagen et al. , 2016 ; Chen et al. , 2016 ) . In CL , typically , there is no automatic gradation of example difficulty during training . In boosting , the examples are graded with respect to a previous weak learner and not an independent accurate complex model . Also , as we later show , our method does not necessarily up-weight hard examples but rather uses a measure that takes into account hardness as assessed by both the complex and simple models . 3 METHODOLOGY . In this section , we first provide theoretical and intuitive justification for our approach . This is followed by a description of our method for improving simple models using both the complex and simple model ’ s predictions . The key novelty lies in the use of the simple model ’ s prediction , which also makes the theory non-trivial yet practical . Rather than use the complex model ’ s prediction , we generalize a concept from ( Dhurandhar et al. , 2018b ) where they attached probe functions to each layer of a neural network , obtained predictions using only the first k layers ( k being varied up to the total number of layers ) , and used the mean of the probe predictions rather than the output of only the last layer . They empirically showed the extra information from previous layers to improve upon only using the final layer ’ s output . Our generalization , which we call graded classifiers and formally define below , extracts progressive information from other models ( beyond neural networks ) . The graded classifiers provide better performance than using only the output of complex model , as illustrated by our various experiments in the subsequent section . 3.1 THEORETICAL MOTIVATION . Our approach in section 3.3 can be motivated by contrasting it with the covariate shift ( Agarwal et al. , 2011 ) setting . If X × Y is the input-output space and p ( x , y ) and q ( x , y ) are the source and target distributions in the covariate shift setting , then it is assumed that p ( y|x ) = q ( y|x ) but p ( x ) 6= q ( x ) . One of the standard solutions for such settings is importance sampling where the source data is sampled proportional to q ( x ) p ( x ) in order to mimic as closely as possible the target distribution . In our case , the dataset is the same but the classifiers ( i.e . complex and simple ) are different . We can think of this as a setting where p ( x ) = q ( x ) as both the models learn from the same data , however , p ( y|x ) 6= q ( y|x ) where p ( y|x ) and q ( y|x ) correspond to the outputs of complex and simple classifiers , respectively . Given that we want the simple model to approach the complex models performance , a natural analog to the importance weights used in covariate shift is to weight samples by p ( y|x ) q ( y|x ) which is the essence of our method as described below in section 3.3 . Now , let us formally show that the expected cross-entropy loss of a model is no greater than the reweighted version with an additional positive slack term . This implies that training the simple model with this reweighting is a valid and sound procedure of the loss we want to optimize . Lemma 3.1 . Let pθ ( y|x ) be the softmax scores on a specific model θ from simple model space Θ . Let θ∗ ∈ Θ be the set of simple model parameters that is obtained from a given learning algorithm for the simple model on a training dataset . Let pc ( y|x ) be a pre-trained complex classifier whose loss is smaller than θ∗ on the training distribution . Let β ≥ 1 be a scalar clip level for the ratio pc ( y|x ) /pθ∗ ( y|x ) . Then we have : E [ − log pθ ( y|x ) ] ≤ E [ max ( 1 , min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) log ( 1 pθ ( y|x ) ) ] − E [ log ( min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ] + log ( β ) . ( 1 ) Proof . E [ − log ( pθ ( y|x ) ) ] = E [ log ( 1 pθ ( y|x ) ·min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ] − E [ log ( min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ] ≤ E [ log ( 1 pθ ( y|x ) ·max ( 1 , min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ) ] − E [ log ( min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ] a ≤ E [ max ( 1 , min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) log ( 1 pθ ( y|x ) ) ] − E [ log ( min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) ] + log ( β ) ( 2 ) ( a ) The inequality log ( wx ) ≤ w log ( x ) + log ( β ) holds for all β ≥ w ≥ 1 , x > 1 where β ≥ 1 is any arbitrary clip level . Remark 1 : We observe that re-weighing every sample by max ( 1 , min ( pc ( y|x ) pθ∗ ( y|x ) , β ) ) and reoptimizing using the simple model training algorithm is a sound way to optimize the cross-entropy loss of the simple model on the training data set . The reason we believe that optimizing the upper bound could be better is because many simple models such as decision trees are trained using a simple greedy approach . Therefore , reweighting samples based on an accurate complex model could induce the appropriate bias leading to better solutions . Moreover , in equation 2 , the second inequality is the main place where there is slack between the upper bound and the quantity we are interested in bounding . This inequality exhibits a smaller slack if w = pc ( y|x ) pθ∗ ( y|x ) is not much smaller than 1 with high probability . This is typical when pc comes from a more complex model that is more accurate than that of θ∗ . Remark 2 : The upper bound used for the last inequality in the proof leads to a quantification of the bias introduced by weighting for a particular dataset . Note that in practice , we determine the optimal β via cross-validation .

Summary: The paper proposes a means of improving the predictions of a "simple" (low-capacity) model. The idea is to take the predictions of a "complex" (high-capacity) model, and weight the loss in the simple model based on the ratio of complex to simple models' predictions. Intuitively, this seeks to focus on instances which the complex model can fit, but the unweighted simple model cannot. Experiments show the proposed method to have some benefits over existing approaches.