Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems

1 Introduction . Progress in machine learning has allowed us to develop increasingly sophisticated artificially intelligent systems , including self-driving cars , virtual assistants , and complex robots . These systems generally contain a large number of machine-learning models that are trained to perform modular tasks such as recognizing image or video content , transcribing speech , analyzing natural language , eliciting user preferences , ranking options to be presented to a user , etc . The models feed each other information : for example , a pedestrian detector may use the output of a depth-estimation model as input . Indeed , machine-learning systems can be interpreted as directed acyclic graphs in which each vertex corresponds to a model , and models feed each other information over the edges in the graph . In practice , various models in this graph are often developed and trained independently [ 22 ] . This modularization tends to lead to more usable APIs but may also be motivated by other practical constraints . For example , some of the models in the system may be developed by different teams ( some models in the system may be developed by a cloud provider ) ; models may be retrained and deployed at a very different cadence ( personalization models may be updated very regularly but image-recognition models may not ) ; new downstream models may be added continuously in the graph ( user-specific models may need to be created every time a user signs up for a service ) ; or models may be non-differentiable ( gradientboosted decision trees are commonly used for selection of discrete features ) . This can make backpropagation through the models in the graph infeasible or highly undesirable in many practical settings . The scenario described above leads to an obvious potential concern : Can improving a machinelearning model make the machine-learning system worse ? We answer this question affirmatively by showing how improvements in an upstream model can deteriorate the performance of downstream models , even if all the downstream models are retrained after updating the upstream model . Such ∗Work performed during internship at Facebook AI Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . self-defeating improvements are caused by entanglement between the models in the system . To better understand this phenomenon , we perform an error decomposition of simple machine-learning systems . Our decomposition sheds light on the different types of errors that can lead to selfdefeating improvements : we provide illustrations of each error type . We also show that self-defeating improvements arise in a realistic system that detects cars and pedestrians in stereo images . Our study opens up a plethora of new research questions that , to the best of our knowledge , have not yet been studied in-depth in the machine-learning community . We hope that our study will encourage the community to move beyond the study of machine-learning models in isolation , and to study machine-learning systems more holistically instead . 2 Problem Setting . We model a machine-learning system as a static directed acyclic graph ( DAG ) of machine-learning models , G = ( V , E ) , where each vertex v ∈ V corresponds to a machine-learning model and each edge ( v , w ) ∈ E represents the output of model v being used as input into model w. For example , vertex v could be a depth-estimation model and vertex w a pedestrian-detection model that operates on the depth estimates produced by vertex v. A model is upstream of v if it is an ancestor of v. A downstream model of v is a descendant of v. An example of a machine-learning system with three models is shown in Figure 1 ( we study this model in Section 3.2 ) . Each vertex in the machine-learning system G corresponds to a model function , v ( · ) , that operates on both data and outputs from its upstream models . Denote the output from model v by fv ( x ) . The output of a source model v is fv ( x ) = v ( x ( v ) ) , where x ( v ) represents the part of input x that is relevant to model v. The output of a non-source model w is given by fw ( x ) = w ( [ x ( w ) , fv ( x ) : v ∈ Pw ] ) , where Pw = { v : ( v , w ) ∈ E } are the parents of w. Training . To train model v , we assume access to a training set Dv = { ( x1 , y1 ) , . . . , ( xNv , yNv ) } with Nv examples xn ∈ X and corresponding targets yn ∈ Yv . For each model , we also assume a training algorithm Av ( Dv ) that selects a model v ∈ Hv from hypothesis setHv based on the training set . The learning algorithm , Av , the hypothesis set , Hv , and the training set , Dv , are fixed during training but they may change each time model v is re-trained or updated . The data space X , the target space Yv , and the way in which inputs x ( v ) are obtained do not change between model updates . We assume that models in G may be trained separately rather than jointly , that is , the learning signal obtained from training a downstream model , v , may not be backpropagated into its upstream dependencies , w ∈ Pv . This assumption is in line with constraints that commonly arise in real-world machine learning systems ; see our motivation below . Note that this assumption also implies that a downstream model can not be trained before all its upstream models are trained . Evaluation . To be able to evaluate the performance of the task at v , we assume access to a fixed test set D̄v that is defined analogously to the training set : it contains Mv test examples x̄m ∈ X and corresponding targets ȳm ∈ Yv . In contrast to the training set , the test set is fixed and does not change when the model is updated . However , we note that the input distribution into a non-root model is not only governed by the test set D̄v but also by upstream models : if an upstream model changes , the inputs into a downstream model may change as well even though the test set is fixed . We also assume access to a test loss function ` v ( v , D̄v ) for each model v , for example , classification error ( we assume lower is better ) . Akin to the test set D̄v , all test loss functions ` v are fixed . Hence , we always measure the generalization ability of a model , v , in the exact same way . The test loss function may not have trainable parameters , that is , it must be a “ true ” loss function . Updating models . The machine-learning system we defined can be improved by updating a model . In such an update , an individual model v is replaced by an alternative model v′ . We can determine if such a model update constitutes an improvement by evaluating whether or not its test loss decreases : ` v ( v ′ , D̄v ) ≤ ` v ( v , D̄v ) . When v is not a leaf in G , the update to v′ may affect the test loss of downstream models as well . In practice , the improvement due to v′ often also improves downstream models ( e.g. , [ 19 ] ) , but this is not guaranteed . We define a self-defeating improvement as a situation in which replacing v by v′ improves that model but deteriorates at least one of the downstream models w , even after all the downstream models are updated ( i.e. , re-trained using the same learning algorithm ) to account for the changes incurred by v′ . Definition 2.1 ( Self-defeating improvement ) . Denoting the set of all downstream models ( descendants ) of v in G as Cv , a self-defeating improvement arises when : ∃w ∈ Cv : ` v ( v′ , D̄v ) ≤ ` v ( v , D̄v ) 6=⇒ ` w ( w′ , D̄w ) ≤ ` w ( w , D̄w ) ) , ( 1 ) where w′ represents a new version of model w that was trained after model v was updated to v′ to account for the input distribution change that the update of v incurs on w. Motivation of problem setting . In the problem setting described above , we have made two assumptions : ( 1 ) models may be ( re- ) trained separately rather than jointly and ( 2 ) the training algorithm , hypothesis set , and training data may change between updates of the model.2 Both assumptions are motivated by the practice of developing large machine-learning systems that may comprise thousands of models [ 22 , 28 ] . Joint model training is often infeasible in such systems because : • Some of the models may have been developed by cloud providers [ 34 ] and can not be changed . • Backpropagation may be technically infeasible or inefficient , e.g. , when models are implemented in incompatible learning frameworks or when their training data live in different data centers . • Some models may be re-trained or updated more often than others . For example , personalization models may be updated every few hours to adapt to changes in the data distribution , but re-training large language models [ 8 ] at that same cadence is not very practical . • Upstream models may have hundreds or even thousands of downstream dependencies , complicating appropriate weighting of the downstream learning signals that multi-task learning [ 10 ] requires . • Some models may be non-differentiable , e.g. , gradient-boosted decision trees are commonly used for feature selection in large sets of features . We assume that changes in the training algorithm , hypothesis set , and training data may occur because model owners ( for example , cloud providers ) constantly seek to train and deploy improved versions of their models with the ultimate goal of improving the system ( s ) in which the models are used . 3 Understanding Self-Defeating Improvements via Error Decompositions . Traditional software design relies heavily on modules : independent and interchangeable components that are combined to form complex software systems [ 7 , §7.4 ] . Well-designed modules provide an explicit specification of their inputs , outputs , and functionality . In traditional software engineering , good modularization allows developers to modify modules independently because the effects of those modifications on the rest of the system are relatively predictable . Machine-learned models do not resemble traditional software modules because they lack an explicit specification of their functionality [ 14 ] . As a result , there is often a significant degree of entanglement between different models in the system [ 22 ] . This makes the system-wide effects of model modifications unpredictable , and can lead to situations where improving a model does not improve the entire system . To understand such self-defeating improvements , we study Bayes error decompositions of simple machine-learning systems . Specifically , we study a system with two models in Section 3.1 and a system with three models in Section 3.2 . 3.1 Error Decomposition of System with Two Models . We adapt standard Bayes error decomposition [ 6 , 33 ] to study a simple system that has two vertices , V = { v , w } , and a dependency between upstream model v and downstream model w , that is , E = { ( v , w ) } . We denote the Bayes-optimal downstream model by w∗ ; the optimal downstream 2The training data is fixed during training , i.e. , we do not consider online learning settings in our setting . model conditioned on an upstream model by w† ; and the optimal upstream-conditional downstream model in the hypothesis setHw by w†Hw . Using these definitions , we can decompose the downstream risk of model w , given upstream model v , as follows : E [ ` w ( w ◦ v ) − ` w ( w∗ ) ] = E [ ` w ( w† ◦ v ) − ` w ( w∗ ) ] ︸ ︷︷ ︸ upstream error +E [ ` w ( w†Hw ◦ v ) − ` w ( w † ◦ v ) ] ︸ ︷︷ ︸ downstream approximation error +E [ ` w ( w ◦ v ) − ` w ( w†Hw ◦ v ) ] ︸ ︷︷ ︸ downstream estimation error , ( 2 ) where ◦ denotes function composition , and the expectations are under the data distribution . The error decomposition is similar to standard decompositions [ 6 , 33 ] but contains three terms instead of two.3 Specifically , the upstream error does not arise in prior error decompositions , and the downstream approximation and downstream estimation errors differ from the standard decomposition . Barring variance in the error estimate due to the finite size of test sets D̄w ( which is an issue that can arise in any model-selection problem ) , a self-defeating improvement occurs when a model update from ( v , w ) to ( v′ , w′ ) reduces the upstream risk but increases the downstream risk . An increase in the downstream risk implies that at least one of the three errors terms in the composition above must have increased . We describe how each of these terms may increase due to a model update : • Upstream error measures the error that is due to the upstream model v not being part of the Bayesoptimal solution w∗ . It increases when an improvement in the upstream loss does not translate in a reduction in Bayes error of the downstream model , i.e. , when : ` w ( w† ◦ v′ ) > ` w ( w† ◦ v ) . An upstream error increase can happen due the loss mismatch : a setting in which the test loss functions , ` , of the upstream and downstream model optimize for different things . For example , the upstream test loss function may not penalize errors that need to be avoided in order for the downstream test loss to be minimized , or it may penalize errors that are irrelevant to the downstream model . Alternatively , the upstream error may increase due to distribution mismatch : situations in which the upstream loss focuses on parts of the data-target space X × Yv that are unimportant for the downstream model ( and vice versa ) . For example , suppose the upstream model v is an image-recognition model that aims to learn a feature representation that separates cats from dogs , and fv ( x ) is a feature representation obtained from that model . A downstream model , w , that distinguishes different dog breeds based on fv may deteriorate when the improvement of model v collapses all representations of dog images in fv . Examples of this were observed in , e.g. , [ 19 ] . • Downstream approximation error measures the error due to the optimal w† not being in the hypothesis setHw . The approximation error increases when the downstream model , w , is unable to exploit improvements in the upstream model , v , because exploiting those improvements would require selection of a model that is not in that hypothesis set . 3For brevity , we do not include optimization error of [ 5 ] in our Bayes error decomposition . Figure 2 illustrates how this situation can lead to a self-defeating improvement . In the illustration , the upstream model , v , predicts a 2D position for each data point . The downstream model , w , separates examples into the positive ( green color ) and negative ( red color ) class based on the prediction of v using a linear classifier . The predictions of the improved upstream model , v′ , better match the ground-truth targets : the predictions of v′ ( right plot ) match the ground-truth targets ( left plot ) more closely than those of v ( middle plot ) . However , the re-trained downstream model , w′ , deteriorates because the resulting classification problem is more non-linear : the downstream model can not capitalize on the upstream improvement because it is linear . The resulting increase in approximation error leads to the self-defeating improvement in Figure 2 . • Downstream estimation error measures the error due to the model training being performed on a finite data sample with an imperfect optimizer , which makes finding w†Hw difficult in practice . The estimation increases , for example , when the upstream model improves but the downstream model requires more than Nw training samples to capitalize on this improvement . Figure 3 shows an example of a self-defeating improvement caused by an increase of the downstream estimation error . As before , the upstream model , v , predicts a 2D position for each data point in the example . The downstream model w is selected from the set , Hw , of quadratic classifiers . It is tasked with performing binary classification into a positive class ( green color ) and negative class ( red color ) based on the 2D positions predicted by the upstream model . To train the binary classifier , the downstream model w ( and w′ ) is provided with Nw = 6 labeled training examples ( the green and red markers in the plot ) . In the example , the upstream model v′ performs better than its original counterpart v : the predictions of v′ ( right plot ) match the ground-truth targets ( left plot ) more closely than those of v ( middle plot ) . However , the upstream improvement hampers the downstream model even though the optimal downstream model w† is in the hypothesis set Hw both before and after the upstream improvement . After the upstream improvement , the optimal downstream model that can be selected from the hypothesis set , w†Hw , is more complex , which makes it harder to find it based on the finite number of Nw training examples . This results in an increase in the estimation error , which causes the self-defeating improvement in Figure 3 .

Summary: This paper studies the problem of "self-defeating improvement" (improving an upstream ML model deteriorates a downstream ML model in the same system). Leveraging Bayes error decomposition, it decomposes the error of a system with multiple ML models into: (excess) upstream error, upstream compatibility error (when more than one upstream model is used), downstream approximation error, and downstream estimation error. The paper conducts a case study in Pseudo-LiDAR for vehicle/pedestrian detection to demonstrate the existence of the studied "self-defeating improvement" in real world.