Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems

1 Introduction . Progress in machine learning has allowed us to develop increasingly sophisticated artificially intelligent systems , including self-driving cars , virtual assistants , and complex robots . These systems generally contain a large number of machine-learning models that are trained to perform modular tasks such as recognizing image or video content , transcribing speech , analyzing natural language , eliciting user preferences , ranking options to be presented to a user , etc . The models feed each other information : for example , a pedestrian detector may use the output of a depth-estimation model as input . Indeed , machine-learning systems can be interpreted as directed acyclic graphs in which each vertex corresponds to a model , and models feed each other information over the edges in the graph . In practice , various models in this graph are often developed and trained independently [ 22 ] . This modularization tends to lead to more usable APIs but may also be motivated by other practical constraints . For example , some of the models in the system may be developed by different teams ( some models in the system may be developed by a cloud provider ) ; models may be retrained and deployed at a very different cadence ( personalization models may be updated very regularly but image-recognition models may not ) ; new downstream models may be added continuously in the graph ( user-specific models may need to be created every time a user signs up for a service ) ; or models may be non-differentiable ( gradientboosted decision trees are commonly used for selection of discrete features ) . This can make backpropagation through the models in the graph infeasible or highly undesirable in many practical settings . The scenario described above leads to an obvious potential concern : Can improving a machinelearning model make the machine-learning system worse ? We answer this question affirmatively by showing how improvements in an upstream model can deteriorate the performance of downstream models , even if all the downstream models are retrained after updating the upstream model . Such ∗Work performed during internship at Facebook AI Research . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . self-defeating improvements are caused by entanglement between the models in the system . To better understand this phenomenon , we perform an error decomposition of simple machine-learning systems . Our decomposition sheds light on the different types of errors that can lead to selfdefeating improvements : we provide illustrations of each error type . We also show that self-defeating improvements arise in a realistic system that detects cars and pedestrians in stereo images . Our study opens up a plethora of new research questions that , to the best of our knowledge , have not yet been studied in-depth in the machine-learning community . We hope that our study will encourage the community to move beyond the study of machine-learning models in isolation , and to study machine-learning systems more holistically instead . 2 Problem Setting . We model a machine-learning system as a static directed acyclic graph ( DAG ) of machine-learning models , G = ( V , E ) , where each vertex v ∈ V corresponds to a machine-learning model and each edge ( v , w ) ∈ E represents the output of model v being used as input into model w. For example , vertex v could be a depth-estimation model and vertex w a pedestrian-detection model that operates on the depth estimates produced by vertex v. A model is upstream of v if it is an ancestor of v. A downstream model of v is a descendant of v. An example of a machine-learning system with three models is shown in Figure 1 ( we study this model in Section 3.2 ) . Each vertex in the machine-learning system G corresponds to a model function , v ( · ) , that operates on both data and outputs from its upstream models . Denote the output from model v by fv ( x ) . The output of a source model v is fv ( x ) = v ( x ( v ) ) , where x ( v ) represents the part of input x that is relevant to model v. The output of a non-source model w is given by fw ( x ) = w ( [ x ( w ) , fv ( x ) : v ∈ Pw ] ) , where Pw = { v : ( v , w ) ∈ E } are the parents of w. Training . To train model v , we assume access to a training set Dv = { ( x1 , y1 ) , . . . , ( xNv , yNv ) } with Nv examples xn ∈ X and corresponding targets yn ∈ Yv . For each model , we also assume a training algorithm Av ( Dv ) that selects a model v ∈ Hv from hypothesis setHv based on the training set . The learning algorithm , Av , the hypothesis set , Hv , and the training set , Dv , are fixed during training but they may change each time model v is re-trained or updated . The data space X , the target space Yv , and the way in which inputs x ( v ) are obtained do not change between model updates . We assume that models in G may be trained separately rather than jointly , that is , the learning signal obtained from training a downstream model , v , may not be backpropagated into its upstream dependencies , w ∈ Pv . This assumption is in line with constraints that commonly arise in real-world machine learning systems ; see our motivation below . Note that this assumption also implies that a downstream model can not be trained before all its upstream models are trained . Evaluation . To be able to evaluate the performance of the task at v , we assume access to a fixed test set D̄v that is defined analogously to the training set : it contains Mv test examples x̄m ∈ X and corresponding targets ȳm ∈ Yv . In contrast to the training set , the test set is fixed and does not change when the model is updated . However , we note that the input distribution into a non-root model is not only governed by the test set D̄v but also by upstream models : if an upstream model changes , the inputs into a downstream model may change as well even though the test set is fixed . We also assume access to a test loss function ` v ( v , D̄v ) for each model v , for example , classification error ( we assume lower is better ) . Akin to the test set D̄v , all test loss functions ` v are fixed . Hence , we always measure the generalization ability of a model , v , in the exact same way . The test loss function may not have trainable parameters , that is , it must be a “ true ” loss function . Updating models . The machine-learning system we defined can be improved by updating a model . In such an update , an individual model v is replaced by an alternative model v′ . We can determine if such a model update constitutes an improvement by evaluating whether or not its test loss decreases : ` v ( v ′ , D̄v ) ≤ ` v ( v , D̄v ) . When v is not a leaf in G , the update to v′ may affect the test loss of downstream models as well . In practice , the improvement due to v′ often also improves downstream models ( e.g. , [ 19 ] ) , but this is not guaranteed . We define a self-defeating improvement as a situation in which replacing v by v′ improves that model but deteriorates at least one of the downstream models w , even after all the downstream models are updated ( i.e. , re-trained using the same learning algorithm ) to account for the changes incurred by v′ . Definition 2.1 ( Self-defeating improvement ) . Denoting the set of all downstream models ( descendants ) of v in G as Cv , a self-defeating improvement arises when : ∃w ∈ Cv : ` v ( v′ , D̄v ) ≤ ` v ( v , D̄v ) 6=⇒ ` w ( w′ , D̄w ) ≤ ` w ( w , D̄w ) ) , ( 1 ) where w′ represents a new version of model w that was trained after model v was updated to v′ to account for the input distribution change that the update of v incurs on w. Motivation of problem setting . In the problem setting described above , we have made two assumptions : ( 1 ) models may be ( re- ) trained separately rather than jointly and ( 2 ) the training algorithm , hypothesis set , and training data may change between updates of the model.2 Both assumptions are motivated by the practice of developing large machine-learning systems that may comprise thousands of models [ 22 , 28 ] . Joint model training is often infeasible in such systems because : • Some of the models may have been developed by cloud providers [ 34 ] and can not be changed . • Backpropagation may be technically infeasible or inefficient , e.g. , when models are implemented in incompatible learning frameworks or when their training data live in different data centers . • Some models may be re-trained or updated more often than others . For example , personalization models may be updated every few hours to adapt to changes in the data distribution , but re-training large language models [ 8 ] at that same cadence is not very practical . • Upstream models may have hundreds or even thousands of downstream dependencies , complicating appropriate weighting of the downstream learning signals that multi-task learning [ 10 ] requires . • Some models may be non-differentiable , e.g. , gradient-boosted decision trees are commonly used for feature selection in large sets of features . We assume that changes in the training algorithm , hypothesis set , and training data may occur because model owners ( for example , cloud providers ) constantly seek to train and deploy improved versions of their models with the ultimate goal of improving the system ( s ) in which the models are used . 3 Understanding Self-Defeating Improvements via Error Decompositions . Traditional software design relies heavily on modules : independent and interchangeable components that are combined to form complex software systems [ 7 , §7.4 ] . Well-designed modules provide an explicit specification of their inputs , outputs , and functionality . In traditional software engineering , good modularization allows developers to modify modules independently because the effects of those modifications on the rest of the system are relatively predictable . Machine-learned models do not resemble traditional software modules because they lack an explicit specification of their functionality [ 14 ] . As a result , there is often a significant degree of entanglement between different models in the system [ 22 ] . This makes the system-wide effects of model modifications unpredictable , and can lead to situations where improving a model does not improve the entire system . To understand such self-defeating improvements , we study Bayes error decompositions of simple machine-learning systems . Specifically , we study a system with two models in Section 3.1 and a system with three models in Section 3.2 . 3.1 Error Decomposition of System with Two Models . We adapt standard Bayes error decomposition [ 6 , 33 ] to study a simple system that has two vertices , V = { v , w } , and a dependency between upstream model v and downstream model w , that is , E = { ( v , w ) } . We denote the Bayes-optimal downstream model by w∗ ; the optimal downstream 2The training data is fixed during training , i.e. , we do not consider online learning settings in our setting . model conditioned on an upstream model by w† ; and the optimal upstream-conditional downstream model in the hypothesis setHw by w†Hw . Using these definitions , we can decompose the downstream risk of model w , given upstream model v , as follows : E [ ` w ( w ◦ v ) − ` w ( w∗ ) ] = E [ ` w ( w† ◦ v ) − ` w ( w∗ ) ] ︸ ︷︷ ︸ upstream error +E [ ` w ( w†Hw ◦ v ) − ` w ( w † ◦ v ) ] ︸ ︷︷ ︸ downstream approximation error +E [ ` w ( w ◦ v ) − ` w ( w†Hw ◦ v ) ] ︸ ︷︷ ︸ downstream estimation error , ( 2 ) where ◦ denotes function composition , and the expectations are under the data distribution . The error decomposition is similar to standard decompositions [ 6 , 33 ] but contains three terms instead of two.3 Specifically , the upstream error does not arise in prior error decompositions , and the downstream approximation and downstream estimation errors differ from the standard decomposition . Barring variance in the error estimate due to the finite size of test sets D̄w ( which is an issue that can arise in any model-selection problem ) , a self-defeating improvement occurs when a model update from ( v , w ) to ( v′ , w′ ) reduces the upstream risk but increases the downstream risk . An increase in the downstream risk implies that at least one of the three errors terms in the composition above must have increased . We describe how each of these terms may increase due to a model update : • Upstream error measures the error that is due to the upstream model v not being part of the Bayesoptimal solution w∗ . It increases when an improvement in the upstream loss does not translate in a reduction in Bayes error of the downstream model , i.e. , when : ` w ( w† ◦ v′ ) > ` w ( w† ◦ v ) . An upstream error increase can happen due the loss mismatch : a setting in which the test loss functions , ` , of the upstream and downstream model optimize for different things . For example , the upstream test loss function may not penalize errors that need to be avoided in order for the downstream test loss to be minimized , or it may penalize errors that are irrelevant to the downstream model . Alternatively , the upstream error may increase due to distribution mismatch : situations in which the upstream loss focuses on parts of the data-target space X × Yv that are unimportant for the downstream model ( and vice versa ) . For example , suppose the upstream model v is an image-recognition model that aims to learn a feature representation that separates cats from dogs , and fv ( x ) is a feature representation obtained from that model . A downstream model , w , that distinguishes different dog breeds based on fv may deteriorate when the improvement of model v collapses all representations of dog images in fv . Examples of this were observed in , e.g. , [ 19 ] . • Downstream approximation error measures the error due to the optimal w† not being in the hypothesis setHw . The approximation error increases when the downstream model , w , is unable to exploit improvements in the upstream model , v , because exploiting those improvements would require selection of a model that is not in that hypothesis set . 3For brevity , we do not include optimization error of [ 5 ] in our Bayes error decomposition . Figure 2 illustrates how this situation can lead to a self-defeating improvement . In the illustration , the upstream model , v , predicts a 2D position for each data point . The downstream model , w , separates examples into the positive ( green color ) and negative ( red color ) class based on the prediction of v using a linear classifier . The predictions of the improved upstream model , v′ , better match the ground-truth targets : the predictions of v′ ( right plot ) match the ground-truth targets ( left plot ) more closely than those of v ( middle plot ) . However , the re-trained downstream model , w′ , deteriorates because the resulting classification problem is more non-linear : the downstream model can not capitalize on the upstream improvement because it is linear . The resulting increase in approximation error leads to the self-defeating improvement in Figure 2 . • Downstream estimation error measures the error due to the model training being performed on a finite data sample with an imperfect optimizer , which makes finding w†Hw difficult in practice . The estimation increases , for example , when the upstream model improves but the downstream model requires more than Nw training samples to capitalize on this improvement . Figure 3 shows an example of a self-defeating improvement caused by an increase of the downstream estimation error . As before , the upstream model , v , predicts a 2D position for each data point in the example . The downstream model w is selected from the set , Hw , of quadratic classifiers . It is tasked with performing binary classification into a positive class ( green color ) and negative class ( red color ) based on the 2D positions predicted by the upstream model . To train the binary classifier , the downstream model w ( and w′ ) is provided with Nw = 6 labeled training examples ( the green and red markers in the plot ) . In the example , the upstream model v′ performs better than its original counterpart v : the predictions of v′ ( right plot ) match the ground-truth targets ( left plot ) more closely than those of v ( middle plot ) . However , the upstream improvement hampers the downstream model even though the optimal downstream model w† is in the hypothesis set Hw both before and after the upstream improvement . After the upstream improvement , the optimal downstream model that can be selected from the hypothesis set , w†Hw , is more complex , which makes it harder to find it based on the finite number of Nw training examples . This results in an increase in the estimation error , which causes the self-defeating improvement in Figure 3 .

Summary: This paper investigates a practical problem referred to as self-defeating improvements when developing machine learning systems for real world applications such as self-driving cars or virtual assistants. That is, given a machine learning system with multiple components depending on each other, developing one component independently could sometimes make the overall system worse. This work first studies the problem in a system with two models (upstream task and downstream task) and decomposes the error into upstream error, downstream approximation error, and downstream estimation error. Building on top of the formulation, this work expands the discussion on a system with two or more upstream models. Finally, it demonstrates the emergence of self-defeating improvements in a realistic pseudo-LiDAR detection system.