Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Year: 2022

Abstract:
Machine-learning systems such as self-driving cars or virtual assistants are composed of a large number of machine-learning models that recognize image content, transcribe speech, analyze natural language, infer preferences, rank options, etc. Models in these systems are often developed and trained independently, which raises an obvious concern: Can improving a machine-learning model make the overall system worse? We answer this question affirmatively by showing that improving a model can deteriorate the performance of downstream models, even after those downstream models are retrained. Such self-defeating improvements are the result of entanglement between the models in the system. We perform an error decomposition of systems with multiple machine-learning models, which sheds light on the types of errors that can lead to self-defeating improvements. We also present the results of experiments which show that self-defeating improvements emerge in a realistic stereo-based detection system for cars and pedestrians.

heading: 1 Introduction
text: Progress in machine learning has allowed us to develop increasingly sophisticated artificially intelligent systems, including self-driving cars, virtual assistants, and complex robots. These systems generally contain a large number of machine-learning models that are trained to perform modular tasks such as recognizing image or video content, transcribing speech, analyzing natural language, eliciting user preferences, ranking options to be presented to a user, etc. The models feed each other information: for example, a pedestrian detector may use the output of a depth-estimation model as input. Indeed, machine-learning systems can be interpreted as directed acyclic graphs in which each vertex corresponds to a model, and models feed each other information over the edges in the graph.
In practice, various models in this graph are often developed and trained independently [22]. This modularization tends to lead to more usable APIs but may also be motivated by other practical constraints. For example, some of the models in the system may be developed by different teams (some models in the system may be developed by a cloud provider); models may be retrained and deployed at a very different cadence (personalization models may be updated very regularly but image-recognition models may not); new downstream models may be added continuously in the graph (user-specific models may need to be created every time a user signs up for a service); or models may be non-differentiable (gradientboosted decision trees are commonly used for selection of discrete features). This can make backpropagation through the models in the graph infeasible or highly undesirable in many practical settings.
The scenario described above leads to an obvious potential concern: Can improving a machinelearning model make the machine-learning system worse? We answer this question affirmatively by showing how improvements in an upstream model can deteriorate the performance of downstream models, even if all the downstream models are retrained after updating the upstream model. Such
∗Work performed during internship at Facebook AI Research.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
self-defeating improvements are caused by entanglement between the models in the system. To better understand this phenomenon, we perform an error decomposition of simple machine-learning systems. Our decomposition sheds light on the different types of errors that can lead to selfdefeating improvements: we provide illustrations of each error type. We also show that self-defeating improvements arise in a realistic system that detects cars and pedestrians in stereo images.
Our study opens up a plethora of new research questions that, to the best of our knowledge, have not yet been studied in-depth in the machine-learning community. We hope that our study will encourage the community to move beyond the study of machine-learning models in isolation, and to study machine-learning systems more holistically instead.

heading: 2 Problem Setting
text: We model a machine-learning system as a static directed acyclic graph (DAG) of machine-learning models, G = (V, E), where each vertex v ∈ V corresponds to a machine-learning model and each edge (v, w) ∈ E represents the output of model v being used as input into model w. For example, vertex v could be a depth-estimation model and vertex w a pedestrian-detection model that operates on the depth estimates produced by vertex v. A model is upstream of v if it is an ancestor of v. A downstream model of v is a descendant of v. An example of a machine-learning system with three models is shown in Figure 1 (we study this model in Section 3.2).
Each vertex in the machine-learning system G corresponds to a model function, v(·), that operates on both data and outputs from its upstream models. Denote the output from model v by fv(x). The output of a source model v is fv(x) = v(x(v)), where x(v) represents the part of input x that is relevant to model v. The output of a non-source model w is given by fw(x) = w ([ x(w), fv(x) : v ∈ Pw ]) , where Pw = {v : (v, w) ∈ E} are the parents of w. Training. To train model v, we assume access to a training set Dv = {(x1,y1), . . . , (xNv ,yNv )} with Nv examples xn ∈ X and corresponding targets yn ∈ Yv. For each model, we also assume a training algorithm Av(Dv) that selects a model v ∈ Hv from hypothesis setHv based on the training set. The learning algorithm, Av, the hypothesis set, Hv, and the training set, Dv, are fixed during
training but they may change each time model v is re-trained or updated. The data space X , the target space Yv , and the way in which inputs x(v) are obtained do not change between model updates. We assume that models in G may be trained separately rather than jointly, that is, the learning signal obtained from training a downstream model, v, may not be backpropagated into its upstream dependencies, w ∈ Pv . This assumption is in line with constraints that commonly arise in real-world machine learning systems; see our motivation below. Note that this assumption also implies that a downstream model cannot be trained before all its upstream models are trained.
Evaluation. To be able to evaluate the performance of the task at v, we assume access to a fixed test set D̄v that is defined analogously to the training set: it contains Mv test examples x̄m ∈ X and corresponding targets ȳm ∈ Yv. In contrast to the training set, the test set is fixed and does not change when the model is updated. However, we note that the input distribution into a non-root model is not only governed by the test set D̄v but also by upstream models: if an upstream model changes, the inputs into a downstream model may change as well even though the test set is fixed.
We also assume access to a test loss function `v(v, D̄v) for each model v, for example, classification error (we assume lower is better). Akin to the test set D̄v , all test loss functions `v are fixed. Hence, we always measure the generalization ability of a model, v, in the exact same way. The test loss function may not have trainable parameters, that is, it must be a “true” loss function.
Updating models. The machine-learning system we defined can be improved by updating a model. In such an update, an individual model v is replaced by an alternative model v′. We can determine if such a model update constitutes an improvement by evaluating whether or not its test loss decreases: `v(v
′, D̄v) ≤ `v(v, D̄v). When v is not a leaf in G, the update to v′ may affect the test loss of downstream models as well. In practice, the improvement due to v′ often also improves downstream models (e.g., [19]), but this is not guaranteed. We define a self-defeating improvement as a situation in which replacing v by v′ improves that model but deteriorates at least one of the downstream models w, even after all the downstream models are updated (i.e., re-trained using the same learning algorithm) to account for the changes incurred by v′. Definition 2.1 (Self-defeating improvement). Denoting the set of all downstream models (descendants) of v in G as Cv , a self-defeating improvement arises when:
∃w ∈ Cv : `v(v′, D̄v) ≤ `v(v, D̄v) 6=⇒ `w(w′, D̄w) ≤ `w(w, D̄w)), (1)
where w′ represents a new version of model w that was trained after model v was updated to v′ to account for the input distribution change that the update of v incurs on w.
Motivation of problem setting. In the problem setting described above, we have made two assumptions: (1) models may be (re-)trained separately rather than jointly and (2) the training algorithm, hypothesis set, and training data may change between updates of the model.2 Both assumptions are motivated by the practice of developing large machine-learning systems that may comprise thousands of models [22, 28]. Joint model training is often infeasible in such systems because:
• Some of the models may have been developed by cloud providers [34] and cannot be changed. • Backpropagation may be technically infeasible or inefficient, e.g., when models are implemented
in incompatible learning frameworks or when their training data live in different data centers. • Some models may be re-trained or updated more often than others. For example, personalization
models may be updated every few hours to adapt to changes in the data distribution, but re-training large language models [8] at that same cadence is not very practical. • Upstream models may have hundreds or even thousands of downstream dependencies, complicating appropriate weighting of the downstream learning signals that multi-task learning [10] requires. • Some models may be non-differentiable, e.g., gradient-boosted decision trees are commonly used for feature selection in large sets of features.
We assume that changes in the training algorithm, hypothesis set, and training data may occur because model owners (for example, cloud providers) constantly seek to train and deploy improved versions of their models with the ultimate goal of improving the system(s) in which the models are used.

heading: 3 Understanding Self-Defeating Improvements via Error Decompositions
text: Traditional software design relies heavily on modules: independent and interchangeable components that are combined to form complex software systems [7, §7.4]. Well-designed modules provide an explicit specification of their inputs, outputs, and functionality. In traditional software engineering, good modularization allows developers to modify modules independently because the effects of those modifications on the rest of the system are relatively predictable.
Machine-learned models do not resemble traditional software modules because they lack an explicit specification of their functionality [14]. As a result, there is often a significant degree of entanglement between different models in the system [22]. This makes the system-wide effects of model modifications unpredictable, and can lead to situations where improving a model does not improve the entire system. To understand such self-defeating improvements, we study Bayes error decompositions of simple machine-learning systems. Specifically, we study a system with two models in Section 3.1 and a system with three models in Section 3.2.

heading: 3.1 Error Decomposition of System with Two Models
text: We adapt standard Bayes error decomposition [6, 33] to study a simple system that has two vertices, V = {v, w}, and a dependency between upstream model v and downstream model w, that is, E = {(v, w)}. We denote the Bayes-optimal downstream model by w∗; the optimal downstream
2The training data is fixed during training, i.e., we do not consider online learning settings in our setting.
model conditioned on an upstream model by w†; and the optimal upstream-conditional downstream model in the hypothesis setHw by w†Hw . Using these definitions, we can decompose the downstream risk of model w, given upstream model v, as follows:
E[`w(w ◦ v)− `w(w∗)] = E[`w(w† ◦ v)− `w(w∗)]︸ ︷︷ ︸
upstream error
+E[`w(w†Hw ◦ v)− `w(w † ◦ v)]︸ ︷︷ ︸
downstream approximation error
+E[`w(w ◦ v)− `w(w†Hw ◦ v)]︸ ︷︷ ︸ downstream estimation error , (2)
where ◦ denotes function composition, and the expectations are under the data distribution. The error decomposition is similar to standard decompositions [6, 33] but contains three terms instead of two.3 Specifically, the upstream error does not arise in prior error decompositions, and the downstream approximation and downstream estimation errors differ from the standard decomposition.
Barring variance in the error estimate due to the finite size of test sets D̄w (which is an issue that can arise in any model-selection problem), a self-defeating improvement occurs when a model update from (v, w) to (v′, w′) reduces the upstream risk but increases the downstream risk. An increase in the downstream risk implies that at least one of the three errors terms in the composition above must have increased. We describe how each of these terms may increase due to a model update:
• Upstream error measures the error that is due to the upstream model v not being part of the Bayesoptimal solution w∗. It increases when an improvement in the upstream loss does not translate in a reduction in Bayes error of the downstream model, i.e., when: `w(w† ◦ v′) > `w(w† ◦ v). An upstream error increase can happen due the loss mismatch: a setting in which the test loss functions, `, of the upstream and downstream model optimize for different things. For example, the upstream test loss function may not penalize errors that need to be avoided in order for the downstream test loss to be minimized, or it may penalize errors that are irrelevant to the downstream model.
Alternatively, the upstream error may increase due to distribution mismatch: situations in which the upstream loss focuses on parts of the data-target space X × Yv that are unimportant for the downstream model (and vice versa). For example, suppose the upstream model v is an image-recognition model that aims to learn a feature representation that separates cats from dogs, and fv(x) is a feature representation obtained from that model. A downstream model, w, that distinguishes different dog breeds based on fv may deteriorate when the improvement of model v collapses all representations of dog images in fv . Examples of this were observed in, e.g., [19]. • Downstream approximation error measures the error due to the optimal w† not being in the hypothesis setHw. The approximation error increases when the downstream model, w, is unable to exploit improvements in the upstream model, v, because exploiting those improvements would require selection of a model that is not in that hypothesis set.
3For brevity, we do not include optimization error of [5] in our Bayes error decomposition.
Figure 2 illustrates how this situation can lead to a self-defeating improvement. In the illustration, the upstream model, v, predicts a 2D position for each data point. The downstream model, w, separates examples into the positive (green color) and negative (red color) class based on the prediction of v using a linear classifier. The predictions of the improved upstream model, v′, better match the ground-truth targets: the predictions of v′ (right plot) match the ground-truth targets (left plot) more closely than those of v (middle plot). However, the re-trained downstream model, w′, deteriorates because the resulting classification problem is more non-linear: the downstream model cannot capitalize on the upstream improvement because it is linear. The resulting increase in approximation error leads to the self-defeating improvement in Figure 2. • Downstream estimation error measures the error due to the model training being performed on a finite data sample with an imperfect optimizer, which makes finding w†Hw difficult in practice. The estimation increases, for example, when the upstream model improves but the downstream model requires more than Nw training samples to capitalize on this improvement.
Figure 3 shows an example of a self-defeating improvement caused by an increase of the downstream estimation error. As before, the upstream model, v, predicts a 2D position for each data point in the example. The downstream model w is selected from the set, Hw, of quadratic classifiers. It is tasked with performing binary classification into a positive class (green color) and negative class (red color) based on the 2D positions predicted by the upstream model. To train the binary classifier, the downstream model w (and w′) is provided with Nw = 6 labeled training examples (the green and red markers in the plot). In the example, the upstream model v′ performs better than its original counterpart v: the predictions of v′ (right plot) match the ground-truth targets (left plot) more closely than those of v (middle plot). However, the upstream improvement hampers the downstream model even though the optimal downstream model w† is in the hypothesis set Hw both before and after the upstream improvement. After the upstream improvement, the optimal downstream model that can be selected from the hypothesis set, w†Hw , is more complex, which makes it harder to find it based on the finite number of Nw training examples. This results in an increase in the estimation error, which causes the self-defeating improvement in Figure 3.

heading: 3.2 Error Decomposition of System with Two Upstream Models
text: More complex types of self-defeating improvements may arise when a downstream model depends on multiple upstream models that are themselves entangled. Consider the system of Figure 1 that has three models, V = {u, v, w}, and dependencies between upstream models u and v and downstream model w, that is, E = {(u,w), (v, w)}. The error decomposition for this system is
similar to Equation 2, but we can further decompose the upstream error. Denoting the Bayes-optimal downstream model by w∗, the optimal model given upstream models u and v by w†u,v , and the optimal upstream model v given upstream model u by v†u, we decompose the upstream error as follows:
E[`w(w†u,v ◦ (u, v))− `w(w∗)]︸ ︷︷ ︸ upstream error = E[`w(w†u,v ◦ (u, v))− `w(w†u,v ◦ (u, v†u))]︸ ︷︷ ︸ upstream compatibility error +E[`w(w†u,v ◦ (u, v†u))− `w(w∗)]︸ ︷︷ ︸ excess upstream error . (3)
The excess upstream error is similar to the upstream error in Equation 2: upstream model u may be suboptimal for the downstream task, for example, because of loss mismatch or distribution mismatch. The key observation in the error decomposition of a system with two upstream models is that the optimal upstream model v is a function of upstream model u (and vice versa). The upstream compatibility error captures the error due to upstream model v not being identical to the optimal v†u. A self-defeating improvement can occur when we update upstream model u to u′ because it may be that v†u 6= v † u′ , which can cause the upstream compatibility error to increase.
We provide two examples of this in Figure 4. The first example (left pane) shows a self-defeating improvement due to upstream models u and v making anti-correlated errors that cancel each other out. The second example (right pane) is a self-defeating improvement due to u′ making more correlated predictions with v. We note that, in both examples, the optimal v depends on u and the self-defeating improvement arises because of an increase in the upstream compatibility error.
In the examples in Figure 4, all three models aim to distinguish two classes: green (positive class) and red (negative class). Upstream models u and v do so based the (x, y)-location of points in the 2D plane. The downstream model operates on the (hard) outputs of the two upstream models. In Figure 4(a), the original upstream models u and v make errors that are anti-correlated. The original downstream model w exploits this anti-correlation to make perfect predictions. When upstream model u is replaced by an improved model u′ that makes no errors, a self-defeating improvement arises: downstream model w no longer receives the information it needs to separate both classes perfectly. In Figure 4(b), upstream model u is improved to u′, which makes the exact same predictions as the other upstream model v. As a result, downstream model w′ no longer has access to the complementary information that u was providing in addition to v, producing the self-defeating improvement.
While the examples in Figure 4 may seem contrived, anti-correlated errors and correlated predictions can arise in practice when there are dependencies in the targets used to train different models. For example, suppose the upstream models are trained to predict ad clicks (will a user click on an ad?) and ad conversions (will a user purchase the product being advertised?), respectively. Because conversion can only happen after a click happens, there are strong dependencies between the targets used to train both models, which may introduce correlations between their predictions.

heading: 4 Case Study: Pseudo-LiDAR for Detection
text: We perform a case study on self-defeating improvements in a system that may be found in self-driving cars. In both cases, the system uses a pseudo-LiDAR [37] as the basis for 3D detection of cars and pedestrians. Figure 5 gives an overview of the models in the system:
• A disparity-estimation model, u, predicts a map d with disparities corresponding to every pixel in the left image of a stereo image (a pair of rectified images from two horizontally aligned cameras). • A point cloud generator, r, uses disparity map d to create a pseudo-LiDAR representation of the 3D environment. The point cloud generator, r, is a simple non-learnable module [37]. • The stereo image and disparity map are input into a model, v, that performs 3D detection of cars. • The same inputs are used in a separate model, w, that aims to perform 3D pedestrian detection.
4.1 System Design
The disparity-estimation and detection models used in our case study are described below.
Disparity estimation. To perform disparity estimation, we adopt the PSMNet model of Chang and Chen [11]. The model receives a stereo image, (xleft,xright), as input and aims to compute a disparity map, d = u(xleft,xright), that contains a disparity estimate for each pixel in image xleft. We experiment with two training loss functions: (1) the depth mean absolute error and (2) the disparity mean absolute error (MAE). Given a ground-truth disparity map, y, and a function that maps disparities to depths, g(d) = Cd , for some camera constant C, the disparity MAE is proportional to ‖d− y‖1 and the depth MAE is proportional to ‖g(d)− g(y)‖1. To evaluate the quality of model u, we measure disparity MAE: `u(u, D̄u) = 1M ∑M m=1‖d − y‖1. We expect that a model, u′, trained to minimize disparity
MAE will provide better disparity estimates (per test loss `u) than a model, u, that is trained to minimize depth MAE. This has downstream effects on performance of the detection models.
Car and pedestrian detection. We experiment with two different models for performing detection of cars and pedestrians, viz., the Point-RCNN model (P-RCNN; Shi et al. [30]) and the Frustum PointNet detection model (F-PTNET; Qi et al. [26]). Both models take stereo image (xleft,xright) and point cloud r(d) as input. Before computing r(d), we perform winsorization on the prediction u(xleft,xright): we remove all points that are higher than 1 meter in the point cloud (where the camera position is the origin). Both 3D car detection model and 3D pedestrian detection model are trained to detect their target objects at any distance from the camera.
Following Geiger et al. [16], we evaluate the test loss of car-detection model, `v, using the negative average precision (AP) for an intersection-over-union (IoU) of 0.7. The test loss of the pedestriandetection model, `w, is the negative AP for an IoU of 0.5. The pedestrian-detection test-loss is only evaluated on pedestrians whose distance to the camera is ≤20 meters. We measure both the car-detection and the pedestrian-detection test losses for both the 3D box view (−AP3D) and the 2D bird-eye view (−APBEV); see Geiger et al. [16] for details on the definition of the test-loss functions.
4.2 Experiments
We evaluate our system on the KITTI dataset [16, CC BY-NC-SA 3.0], using the training-validation split of [12]. Our baseline disparity-estimation model, u, that is trained to minimize depth MAE obtains a test loss of 1.28 (see Table 2). The improved version of that model, u′, is trained to minimize disparity MAE and obtains a test loss of 1.21, confirming the model improvement.
Table 1 presents the test losses of the downstream models for car and pedestrian detection, for both the baseline upstream model u and the improved model u′. We observe that the improvement in disparity estimation leads to selfdefeating improvements on both the car-detection and the pedestrian-detection tasks. The AP3D of the car detector drops from 39.91 to 38.75 (from 50.34 to 49.30 in APBEV).
For pedestrian detection, AP3D is unchanged but APBEV drops from 50.49 to 48.29. Interestingly, these self-defeating improvements are due to increases in different error terms.
Self-defeating improvement in car detection. The selfdefeating improvement observed for car detection is likely due to an increase in the upstream error. A disparity-estimation model that is trained to minimize depth MAE (i.e., model u) focuses on making sure that small disparity values (large depth values), are predicted correctly. By contrast, a model trained to minimize disparity MAE (model u′) aims to predict large disparity values (small depth values) correctly, sacrificing accuracy in predictions of large depth values (see Table 2). This negatively affects the car detector because most cars tend to be relatively far away. In other words, the loss function that improves model u deteriorates the downstream model because it focuses on errors that are less relevant to that model, which increases the upstream error.
Self-defeating improvement in pedestrian detection. Different from car detection, the self-defeating improvement in pedestrian detection is likely due to an increase in the downstream approximation or estimation error. Table 2 shows that whereas the disparity MAE test loss of u′ increases for large depth values, it decreases in the 0-20 meter range. As the pedestrian-detection evaluation focuses on pedestrians within 20 meters, we expect the pedestrian detector w′ to improve along with disparity-estimation model u′: that is, the upstream error likely decreases.
However, the point representations produced by model u′ are likely noisier for far-away pedestrians than those produced by u (see Table 2). Because the downstream model is trained to (also) detect these far-away pedestrians, the noise in the pedestrian point clouds makes it harder for that model to learn a good model of the shape of pedestrians. This negatively affects the ability of w′ to detect nearby pedestrians. Indeed, if the pedestrian detector is only trained on nearby pedestrians, the self-defeating improvement disappears; see Table 3. Hence, the observed self-defeating improvement was due to an increase in a downstream error term: the upstream model did improve for the downstream task, but the downstream model was unable to capitalize on that improvement.

heading: 5 Related Work
text: This study is part of a larger body of work around machine-learning engineering [9, 21, 23]. Sculley et al. [28] provides a good overview of the problems that machine-learning engineering studies. The self-defeating improvements studied in this paper are due to the entanglement problem [1, 22], where “changing anything changes everything” [28]. Entanglement has also been studied in work on “backward-compatible” learners [29, 32, 35]. Prior work has also studied how the performance of humans operating a machine-learning system may deteriorate when the system is improved [3].
Some aspects of self-defeating improvements have been studied in other domains. In particular, the effect of an upstream model change on a downstream model can be viewed as domain shift [4, 25]. However, approaches to domain shift such as importance weighting of training examples [31] are not directly applicable because they require the input space of both models to be identical. The study of self-defeating improvements is also related to the study of transfer learning, e.g., [2, 19, 24]. Prior work on transfer learning has reported examples of self-defeating improvements: e.g., [19] reports that some regularization techniques improve the accuracy of convolutional networks on ImageNet, but negatively affect the transfer of those networks to other recognition tasks.
Finally, work on differentiable programming [18, 36] is relevant to the study of self-defeating improvements. In some situations, a self-defeating improvement may be resolved by backpropagating learning signals from a model further upstream. However, such an approach is not a panacea as downstream tasks may have conflicting objectives. Moreover, technical or practical obstacles may prevent the use of differentiable programming in practice (see Section 2).

heading: 6 Conclusion and Future Work
text: This study explored self-defeating improvements in modular machine-learning systems. We presented a new error decomposition that sheds light on the error sources that can give rise to self-defeating improvements. The findings presented in this study suggest a plethora of directions for future work:
• It may be possible to derive bounds on the error terms in Equation 2 that go beyond the current art. For example, it is well-known that the trade-off between approximation and estimation error leads to an excess risk that scales between the inverse and the inverse square root of the number of training examples [27, 39]. New theory may provide more insight into the effect of the upstream hypothesis set and training set size on those errors, and may bound the upstream error term in 2. • Our study focused on first-order entanglement between two models (either an upstream and a downstream model or two upstream models). Some settings may give rise to higher-order entanglements, for example, between three upstream models whose errors cancel each other out. • Our definition of different types of entanglement may pave the way for the development of diagnostic tools that identify the root cause of a self-defeating improvement. • We need to develop best practices that can help prevent self-defeating improvements from occurring. One way to do so may be by tightening the “API” of the machine-learning models, e.g., by calibrating classifier outputs [17] or whitening feature representations [20]. Such transforms constrain a model’s output distribution, which may reduce the downstream negative effects of model changes. Alternatively, we may train models that are inherently more robust to changes in their input distribution, e.g., using positive-congruent training [38] or meta-learning [15]. • We also need to study self-defeating improvements of other aspects. An important open question is [13]: Can making an upstream model fairer make its downstream dependencies less fair?
We hope this paper will encourage the study of such questions, and will inspire the machine-learning community to rigorously study machine-learning systems in addition to individual models.

heading: Acknowledgements
text: The authors thank Yan Wang, Yurong You, and Ari Morcos for helpful discussions and advice.

Review summaries:
This paper studies the problem of "self-defeating improvement" (improving an upstream ML model deteriorates a downstream ML model in the same system). Leveraging Bayes error decomposition, it decomposes the error of a system with multiple ML models into: (excess) upstream error, upstream compatibility error (when more than one upstream model is used), downstream approximation error, and downstream estimation error. The paper conducts a case study in Pseudo-LiDAR for vehicle/pedestrian detection to demonstrate the existence of the studied "self-defeating improvement" in real world.

This paper explores, "self-defeating improvements, potential negative effects of improving a part of a module machine learning system. The causes of such self-defeating improvements are analyzed through Bayes error decomposition. The paper presents a case study with pedestrian and car detection where both tasks rely on an upstream disparity estimation model. The results show that despite improvements in disparity estimation (when tested alone), the overall system performance degrades. 

This paper investigates a practical problem referred to as self-defeating improvements when developing machine learning systems for real world applications such as self-driving cars or virtual assistants. That is, given a machine learning system with multiple components depending on each other, developing one component independently could sometimes make the overall system worse. This work first studies the problem in a system with two models (upstream task and downstream task) and decomposes the error into upstream error, downstream approximation error, and downstream estimation error. Building on top of the formulation, this work expands the discussion on a system with two or more upstream models. Finally, it demonstrates the emergence of self-defeating improvements in a realistic pseudo-LiDAR detection system.