Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Authors: Ruihan Wu
Chuan Guo
Awni Hannun
Laurens van der Maaten
Venue: NeurIPS 2021 Poster (2021)
Keywords: System-level Evaluation
Model Compatibility
Error Decomposition

Abstract: Machine-learning systems such as self-driving cars or virtual assistants are composed of a large number of machine-learning models that recognize image content, transcribe speech, analyze natural language, infer preferences, rank options, etc. Models in these systems are often developed and trained independently, which raises an obvious concern: Can improving a machine-learning model make the overall system worse? We answer this question affirmatively by showing that improving a model can deteriorate the performance of downstream models, even after those downstream models are retrained. Such self-defeating improvements are the result of entanglement between the models in the system. We perform an error decomposition of systems with multiple machine-learning models, which sheds light on the types of errors that can lead to self-defeating improvements. We also present the results of experiments which show that self-defeating improvements emerge in a realistic stereo-based detection system for cars and pedestrians.

Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Ruihan Wu" Cornell University rv565@corne11.co edu
Chuan Guo
Awni Hannun
Laurens van der Maaten
Facebook AI Research {chuanguo, awmi, wawmiiilidaaaeaaaaaa 1vdmaaten)Offb... com
Abstract
Machine-learning systems such as self-driving cars or virtual assistants are com- posed of large number of machine-learning models that recognize image content, transcribe speech, analyze natural language, infer preferences, rank options, etc. Models in these systems are often developed and trained independently, which raises an obvious concern: Can improving a muchine-learning model make the overall system worse? ? We answer this question affirmatively by showing that improving a model can deteriorate the performance of downstream models, even after those downstream models are retrained. Such self-defeating improvements are the result of entanglement between the models in the system. We perform an error decomposition of systems with multiple machine-leamme light on the types of errors that can lead to self-defeating improvements. We also present the results of experiments which show that self-defeating improvements emerge a realistic stereo-based detection system for cars and pedestrians.
models, which sheds
Introduction
Progress in machine learning has allowed to develop increasingly sophisticated artificially intel- ligent systems, including self-driving cars, virtual assistants, and complex robots. These systems generally contain large number of machine-leamingg models that are trained to perform modular tasks such as recognizing image or video content, transcribing speech, analyzing natural language, eliciting preferences, ranking options be presented user, etc. The models feed each other information: for example, a pedestrian detector may use the output of depth-estimation model as input. Indeed, machine-leaman learning systems can be interpreted as directed acyclic graphs in which each vertex corresponds to model, and models feed each other information over the edges in the graph.
In practice, various models in this graph are often developed and trained independently [22] This modularization tends to lead to more usable APIs but may also be motivated by other practical constraints. For example, some of the models in the system may be developed by different teams (some models in the system may be developed by a cloud provider); models may be retrained and deployed at very different cadence (personalization models may he updated very regularly but image-recognitionn models may not); new downstream models may be added continuously in the graph (user-specific models may need be created every time boosted decision trees are commonly used for selection of discrete features). This can make backpropagation through the models in the graph infeasible or highly undesirable in many practical settings.
signs up for service); models may be non-differentiable (gradient-
The scenario described above leads to an obvious potential concern: Can improving a machine- learning model make the machine-learnin system worse? We answer this question affirmatively by showing how improvements in an upstream model can deteriorate the performance of downstream models, even all the downstream models are retrained after updating the upstream model. Such
Work performed during internship at Facebook Al Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

self-defeating improvements are caused by entanglement between the models in the system. To better understand this phenomenon, we perform an error decomposition of simple machine-learning systems. Our decomposition sheds light on the different types of errors that can lead to self- defeating improvements: we provide illustrations of each error type. We also show that self-defeating improvements arise in realistic system that detects cars and pedestrians in stereo images.
Our study opens up a plethora of new research questions that, to the best of our knowledge, have not yet been studied in-depth in the machine-learning community. We hope that our study will encourage the community to move beyond the study of machine-leamme learning models in isolation, and to study machine-leaminng systems more holistically instead.
2 Problem Setting
We model a machine-learning system as a static directed acyclic graph (DAG) of machine-learning models, G (V.E), E), where each vertex V corresponds machine-learning model and each edge (u, w) E E represents the output of model v being used as input into model w. For example, vertex v could be a depth-estimation model and vertex w a pedestrian-detection model that operates on the depth estimates produced by vertex v. A model is upstream of v if is an ancestor of v. A downstream model of is a descendant of v. An example of machine-learning system with three models is shown in Figure (we study this model in Section 3.2)
Each vertex in the beccmine system G corresponds to a model function, v(-), that op- erates on both data and outputs from its up- stream models. Denote the output from model by (x). The output of source model is [u(x) u(x(1)), where x(1) represents the part of input x that is relevant to model u. The output of a non-source model is given by (x) w ([x("0), ((xx)
a
fx
(x))
f
x(+))
f.
V E Pwl)) where P = (u: (u, w) E E) are the parents of w.
V
Training. To train model v. we as- sume access to a training set D. = (1x), YN.)) with No examples Xn E X'' and corresponding targets Yn E V For each model, we also assume training al- gorithm (D.)) that selects a model E He from hypothesis set H. based on the training set. The learning algorithm, A.. the hypothesis set, He, and the training set, D,, are fixed during training but they may change each time model v is re-trained or updated. The data space X, the target space Vo, and the way in which inputs (()) are obtained do not change between model updates.
Figure 1: Example of a machine-learning system with three models, V = {u, v, w. and two edges, = {(u, w), (v, w))- The system receives an example x as input, of which parts x(),, x("), and x("") are fed into u, v, and w, respectively. Model w's input is concatenated with the outputs of its parents, (x) and f (x).
We assume that models in G may be trained separately rather than jointly, that is, the learning signal obtained from training a downstream model, v. may not be backpropagated into its upstream dependencies, P... This assumption is in line with constraints that commonly arise in real- world machine learning systems; see our motivation below. Note that this assumption also implies that a downstream model cannot be trained before all its upstream models are trained.
Evaluation. To be able to evaluate the performance of the task at V we assume access to a fixed test set D. that is defined analogously to the training set: it contains M, test examples Xm X and corresponding targets ym E yv. In contrast to the training set, the test set is fixed and does not change when the model is updated. However, we note that the input distribution into non-root model is not only governed by the test set D. but also by upstream models: an upstream model changes, the inputs into downstream model may change as well even though the test set is fixed.
We also assume access to a test loss function f. D.,) for each model v, for example, classification error (we assume lower is better). Akin to the test set DD,, all test loss functions f, are fixed. Hence, we always measure the generalization ability of a model, v, the exact same way. The test loss function may not have trainable parameters, that is, it must be "truc" loss function.
2

---

Updating models. The machine- learning system we defined can be improved by updating model. In such an update, an individual model is replaced by an alternative model v'. We can determine such a model update constitutes an improvement by evaluating whether or not its test loss decreases: (o (D)) S (v, D.). When v is not leaf in G. the update to e' may affect the test loss of downstream models as well. In practice, the improvement due to o' often also improves downstream models (e.g., s. (19). but this is not guaranteed. We define a self-defeating improvement as a situation in which replacing by v' improves that model but deteriorates at least one of the downstream models w, even after all the downstream models are updated (i.e., re-trained using the same learning algorithm) to account for the changes incurred by v'.
Definition 2.1 (Self-defeating improvement). Denoting the set of all downstream models (descen- dants) of in G as Cv, a
improvement arises when:
Ew E C. Lo D.) S D.)
+ (w (w)) S (w, Dw)),
(1)
where w' represents a new version of model wo that was trained after model e was updated to v' to account for the input distribution change that the update of incurs on w.
Motivation of problem setting. In the problem setting described above, we have made two assump- tions: (1) models may be re-))trineed separately rather than jointly and (2) the training algorithm,
hypothesis set, and training data may change between updates of the model 2 Both assumptions are motivated by the practice developing large machine-learning systems that may comprise thousands
of models 22 28 Joint model training is often infeasible in such systems because:
Some of the models may have been developed by cloud providers 34 and cannot be changed. Backpropagation may be technically infeasible or inefficient, egg., when models are implemented
in incompatible learning frameworks or when their training data live in different data centers. Some models may be re-trained or updated more often than others. For example, personalization models may be updated every few hours adapt changes the data distribution, but re-training large language models [ at that same cadence is not very practical.
Upstream models may have hundreds or even thousands of downstream dependencies, complicating
appropriate weighting of the downstream learning signals that multi-task learning [10] requires. Some models may be non-differentiable, e.g., gradient- boosted decision trees are commonly used for feature selection in large sets of features.
We assume that changes in the training algorithm, hypothesis set, and training data may occur because model owners (for example, cloud providers) constantly seek to train and deploy improved versions of their models with the ultimate goal of improving the system(s) in which the models are used.
3 Understanding Self-Defeating Improvements via Error Decompositions
Traditional software design relies heavily on modules: independent and interchangeable components that are combined to form complex software systems 7 $7.4]. Well-designed modules provide an explicit specification of their inputs, outputs, and functionality. In traditional software engineering, good modularization allows developers to modify modules independently because the effects of those modifications on the rest of the system are relatively predictable.
Machine-leared models do not resemble traditional software modules because they lack explicit specification of their functionality [14]. As a result, there is often a significant degree of entanglement between different models in the system (22). This makes the system-wide effects of model modifica- tions unpredictable, and can lead to situations where improving a model does not improve the entire system. To understand such self-defeating improvements, we study Bayes error decompositions of simple machine- learning systems. Specifically, we study system with two models in Section 3.1 and system with three models in Section 3.2
3.1 Error Decomposition of System with Two Models
We adapt standard Bayes error decomposition 6 33 to study a simple system that has two vertices, {u, w). and a dependency between upstream model and downstream model w. that is,
E = ((v, w We denote the Bayes-optimal downstream model by w"; the optimal downstream
'The training data is fixed during training, i.e., do not consider online learning settings in setting.
3

---

Downstream model
Downstream model x²
ppstraum-oondiiooee optimal downstream model w²
g/
22
U2
1/ Upstream targets
1 Upstream model y
1 Upstream model v²
Figure 2: Illustration of self-defeating improvement due to increase in the downstream approx- imation error. Upstream model u predicts a 2D point for each example. Downstream model w is linear classifier separating the points into two classes (red and green). Left: Ground- target distribution for the upstream model. Middle: Point predictions by upstream model v, the optimal o-conditioned downstream classifier w and the downstream classifier learne.. Right: Point predictions by improved upstream model v', the optimal v' o'-conditioned downstream classifier wi, and the downstream classifier w learned. Note how the predictions of e' better match the ground-truth, but w' cannot exploit the improvement because it is restricted to be linear.
model conditioned model in the hypothesis set Ho by w Using these definitions, we can decompose the downstream risk of model w, given upstream model v, as follows:
upstream model by w and the optimal upstream- conditioonn downstream
(w o v)- (ww)w)]
E ( v) (ww"
E[e (w ou) (ww v]] ([e Lw (w) o v)].
upstream orror
downstream approximation error
downstream estimation
where denotes function composition, and the expectations are under data distribution. The error decomposition is similar to standard decompositions 6 (33 but contains three terms instead of two Specifically, the upstream error does not arise in prior error decompositions, and the downstream approximation and downstream estimation errors differ from the standard decomposition.
B
Barring variance in the error estimate due the finite size of test sets D. (which an issue that can arise in any model-selection problem), self-defcating improvement occurs when a model update from (v, w=) to (v', w') reduces the upstream risk but increases the downstream risk. An increase in the downstream risk implies that at least one of the three errors terms in the composition above must have increased. We describe how each of these terms may increase due to a model update:
Upstream error measures the error that is due to the upstream model not being part of the Bayes- optimal solution w". It increases when an improvement in the upstream loss does reduction in Bayes eтror of the downstream model, i.e., when: lw (w" o o') ) > (w o v). An upstream eиrоr increase can happen due the loss mismatch: setting which the test loss functions, f. of the upstream and downstream model optimize for different things. For example, the upstream test loss function may not penalize errors that need to avoided in order for the downstream test loss to be minimized, or it may penalize errors that are irrelevant to the downstream model. Alternatively, the upstream error may increase due to distribution mismatch: situations in which the upstream loss focuses on parts of the data-target space X' x V, that are unimportant for the downstream model (and vice versa). For example, suppose the upstream model is an image-recognition model that aims to learn feature representation that separates cats from dogs, and (x) is a feature representation obtained from that model. A downstream model, w, that distinguishes different dog breeds based on fo may deteriorate when the improvement of model collapses all representations of dog images in f.. Examples of this were observed in, e.g., [19]. Downstream approximation error measures the error due to the optimal wi not being in the hypothesis set H.,. The approximation error increases when the downstream model, w, is unable to exploit improvements in the upstream model, v, because exploiting those improvements would require selection of a model that is not in that hypothesis set.
translate in
For brevity, do include optimization error of in our Bayes decomposition.

---

Downstream model
[pstream-conditoon optimal downsteam model wi
Downstream model w²
Upstream-condn conditioned optimal model wk. in hypothesis set
U/
yy Upstream targets
V Upstream model v²
Upstream model v
Figure 3: Illustration of self-defeating improvement due increase in the downstream estimation error. Upstream model v predicts 2D point for each example. Downstream model w is a quadratic classifier separating the points into two classes (red and green). Left: Ground-trute target samples and distribution for the upstream model. Middle: Point predictions by upstream model v, the corresponding optimal u-conditioned downstream model in H.w. w and the downstream classifier learned based on the training examples. Right: Point predictions by improved upstream model v', the corresponding optimal o'-conditioned downstream model in H.. WH,, and the downstream classifier w' learned based on the training examples. Note how the predictions of better match the ground-truth, but w' deteriorates because it only receives Nw 6 training examples.
Figure illustrates how this situation can lead to self-defeating improvement. In the illustration, the upstream model, v, predicts a 2D position for each data point. The downstream model, w, separates examples into the positive (green color) and negative (red color) class based on the prediction of v using a linear classifier. The predictions of the improved upstream model, v', better match the ground-truth targets: the predictions of u' (right plot) match the ground-truth targets (left plot) more closely than those of v (middle plot). However, the re- re-trained downstream model, w', deteriorates because the resulting classification problem is more non-linear: the downstream model cannot capitalize on the upstream improvement because it is linear. The resulting increase in approximation error leads the self-defeating improvement in Figure
Downstream estimation error measures the error due to the model training being performed on
finite data sample with an imperfect optimizer, which makes finding W difficult in practice. The estimation increases, for example, when the upstream model improves but the downstream model requires more than N.. training samples to capitalize on this improvement.
Figure 3 shows an example of a self-defeating improvement caused by an increase of downstream estimation error. As before, the upstream model, v, predicts a 2D position for each data point in the example. The downstream model w is selected from the set, Har of quadratic classifiers. It is tasked with performing binary classification into positive class (green color) and negative class (red color) based on the 2D positions predicted by the upstream model. To train the binary classifier, the downstream model (and w/) is provided with N... 6 labeled training examples (the green and red markers the plot). In the example, the upstream model e' performs the predictions of e' (right plot) match the ground-trui
better than its original counterpart targets (left plot) more closely than those of u (middle plot). However, the upstream improvement hampers the downstream model even though the optimal downstream model wt is in the hypothesis set H.. both before and after the upstream improvement. After the upstream improvement, the optimal downstream model that can be selected from the hypothesis set, w/ is more complex, which makes it harder to find it based on the finite number of N training examples. This results in an increase in the estimation error, which causes the self-defcating improvement in Figure
3.2 Error Decomposition of System with Two Upstream Models
More complex types of self-defeating improvements may arise when a downstream model depends on multiple upstream models that are themselves entangled. Consider the system of Figure [ that has three models, V {u, v, w). and dependencies between upstream models u and v and downstream model w, that is, E {(u, w), (v, w)}. The error decomposition for this system is
5

---

Pusitive (groand truth)
Model before update (e, Model after update (u², w]
Negative (ground trith)
Upetresen model
Upstream model
Downsteeem model
Upstream model.
Upstream model
Dowustream model w
Upstream model
Upstream model
Downstream model wi
Upetream model Upstream model
Downstream model w²
Auti-correlated errors
(b) Correlated predictions
Figure Illustrations of self-defeating improvement due an increase the upstream compatibility error. Green areas should be labeled positive; red areas negative. Left (a and b): Original upstream model u (top; blue line and +. symbols) and its improved version u' (bottom; red). Middle (a and b): Upstream model Right (a and b): Original downstream model (top) and its improved version w' (bottom). In (a), the errors u and v are anti-correlated. As a result, the improved upstream model u' negatively impacts the re-trained downstream model w'. In (b), the improved upstream model u' is identical to the other upstream model v. As a result, w/ looses the additional information previously provided by u, negatively impacting its performance.
similar Equation but further decompose the upstream error. Denoting the Bayes-optimal downstream model by w", , the optimal model given upstream models u and v by w and the optimal upstream model given upstream model by vt. we decompose the upstream error as follows:
E (lo (u, v)) en (ww())
upstream errur
(lw (u, v)) eu (wol i o (u, w!Ji] + B[e (u)) e (w"))
upstream compatihility error
excess upstream
The excess upstream error is similar to the upstream error in Equation 2 upstream model may be suboptimal for the downstream task, for example, because of loss mismatch or distribution mismatch. The key observation in the error decomposition of a system with two upstream models is that the optimal upstream model v is a function of upstream model (and vice versa). The upstream compatibility error captures the error due to upstream model not being identical to the optimal vl self-defeating improvement can occur when we update upstream model u to u' because may be that vi A v which can cause the upstream compatibility error to increase.
We provide two examples of this in Figure 4 The first example (left pane) shows a self-defeating improvement due upstream models u and making anti-correlated errors that cancel each other out. The second example (right pane) is self-defeating improvement due to u' making more correlated predictions with u. We note that, in both examples, the optimal v depends on and the self-defeating improvement arises because of increase in the upstream compatibility error.
In the examples in Figure all three models aim to distinguish and red (negative class). Upstream models u and v do so based the (z, y)-location of points in the 2D plane. The downstream model operates on the (hard) outputs of the two upstream models. In Figure aa), aa), the original upstream models u and make errors that are anti-correlated. The original downstream model u exploits this anti-correlation make perfect predictions. When upstream model is replaced by an improved model u' that makes no errors, self-defeating improvement arises: downstream model w no longer receives the information needs to separate both classes perfectly, In Figure upstream model u is improved to u', which makes the exact same predictions as the other upstream model v. As result, downstream model w' no longer access to the complementary information that providing in addition to v, producing the self-defcating improvement.
classes: green (positive class)

---

Sterco image aa: rggr)
Dispariiy map
Psendo LiDAR
Disparity estimation model
Point cloud gemerator
Car
Pedestcian detectton model vo
detection
model
Detected causs (in 3D)
Detected pedlestriass (in 3D)
Figure 5: Overview of the car and pedestrian detection system used our case study in Section 4 The system consists of three models: (I) disparity-estimart model that generates EsDUdo-LiDAR representation of the environment, (2) a car-detection model e that operates on 3D point clouds, and (3) pecesstrinnddetecton model wo that performs 3D pedestrian detection.
While the examples in Figure may seem contrived, anti-correlated errors and correlated predictions can arise in practice when there are dependencies in the targets used to train different models. For example, suppose the upstream models are trained to predict ad clicks (will user click on an ad?) and ad conversions (will a user purchase the product being advertised?), respectively. Because conversion can only happen after click happens, there are strong dependencies between the targets used to train both models, which may introduce correlations between their predictions.
4 Case Study: Pseudo-LiDAR for Detection
We perform a case study on self-defcating improvements in a system that may be found in self-driving cars. In both cases, the system pedestrians. Figure gives an overview of the models in the system:
LiDAR [37] as the basis for 3D detection of cars and
A disparity-estimation model, u, predicts map d with disparities corresponding to every pixel in the left image of stereo image (a pair of rectified images from two horizontally aligned cameras). A point cloud generator, r, uses disparity map d to create pseudo-LiDAR representation of the
3D environment. The point cloud generator, r, a simple non-learnable module (37).
The stereo image and disparity map are input into model, that performs 3D detection of cars.
The same inputs are used in a separate model, w, that aims to perform pedestrian detection.
4.1 System Design
The disparity-estimaaio and detection models used in our case study are described below.
Car detection
Pedestrian detection P-RCNN F-PTNET
P-RCNN F-PTNET
Disparity estimation. To perform disparity es- timation, adopt the PSMNet model of Chang and Chen [1I. The model receives a stereo im- age, (X Xright) input and aims compute a disparity map, d = u(x ((x Xeigtt., that con- tains disparity estimate for each pixel in image Xleft We experiment with two training loss func- tions: (1) the depth mean absolute error and (2) the disparity mean absolute error (MAE). Given a ground trrtt disparity map. y, and a function that maps disparities to depths. g(d) = C for some camera constant C, the disparity MAE is proportional to ||d y and the depth MAR is proportional to (g) ()- g To evaluate the quality of model ur we measure disparity MAE:
39.91
33.64
34.42
43.02
AP3o
w'
38.75
32.78
27.64
43.10
68.06
55.46
55.19
62.16
u 50.34
43.79
38,51
50.49
APpeV
49.30
42,77
31.47
48.29
3/ 78.43
67.35
63.00
65.92
Table 1: Average precision of car detection and pedestrian detection measured for 3D box view (AP3p) and 2D birds-eye view (APBEv). Higher is better. Results are shown for baseline disparity- estimation model u and an improved disparity- estimation model u' that is trained using a differ- ent loss. The AP oracle disparity-estimation model is shown for reference. Improving the disparity-estimatio model leads to a self-defeating improvement in both the car detection model and the pedestrian detection model.
(u, D.,)) 1 Σ EM ||d -ylh. We expect that a model, u', trained to minimize disparity
7

---

MAE will provide better disparity estimates (per test loss ku) than a model, u, that is trained to minimize depth MAE. This has downstream effects on performance of the detection models.
Car and pedestrian detection. We experiment with two different models for performing detection of cars and pedestrians, viz., the Point-RCNN model (P-RCNN; Shi et al. (0)) and the Frustum PointNet detection model (F-PTNET; Qi et al. (26)). Both models take stereo image (Xieft Xright) and point cloud r(d) input. Before computing r(d), we perform winsorization on the prediction u(Xieft Xright): we remove all points that are higher than 1 meter in the point cloud (where the camera position the origin). Both 3D car detection model and 3D pedestrian detection model are trained to detect their target objects at any distance from the camera.
Following Geiger et al. [T6], we evaluate the test loss of car-detection model, Lv. using the negative (IoU) of 0.7. The test loss of the pedestrian- detection model, fur, is the negative AP for an IoU of 0.5. The pedestrian-detection test-loss is only evaluated on pedestrians whose distance to the camera is S 20 meters. We measure both the car-detection and the pedestrian-detection test losses for both the 3D box view (-AP3D) and the 2D bird-eye view (-APBEv); see Geiger et al. [16] for details on the definition of the test-loss functions.
average precision (AP) for an intersection-over-uiig
4.2 Experiments
We evaluate our system on the KITTI dataset [6 CC
Range
0-max
0-20
20-40
40-max
B BY-NC-SA 3.0]. using the valliainvaiiiationn split of [12).
Our baseline Hisparity-estimation model, u, that is trained to minimize depth MAE obtains a test loss of 1.28 (see Table D. The improved version of that model, u', trained to minimize disparity MAE and obtains a test loss of 1.21, confirming the model improvement.
Model
1.28
1.32
1.12
1.11
Model u''
1.21
1.21
1.20
1.20
Table 2: Disparity MAE test loss of disparity models u. (trained to minimize depth MAE) and u' (trained to minimize disparity MAE), split out per range of ground-truth depth values (in meters). Model better predicts the disparity of nearby points, but u' works better on distant points.
Table [ presents the test losses of the downstream models for and pedestrian detection, for both the baseline upstream model u and the improved model u'. We observe that the improvement in disparity estimation leads to self- defeating improvements on both the car-detection and the cedestrian-detection tasks. The AP3D of the car detector drops from 39.91 to 38.75 (from 50.34 to 49.30 in AP For pedestrian detection, AP3D is unchanged but APBEV drops from 50.49 to 48.29. Interestingly, these self self-defcating improvements are due to increases in different error terms.
Self-defeating improvement in car detection. The self- defeating improvement observed for car detection is likely due to model
P-RCNN F-PTNET
an increase in the upstream error. A disparity-estimation that is trained to minimize depth MAE (i.e., model u) focuses on making sure that small disparity values (large depth values), are predicted correctly. By contrast, model trained to minimize disparity MAE (model u') aims to predict large disparity values (small depth values) correctly. sacrificing accuracy in predictions of large depth values (see Table 2 This negatively affects the car detector because most cars tend to be relatively far away. In other words, the loss function that improves model deteriorates the downstream model because it focuses on errors that are less relevant to that model, which increases the upstream error.
30.53
44.10
AP
w/
34.20
46.14
y/ 55.19
62.16
34.09
51.84
AP
w/
38,83
53.99
64.25
65.23
Table 3: Average precision at IoU 0.5 of pedestrian detec- tion measured for 3D box view (AP:n) and 2D birds-eye view (APnev) for two different detec- tion models (P-RCNN and F- PTNET). Pedestrian detector is trained by removing pedestrians farther than 20 meters. Results are shown for the baseline dis- parity estimation model, u, the improved model, u', and an ora- cle model, y. Higher is better.
Self-defeating improvement in pedestrian detection. Different from car detection, the self-defeating improvement in pedestrian detection is likely due to an increase in the downstream approxima- tion or estimation error. showes that whereas the disparity MAE test loss of u' increases for large depth values, it decreases in the 0-20 meter range. As the sndestrian-detection evaluation focuses on pedestrians within 20 meters, we expect the pedestrian detector w' to improve along with disparity-estimatinn model u': that is, the upstream error likely decreases.
8

---

However, the point representations produced by model u' are likely noisier for far-away pedestrians than those produced by u (see 2 Because the downstream model trained to (also) detect these far-away pedestrians, the noise in the pedestrian point clouds makes it harder for that model to learn a good model of the shape of pedestrians. This negatively affects the ability of w' to detect nearby pedestrians. Indeed, if the pedestrian detector is only trained on nearby pedestrians, the self-defeating improvement disappears; see Table 3 Hence, the observed self-defeating improvement was due to an increase in a downstream error term: the upstream model did improve for the downstream task, but the downstream model was unable to capitalize on that improvement.
Related Work
This study part of larger body of work around machine-learning engineering 9 21 23 Sculley
et al. (28) provides a good overview of the problems that machine-learning engineering studies.
The self-defeating improvements studied this paper are due to the entanglement problem [1 where "changing anything changes everything" [28). Entanglement has also been studied in work on
"backward-comie compattil" learners (29) 32 35). Prior work has also studied how the performance of
humans operating a machine-leammin system may deteriorate when the system is improved [3.
Some aspects of self-defeating improvements have been studied in other domains. In particular, the effect of an upstream model change on downstream model can be viewed as domain shift [4] 25
However, approaches domain shift such as importance weighting of training examples [3] are not directly applicable because they require the input space of both models to be identical. The study of self-defeating improvements also related the study of transfer learning, e.g.,
19 24]. Prior
work on transfer learning has reported examples of self-defcating improvements: e.g., 19 reports
that some regularization techniques improve the accuracy of convolutional networks on ImageNet, but negatively affect the transfer of those networks to other recognition tasks.
Finally, work differentiable programming [8 36 is relevant to the study of self-defeating improvements. In some situations, a self-defeating improvement may be resolved by backpropagating learning signals from model further upstream. However, such approach is not panacea as downstream tasks may have conflicting objectives. Moreover, technical or practical obstacles may prevent the use of differentiable programming in practice (see Section 2
6 Conclusion and Future Work
This study explored self-defeating improvements in modular machine- leaaiin- systems. We presented a new error decomposition that sheds light on the error sources that can give rise to self-defeating improvements. The findings presented in this study suggest plethora of directions for future work:
It may be possible to derive bounds on the error terms in Equation 2 that go beyond the current art. For example, it is well-known that the trade-off between approximation and estimation error leads to an excess risk that scales between the inverse and the inverse square root of the number of training examples [27 39]. New theory may provide more insight into the effect of the upstream hypothesis set and training set size on those errors, and may bound the upstream error term in ? Our study focused on first-order entanglement between two models (either an upstream and downstream model or two upstream models). Some settings may give rise to higher-order entanglements, for example, between three upstream models whose errors cancel each other out. Our definition of different types of entanglement may pave the way for the development of diagnostic tools that identify the root cause of self-defeating improvement.
We need to develop best practices that can help prevent self-defcating improvements from occurring. One way to do may be by tightening the "API" of the machine- learning models, e.g., by calibrating classifier outputs [7]] or whitening feature representations [20]. Such transforms constrain model's output distribution, which may reduce the downstream negative effects of model changes. Alternatively, we may train models that are inherently more robust to changes in their input distribution, e.g., using positive-congruent training (38) meta-leaming [[5].
We also need to study self-defeating improvements of other aspects. An important open question is K Can making an upsiream model fairer make its downstream dependencies less fair?
We hope this paper will encourage the study of such questions, and will inspire the machine-learning community to rigorously study machine-learnin systems in addition to individual models.

---

Acknowledgements
The authors thank Yan Wang, Yurong You, and Ari Morcos for helpful discussions and advice.
References
[1] S. Amershi, A. Begel, C. Bird. R. DeLine, H. Gall, E. Kamar, N. Nagappan, Nushi, and T. Zimmermann. Software engineering for machine learning: A case study, In Proceedings of the International Conference on Software Engineering, 2019.
[2] H. Azizpour, Razavian, J. Sullivan, A. Maki, and Carlsson. Factors of transferability for a generic convnet representation. IEEE Transactions Pattern Analysis and Machince Intelligence, 3899):190-1882,, 2016.
[3] G. Bansal, B. Nushi, E. Kamar, D. S. Weld, W. S. Lasecki, and E. Horvitz. Updates in human-Al teams: Understanding and addressing the performance/compatiiity tradeoff. In P Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2429-243) 2020.
[4] S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79:151-175, 2010.
[5] L. Bottou and O. Bousquet. The tradeoffs of large-scale learning. Advances in Neural Information Processing Systems (NeurIPS), pages 161-168, 2008.
[6] S. Boucheron, O. Bousquet, and G. Lugosi. Theory of classification: survey of recent advances. ESAIM: Probability and Statistics, 9:323-375, 2005.
[7] J. Brookshear. Computer Science. An Overview (9th edition). Addison- Addison-Wesley, 2009.
[8] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, Herbert- Vosser G. M. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, B. Chess, J. Clark, C. Berner, McCandlish, A. Radford, I. Sutskever, and D. Amodci. Language models are few-shot learners. In Advances in Neural Information Processing, 2020.
[9] A. Burkov. Machine Learning Engineering. True Positive Inc., 2020.
[10] R. Caruana. Multi-task learning. Machine Learning. 28:41-75, 1997.
[11] J.-R. Chang and Y.-S. Chen. Pyramid stereo matching network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5410-5418, 2018.
[12] X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and R. Urtasun. 3D object proposals for accurate object class detection. In Advances Neural Information Processing Systems, 2015.
[13] C. Dwork and C. Ilvento. Fairness under composition. In arXiv: 06100612222222 2018
[14] A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffman, F. Hormozdiari, N. Houlsby, S. Hou, G. Jerfel, A. Karthike- salingam, M. Lucic, Y. Ma, C. McLean, D. Mincu, Mitani, A. Montanari, Nado, Natara- jan, C. Niclson, T. F. Osborne, R. Raman, K. Ramasamy, R. Sayres, J. Schrouff, M. Seneviratne, Sequeira, H. Suresh, V. Veitch, M. Vladymyrov, X. Wang, K. Webster, S. Yadlowsky, T. Yun, X. Zhai, and D. Sculley. Undersppecification presents challenges for credibility in modern machine learning. In arXiv:2011 2020.
[15) C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning, pages 1126 1135, 2017.
[16] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012.
10

---

[17] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks, In Proceedings of the International Conference on Machine Learning, pages 1321-1330. 2017.
[18] M. Innes, A. Edelman, K. Fischer, C. Rackauckas, E. Saba, V. Shah, and W. Tebbutt. A differentiable programming system to bridge machine learning and scientific computing. In arXiv:1907.07587. 2019.
[19] S. Kornblith, J. Shlens, and Q. V. Le. Do better ImageNet models transfer better? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2661-2671, 2019.
[20] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[21] I.. Lwakatarea, A. Raj, I. Crnkovica, Bosch, and H. H. Olsson. Large-scale machine learning systems in roal-world industrial settings: review of challenges and solutions. Information and Software Technology, 127, 2020.
[22] B. Nushi, E. Kamar, E. Horvitz, and D. Kossmann. On human intellect and machine failures: Troubleshooting integrative machine learning systems. In Proceedings of AAAI Conference on Artificial Intelligence, 2017.
[23] I. Ozkaya. What is really different in engineering Al-enabled systems? IEEE Software, 37(4), 2020.
[24] J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 22:1345-1359 2010.
[25] V. M. Patel, R. Gopalan, R. Li, and R. Chellappa, Visual domain adaptation: A survey of recent advances. IEEE Signal Processing Magazine, 32:53-69, 2015.
[26] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas. Frustum pointnets for 3D object detection from RGB-D data. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 918-927, 2018.
[27] C. Scovel and I. Steinwart. Fast rates for support vector machines. In Proceedings of the Conference on Learning Theory (COLT 2005), pages 279-294, 2005.
[28] D. Sculley, G. Holt, D. Golovin, E. Davydov, T. Phillips, D. Ebner, V. Chaudhary, M. Young, J.-F. Crespo, and D. Dennison. Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems, 2015.
(29] Y. Shen, Y. Xiong, W. Xia, and S. Soatto. Towards backward-compatibbe representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3368-6377 2020.
[30] Shi, X. Wang. and H. Li. PointRCNN: 3D objject proposal generation and detection from point cloud. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 770-779, 2019.
[31] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90:227-244, 2000.
[32] M. Srivastava, B. Nushi, E. Kamar, S. Shah, and E. Horvitz. An empirical analysis of backward compatibility in machine learning systems. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
[33] V.N. Vapnik. Estimation of Dependences Based on Empirical Data. Peringer-Verlagee Berlin, 1982.
[34] C. Wan, S. Liu, H. Hoffmann, M. Maire, and S. Lu. Are machine learning cloud apis used correctly? In International Conference on Software Engineering (ICSE), 2021.
[35] C.-Y. Wang, Y.-L. Chang, S.-T. Yang, D. Chen, and S.-H. Lai. Unified representation learning for cross model compatibility. In Proceedings of the British Machine Vision Conference, 2020.
11

---

[36] F. Wang, J. Decker, X. Wu, G. Essertel, and T. Rompf. Backpropagation with callbacks: Foundations for efficient and expressive differentiable programming. In Advances in Neural Information Processing Systems, volume 31, 2018.
[37] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo- LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8445-8453, 2019.
[38] S. Yan, Y. Xiong, K. Kundu, S. Yang. S. Deng, M. Wang, W. Xia, and S. Soatto. Positive- congruent training: Towards regression-free model updates. In arXiv:2011.09100 2021.
[39] Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 32:56-8 85, 2004.
12


Reviews:
[{"review_id": "lISCOhlOKmr", "reviewer": "Reviewer_RhPc", "summary": "This paper explores, \"self-defeating improvements, potential negative\neffects of improving a part of a module machine learning system. The\ncauses of such self-defeating improvements are analyzed through Bayes\nerror decomposition. The paper presents a case study with pedestrian\nand car detection where both tasks rely on an upstream disparity\nestimation model. The results show that despite improvements in\ndisparity estimation (when tested alone), the overall system\nperformance degrades.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explores a relatively less studied problem which will be\nmore relevant as the number of more complex (and modular) machine\nlearning systems increase. The main contribution  of the current paper\nis demonstrating the problem, and providing an analysis of sources of\nthe self-defeating improvements.\n\nThe following of some points (roughly ordered by importance) that I\nmiss or think would be improve the paper.\n\n- I think raising awareness and promoting discussion are reasonable\n  contributions. However, paper would be stronger if included some\n  concrete discussion towards solutions (e.g., diagnostics for\n  detecting the problems, or suggestions to avoid them).\n- It is not clear to me whether the case study is a constructed\n  demonstration, or an example that the authors happened to observe in\n  a real study.\n- Although the authors state in the checklist that they did not report\n  variation (error bars) because it was low, I think it would still\n  help reader decide the extent of the differences if the variation\n  was presented in Table 1 and Table 3. Some of the scores in Table 1\n  are particularly close.\n- The paper is rather dense. In particular, the description of the\n  case study is difficult to understand people outside the field\n  without consulting quite a few additional sources. I think it is\n  important for the current paper to aim for a broader audience.\n- Part of the \"motivation and problem setting\" subsection (line 102)\n  is already stated in the introduction, which, I think, is a better\n  place for motivation anyway.\n- Footnote marks should follow punctuation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the limitations of the work are adequately stated in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-8"}, "extra_text": {}}, {"review_id": "NvGFT8Fp83X", "reviewer": "Reviewer_QoL7", "summary": "This paper investigates a practical problem referred to as self-defeating improvements when developing machine learning systems for real world applications such as self-driving cars or virtual assistants. That is, given a machine learning system with multiple components depending on each other, developing one component independently could sometimes make the overall system worse. This work first studies the problem in a system with two models (upstream task and downstream task) and decomposes the error into upstream error, downstream approximation error, and downstream estimation error. Building on top of the formulation, this work expands the discussion on a system with two or more upstream models. Finally, it demonstrates the emergence of self-defeating improvements in a realistic pseudo-LiDAR detection system.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* [S1] This paper studies a very important problem in developing machine learning systems in the real world with several dependent modules. This work has a clear definition and formulation of the problem.\n* [S2] This work decomposes the self-defeating error into three components and provides analysis for each of them. The reviewer finds figure 2-4 very illustrative.\n* [S3] This work also demonstrates the real case in the Pseudo-LiDAR platform for object detection. This shows the real-world impact of the problem this paper studies. \n\nWeaknesses:\n* [W1] The actual impact is still limited as this work does not provide a solution to the problem. \n* [W2] While the error decomposition is interesting, the reviewer would expect to see such decomposition in the case study. Is it possible to quantify the error decomposition in the experiments? Please elaborate on this in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N / A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "I7zSsFQT31V", "reviewer": "Reviewer_JHen", "summary": "Authors present evidence that improvements in models that are components of larger system may lead to worse results for the whole system or downstream models. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows that in a system with multiple machine learning models improving some models may lead to deterioration of results of the whole system or models dependent on improved models.\n\nAuthors present problem and some formalization of it. They do not suggest solutions for the problem. Authors also show , illustrative examples and examples from real world.\n\nI believe that the issue of \"self-defeating improvements\" is known. Authors do an adequate job of presenting it in general and via examples. \n\nThe formalization of problem seems to be overly complicated perhaps by the choice of math notation. It is not clear that the formalization as presented provides much benefit. This is possibly made worse by authors' choices in the notation.\n\nPlease do not use the same letters (u, v, w) to denote concrete nodes in the graph (see Figure 1) and as abstract variables in formulas. This \"reuse\" of the letters makes the text unnecessarily hard to read, because reader has to continuously check if you are referring to an abstract variable or to the concrete node in the graph.\n\nIt seems that the notation in (2) is not the same as notation in (3). Overall notation seems to be non-intuitive. For example, why name \"optimal model conditioned on an upstream model\" w-dagger instead of showing the conditioning via subscript or superscript? Especially since in (3) you use w-dagger-with-u,v-subscript? The notation can be better thought out so it would be more readable and understandable.\n\nWhat does \"anti-correlated\" mean? Is this defined mathematically or just an informal expression?\n\nLine 84: what do you mean that test set of downstream model is fixed even though inputs of it may change. If inputs change, how can you say that the test set is fixed?\n\nLine 212: I don't think you can say that \"optimal v depends on u\". The right expression might be that \"v output that is beneficial to w depends on u\". Or you should distinguish v optimality in general vs v optimality for w given u.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors present the limitations of their work. The major limitation is that they do not propose solutions, just identify and present the problem. Although presenting the problem is useful, the presentation is not particularly novel.\n\nAuthors mention some societal impacts related to their work, i.e. possibility of self-defeating improvements in bias (i.e. improving bias of subalgorithm possibly making system more biased). This is worthwhile societal observation. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "AJPbJJaAQ3J", "reviewer": "Reviewer_ARBJ", "summary": "The paper asks a question about performance of a system that uses multiple independently trained models, and studies cases where improvements to a model makes overall performance worse (due to errors in other components)", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper explores an interesting point, but I'm not sold on the formalism or its connection to the experiments.  The exposition was also not great, IMO.\n\nSection 3: The discussion feels misleading to me.  The claim that improvements to a module are guaranteed to improve the entire system for traditional software systems seems generally wrong.  Even efficiency improvements to software aren't guaranteed to improve the whole system - maybe Cloudflare crashes when another component becomes so fast it dos-es it.  I don't see any clear key distinction between software and ML - the examples in section 4 seem pretty modular.  \n\n3.1 \nEquation 2:  A number of things that would've made this easier to understand:\n- It was confusing that w* was described as a downstream model when it doesn't seem to take outputs from an upstream model.\n- I would've appreciated a more straightforward pointer or an in-text explanation of the \"standard decomposition[s]\".\n- The notation for optimal model condition on an upstream model feels like it should have a v in it - maybe w*(v) or something.  This seems especially important in the discussion of upstream error.  I can't immediately tell exactly what the equation there is saying (though I have a good guess)\nIn this section, I also expected a discussion of (v, w) vs. (v', w) and which terms would be expected to increase.  Overall I don't have a clear takeaway of how this decomposition helps clarify much about self-defeating improvements.\n\n3.2 \n\nI'm not convinced you need two upstream models for things to be interesting. Couldn't a similar anti-correlation of errors issue occur between an upstream and downstream models?\n\n4\n\nBeing unfamiliar with the self-driving field, it's not clear to me whether minimizing depth/disparity MAE and maximizing average precision are aligned objectives.  Is it the case that having optimal models separately for minimizing depth MAE and maximizing average precision would yield optimal performance on the KITTI dataset?  If not, then it seems important to also disentangle the error from poor choice of loss function.  I can't tell that depth MAE isn't just better than disparity MAE and that you didn't simply optimize the wrong (or a wronger) thing.\n\nOverall, I'm not sure I had a clear takeaway from this section, as I think it's fairly obvious one could construct scenarios where this sort of thing happens.  I think the most interesting fact is that it occurred in a realistic setting, which is definitely a point practitioners should definitely be aware of.  I would've liked to see some early steps towards predicting how these errors occur and how to mitigate them.\n\n5\n\nIs backpropagating really fundamentally different at all than just alternating optimization steps between the upstream/downstream models?  It seems like either way you run into local optima issues potentially.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "maybe could be more explicit about how the problems mentioned could cause real world harm", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "1eZkHm9LUhW", "reviewer": "Reviewer_yjBZ", "summary": "This paper studies the problem of \"self-defeating improvement\" (improving an upstream ML model deteriorates a downstream ML model in the same system). Leveraging Bayes error decomposition, it decomposes the error of a system with multiple ML models into: (excess) upstream error, upstream compatibility error (when more than one upstream model is used), downstream approximation error, and downstream estimation error. The paper conducts a case study in Pseudo-LiDAR for vehicle/pedestrian detection to demonstrate the existence of the studied \"self-defeating improvement\" in real world.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- well motivated and important topic\n- well-written; intuitive and clean formulation of the problem\n- good visualization of the formulation on toy examples\n- case study demonstrates the existence of the studied problem in practice\n\nCons:\n- only one case study is conducted\n- no systematic method is presented to determine which component of the error is the cause of a certain \"self-defeating improvement\"\n\nThis paper studies an important topic of \"self-defeating improvement\", which seems to be relatively less formally studied in the field. Overall, I find this paper is a pleasure to read and believe it will encourage more exploration on this important topic.  One minor weakness is that the paper only conducts one case study so I am not sure how common the studied \"self-defeating improvement\" is in real-world ML systems.\n\nIn the case study, the paper analyzes the potential causes for the deterioration of detection for vehicle and pedestrian. Regarding the detection for pedestrian, I wonder if it is possible to distinguish between downstream approximation error and downstream estimation error? Is there a possible systematic way to decompose the error based on the formulation in real-world application like the one in the case study?\n\n\n\nOther comments:\n- Regarding fairness in the final bullet point in Conclusion and Future Work, it might worth looking into \"Fairness Under Composition\" by Cynthia Dwork and Christina Ilvento,\n\n\nQuestions:\n- Is it possible to also make a visualization for the upstream error similar to Figure 2 and Figure 3? \n- For Figure4(a), I am a bit confused here. Are the ground-truth for u, v, and w different? Are you using the same color code here as in Figure2 and Figure3?\n----------------------------------------------------------------------------------------------------------------------------------------\nThe author's response addresses my concerns and I will keep my initial score.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers gave favorable scores, one borderline, and one negative (ARBJ). The last reviewer engaged in a productive discussion with the authors, so we can expect the final paper to be improved. The paper addresses a important question and has been refined a lot since it was first submitted to ICML 2021, so it deserves to be published now.

The paper addresses a problem that is important and will be of interest to a broad audience. The issue is that in a system that uses multiple machine learning models, improving the accuracy of some component models may lead to worse results for the system as a whole. The paper explains the problem and some formalizations of it. Although solutions are nor provided, illustrative examples and cases from the real world are described.
Decision: Accept (Poster)