Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explores, "self-defeating improvements, potential negative
effects of improving a part of a module machine learning system. The
causes of such self-defeating improvements are analyzed through Bayes
error decomposition. The paper presents a case study with pedestrian
and car detection where both tasks rely on an upstream disparity
estimation model. The results show that despite improvements in
disparity estimation (when tested alone), the overall system
performance degrades.

main_review: The paper explores a relatively less studied problem which will be
more relevant as the number of more complex (and modular) machine
learning systems increase. The main contribution  of the current paper
is demonstrating the problem, and providing an analysis of sources of
the self-defeating improvements.

The following of some points (roughly ordered by importance) that I
miss or think would be improve the paper.

- I think raising awareness and promoting discussion are reasonable
  contributions. However, paper would be stronger if included some
  concrete discussion towards solutions (e.g., diagnostics for
  detecting the problems, or suggestions to avoid them).
- It is not clear to me whether the case study is a constructed
  demonstration, or an example that the authors happened to observe in
  a real study.
- Although the authors state in the checklist that they did not report
  variation (error bars) because it was low, I think it would still
  help reader decide the extent of the differences if the variation
  was presented in Table 1 and Table 3. Some of the scores in Table 1
  are particularly close.
- The paper is rather dense. In particular, the description of the
  case study is difficult to understand people outside the field
  without consulting quite a few additional sources. I think it is
  important for the current paper to aim for a broader audience.
- Part of the "motivation and problem setting" subsection (line 102)
  is already stated in the introduction, which, I think, is a better
  place for motivation anyway.
- Footnote marks should follow punctuation.

limitations_and_societal_impact: I think the limitations of the work are adequately stated in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates a practical problem referred to as self-defeating improvements when developing machine learning systems for real world applications such as self-driving cars or virtual assistants. That is, given a machine learning system with multiple components depending on each other, developing one component independently could sometimes make the overall system worse. This work first studies the problem in a system with two models (upstream task and downstream task) and decomposes the error into upstream error, downstream approximation error, and downstream estimation error. Building on top of the formulation, this work expands the discussion on a system with two or more upstream models. Finally, it demonstrates the emergence of self-defeating improvements in a realistic pseudo-LiDAR detection system.
main_review: Strengths:
* [S1] This paper studies a very important problem in developing machine learning systems in the real world with several dependent modules. This work has a clear definition and formulation of the problem.
* [S2] This work decomposes the self-defeating error into three components and provides analysis for each of them. The reviewer finds figure 2-4 very illustrative.
* [S3] This work also demonstrates the real case in the Pseudo-LiDAR platform for object detection. This shows the real-world impact of the problem this paper studies. 

Weaknesses:
* [W1] The actual impact is still limited as this work does not provide a solution to the problem. 
* [W2] While the error decomposition is interesting, the reviewer would expect to see such decomposition in the case study. Is it possible to quantify the error decomposition in the experiments? Please elaborate on this in the rebuttal.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Authors present evidence that improvements in models that are components of larger system may lead to worse results for the whole system or downstream models. 
main_review: The paper shows that in a system with multiple machine learning models improving some models may lead to deterioration of results of the whole system or models dependent on improved models.

Authors present problem and some formalization of it. They do not suggest solutions for the problem. Authors also show , illustrative examples and examples from real world.

I believe that the issue of "self-defeating improvements" is known. Authors do an adequate job of presenting it in general and via examples. 

The formalization of problem seems to be overly complicated perhaps by the choice of math notation. It is not clear that the formalization as presented provides much benefit. This is possibly made worse by authors' choices in the notation.

Please do not use the same letters (u, v, w) to denote concrete nodes in the graph (see Figure 1) and as abstract variables in formulas. This "reuse" of the letters makes the text unnecessarily hard to read, because reader has to continuously check if you are referring to an abstract variable or to the concrete node in the graph.

It seems that the notation in (2) is not the same as notation in (3). Overall notation seems to be non-intuitive. For example, why name "optimal model conditioned on an upstream model" w-dagger instead of showing the conditioning via subscript or superscript? Especially since in (3) you use w-dagger-with-u,v-subscript? The notation can be better thought out so it would be more readable and understandable.

What does "anti-correlated" mean? Is this defined mathematically or just an informal expression?

Line 84: what do you mean that test set of downstream model is fixed even though inputs of it may change. If inputs change, how can you say that the test set is fixed?

Line 212: I don't think you can say that "optimal v depends on u". The right expression might be that "v output that is beneficial to w depends on u". Or you should distinguish v optimality in general vs v optimality for w given u.


limitations_and_societal_impact: Authors present the limitations of their work. The major limitation is that they do not propose solutions, just identify and present the problem. Although presenting the problem is useful, the presentation is not particularly novel.

Authors mention some societal impacts related to their work, i.e. possibility of self-defeating improvements in bias (i.e. improving bias of subalgorithm possibly making system more biased). This is worthwhile societal observation.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper asks a question about performance of a system that uses multiple independently trained models, and studies cases where improvements to a model makes overall performance worse (due to errors in other components)
main_review: Overall the paper explores an interesting point, but I'm not sold on the formalism or its connection to the experiments.  The exposition was also not great, IMO.

Section 3: The discussion feels misleading to me.  The claim that improvements to a module are guaranteed to improve the entire system for traditional software systems seems generally wrong.  Even efficiency improvements to software aren't guaranteed to improve the whole system - maybe Cloudflare crashes when another component becomes so fast it dos-es it.  I don't see any clear key distinction between software and ML - the examples in section 4 seem pretty modular.  

3.1 
Equation 2:  A number of things that would've made this easier to understand:
- It was confusing that w* was described as a downstream model when it doesn't seem to take outputs from an upstream model.
- I would've appreciated a more straightforward pointer or an in-text explanation of the "standard decomposition[s]".
- The notation for optimal model condition on an upstream model feels like it should have a v in it - maybe w*(v) or something.  This seems especially important in the discussion of upstream error.  I can't immediately tell exactly what the equation there is saying (though I have a good guess)
In this section, I also expected a discussion of (v, w) vs. (v', w) and which terms would be expected to increase.  Overall I don't have a clear takeaway of how this decomposition helps clarify much about self-defeating improvements.

3.2 

I'm not convinced you need two upstream models for things to be interesting. Couldn't a similar anti-correlation of errors issue occur between an upstream and downstream models?

4

Being unfamiliar with the self-driving field, it's not clear to me whether minimizing depth/disparity MAE and maximizing average precision are aligned objectives.  Is it the case that having optimal models separately for minimizing depth MAE and maximizing average precision would yield optimal performance on the KITTI dataset?  If not, then it seems important to also disentangle the error from poor choice of loss function.  I can't tell that depth MAE isn't just better than disparity MAE and that you didn't simply optimize the wrong (or a wronger) thing.

Overall, I'm not sure I had a clear takeaway from this section, as I think it's fairly obvious one could construct scenarios where this sort of thing happens.  I think the most interesting fact is that it occurred in a realistic setting, which is definitely a point practitioners should definitely be aware of.  I would've liked to see some early steps towards predicting how these errors occur and how to mitigate them.

5

Is backpropagating really fundamentally different at all than just alternating optimization steps between the upstream/downstream models?  It seems like either way you run into local optima issues potentially.
limitations_and_societal_impact: maybe could be more explicit about how the problems mentioned could cause real world harm
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of "self-defeating improvement" (improving an upstream ML model deteriorates a downstream ML model in the same system). Leveraging Bayes error decomposition, it decomposes the error of a system with multiple ML models into: (excess) upstream error, upstream compatibility error (when more than one upstream model is used), downstream approximation error, and downstream estimation error. The paper conducts a case study in Pseudo-LiDAR for vehicle/pedestrian detection to demonstrate the existence of the studied "self-defeating improvement" in real world.
main_review: Pros:
- well motivated and important topic
- well-written; intuitive and clean formulation of the problem
- good visualization of the formulation on toy examples
- case study demonstrates the existence of the studied problem in practice

Cons:
- only one case study is conducted
- no systematic method is presented to determine which component of the error is the cause of a certain "self-defeating improvement"

This paper studies an important topic of "self-defeating improvement", which seems to be relatively less formally studied in the field. Overall, I find this paper is a pleasure to read and believe it will encourage more exploration on this important topic.  One minor weakness is that the paper only conducts one case study so I am not sure how common the studied "self-defeating improvement" is in real-world ML systems.

In the case study, the paper analyzes the potential causes for the deterioration of detection for vehicle and pedestrian. Regarding the detection for pedestrian, I wonder if it is possible to distinguish between downstream approximation error and downstream estimation error? Is there a possible systematic way to decompose the error based on the formulation in real-world application like the one in the case study?



Other comments:
- Regarding fairness in the final bullet point in Conclusion and Future Work, it might worth looking into "Fairness Under Composition" by Cynthia Dwork and Christina Ilvento,


Questions:
- Is it possible to also make a visualization for the upstream error similar to Figure 2 and Figure 3? 
- For Figure4(a), I am a bit confused here. Are the ground-truth for u, v, and w different? Are you using the same color code here as in Figure2 and Figure3?
----------------------------------------------------------------------------------------------------------------------------------------
The author's response addresses my concerns and I will keep my initial score.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: Three reviewers gave favorable scores, one borderline, and one negative (ARBJ). The last reviewer engaged in a productive discussion with the authors, so we can expect the final paper to be improved. The paper addresses a important question and has been refined a lot since it was first submitted to ICML 2021, so it deserves to be published now.

The paper addresses a problem that is important and will be of interest to a broad audience. The issue is that in a system that uses multiple machine learning models, improving the accuracy of some component models may lead to worse results for the system as a whole. The paper explains the problem and some formalizations of it. Although solutions are nor provided, illustrative examples and cases from the real world are described.
Decision: Accept (Poster)