Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of algorithmic triage, where a predictive model and human experts together generate predictions. The authors introduced a theoretical analysis to model the interplay between the prediction accuracy of the model and the human experts. The authors later introduced a gradient based algorithm to find a sequence of predictive models and triage politics of increasing performance.  Extensive experiments on both synthetic dataset and two real world datasets showed improvement of proposed method over baseline methods. 
main_review: This paper studies a very interesting problem, where human experts can work together with predictive models to generate predictions. Relevant but slightly different from active learning, which tries to generate better predictive models with humans in the loop, the algorithmic triage problem targets at minimizing the overall loss with constraint to cost on human experts.
 
However, it might still be good to include more discussions on how active learning based methods would perform in this setup. 

The paper is well written and easy to follow. The authors also provided theoretical analysis and support for their proposed method, which seems to be technically sound and efficient in providing policy and predictive models given cost constraints. 

The author conducted experiments on synthetic datasets to show how the proposed method works and then showed the improvement on two real world datasets. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers the setting of "algorithmic triage", where a predictive model can choose to route some data points to a human expert for manual review.  The authors formally define the settings under which triage is useful, and show a motivating result that training a model for non-triage settings may be sub-optimal for learning under triage.  They then derive the optimal triage policy as a deterministic threshold rule, show a gradient-based algorithm for this, and demonstrate strong empirical performance on several synthetic and real datasets.
main_review: This paper proposes an approach to algorithmic triage where the goal is to learn both (i) a predictive model (ie the standard classifier) and *also* (ii) a triage policy model which determines whether the predictive model or a human expert should label a given instance, by optimizing the overall loss subject to some limit on human annotator time.  The paper:
- (i) Formalizes the notion that expert annotator uncertainty, as a function of the data, needs to be taken into account to optimize the loss in Prop. 1
- (ii) Establishes formally that a model worse than human experts on some instances will always benefit from an optimal triage policy (which seems like a trivial statement...?)
- (iii) Establishes that the optimal model without triage is *not* the optimal model under triage- a very interesting result and great motivation for this line of work!
- (iv) Proposes an iterative gradient/MC-based algorithm for estimating the model and triage policy pair, which they show in Prop. 6 is locally optimal under mild conditions
- (v) Validates the approach on a range of synthetic and empirical experiments

Overall this is a strong and thoroughly written paper that motivates the setting well, proposes a sensible algorithmic approach, and shows strong empirical results.  Weaknesses include the fact that there is only local convergence shown formally for the actual proposed algorithm; the experimental setup could be better described, especially the description of exact implementation of baselines; and more ablations exposing limitations of the proposed method would have been very helpful.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a methodology for learning models when some of the prediction tasks are deferred to human experts (i.e., triage). 
In the paper, the authors: (i) first derive the optimal level of triage for a given a model; (ii) then, they provide conditions under which models learnt under full automation will achieve lower predictive performance in presence of triage; (iii) then, they propose an iterative procedure to learn model and optimal level of triage jointly; (iv) lastly, they empirically verify the proposed algorith via a simulation and two case studies of real-world datasets. 
The key contributions of this paper are: (1) theorem 3, i.e., the analytical derivation of the optimal triage level; (2) the proposed methodology to learn the model under triage. 
main_review: Overall, I found the paper to be well written, except for some typos and lack of clarity in the figures (see below). In particular, I found the introduction of the framework to be very clear (initial part of section 2). 
The results were interesting--especially theorem 3--and the derivation of most of them was fairly simple, which is a plus. 
The results also seem to be novel, but I am not fully aware of all works in this part of the literature and in particular I only have passing knowledge of [6]. 

One minor concern: I am confused by the definition of \mu_h(x) in line 90-91: One may be interested in the majority prediction (in case of binary classification) or in the average prediction (in case of regression), so I do not fully understand why \mu_h is defined as the argmin of the same loss (with respect to h) that appears in (2). In addition, the proof of proposition 1 seems to rely exactly on \mu_h(x)=E[h(x)], which I think it may not correspond to the \mu_h as defined in lines 90-91 (e.g., it does for the squared loss but may not for other types). That said, I found the result in this proposition to be interesting. 

Other minor issues and suggestions:
* line 28: reference [7] is not about models having higher predictive performance than humans
* lines 34-36: the sentence seems slightly confusing
* line 39: algorithm*s*
* when the paper is the subject of the sentence, sometimes it is taken as singular and other times as a plural noun (line 40 vs. 42)
* another perhaps useful reference for the learning to defer literature is [Madras et at., 2017]
* line 107: it become*s*
* line 147: the loss should decrease, not increase (the inequality in the proof seems to be in the correct direction)
* figure 1: the caption is fairly clear (if one also carefully reads the main text), but the legend is not. For example, do the four columns correspond to models 1-4? In particular, I found confusing the fact that in this figure the color refers the noise whereas in figure 3 it refers to the triage level. There is also a similar issue for the tone
* line 210: there is no panel (a) in figure 1

Madras, David, Toniann Pitassi, and Richard Zemel. "Predict responsibly: improving fairness and accuracy by learning to defer." arXiv preprint arXiv:1711.06664 (2017).
limitations_and_societal_impact: The authors do not discuss the potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method triaging  / deferring examples to human labellers in machine learning systems. Their approach is conceptually simple: begin with a model trained on the full dataset and then iteratively improve the triaged model by updating it on a subset of the data that excludes examples where the previous iteration was worst compared to the human labellers. The approach is justified through a series of theorems showing that, as long as there exist parts of the input space for which the human labelers outperform the model, then triaging offers the possibility of improving the (combined model & human) performance; and further, that gradient descent can improve the model on the restricted space implied by the thresholding approach.

This is not my area, but I generally found the paper to be clear and well written (aside from style issues discussed below). 
main_review: **[Building a sequence of triage policies]** I understand that we can sequentially improve the combined policy of the models an humans up to our budget by retraining after excluding the subset of examples that the previous iteration performed worst on, but, assuming I know $b$, it wasn't clear whether that sequence is of length two (i.e. train and unconstrained model and then retrain with the subset implied by $\max(\lceil(1−b) |D|\rceil,p)$), or whether I sequentially retrain with a sequence of $b_i \leq b$? And if so, how do I choose that sequence? 

**[Evaluation]** I really would have liked to see more than two datasets evaluated. This is a generic method that seems to apply to any dataset, so aside from compute requirements, there doesn't seem to be anything preventing an evaluation that check whether the observed improvements over prior approaches hold across a statistically significant number of datasets?

**["Mathiness" style issues]** While the prose is generally well-written, the presentation of the propositions in section 3 (and to a lesser extent in the other sections), runs dangerously close to what Lipton & Steinhardt [2018] call 'mathiness' to "bulldoze rather than to clarify". I don't think that the authors intended this---and these comments haven't influenced my review score---but the paper could be a lot clearer if the propositions included a sentence or two on why each statement is true. For example, Prop 4 is prefaced with, "we can *identify the circumstances* under which the optimal predictive model... is suboptimal under algorithmic triage. Formally, *our main result* is the following Proposition..." (my emphasis). That sounds mysterious - when the "circumstances" are simple - as long as the thresholded gradients don't sum to zero, the model is suboptimal under triage. Making it clear that you're mostly just using the fact that loss and gradients on a subspace are typically not the same as those on the whole space would make the whole presentation easier to follow. 

To be clear - I'm very happy for theorems to formally delineate an argument, but as far as possible, we should use them to tell the reader why something is true, not just what is true.
limitations_and_societal_impact: These are relatively small issues, but I would have liked to see some discussion on them:

[Interpolation regimes] Modern deep networks are often massively over-parameterized such that they can interpolate the training data (driving the loss essentially to zero on each training example); essentially memorizing training examples, while still generalizing. It seems like the method would fail in this setting because you wouldn't threshold anything? Can the approach be generalized to select which examples to triage on a validation set to avoid this issue?

[Non-convexity] It is not clear how the sequential approach that is presented here interacts with non-convexity? Is there a concern that starting the optimal model on the full dataset and then thresholding can lead you to a worse solution than you would have found if you could optimize for the thresholded loss directly? I see this isn't an issue in the convex setting, but I'm curious whether you've thought about it in the non-convex setting?
review_title: null

Meta-review: Strengths:
- Sensible algorithmic approach
- Sound theoretical analysis for setup and algorithm
- Thorough experiments on both synthetic and real data

Weaknesses:
- Clarity could be improved, over-usage of math and notation
- In experiments - better describing baselines and adding ablation studies


Summary:

Reviewers are in agreement that this is a good paper and should be accepted. Most concerns regarded particular inclarities, but discussions were helpful in this respect. The authors also extended their local convergence result to a global results under some assumptions, which addressed a concern raised by one of the reviewers. 

Decision: Accept (Poster)