Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents and investigates methods for combining human and machine predictions. In the proposed setup, such a combination relies on instance-level probability outputs from a machine learning model and class-level information from human annotators. In a series of extensive experiments, the authors show that their method, which relies on model recalibration and confusion matrices, outperforms both humans and ML models. To support the strong empirical results, the authors demonstrate theoretically how this combination gains from the recalibration process.

The main contributions of this paper are its simple and sound methodology, its strong experimental analysis and its useful theoretical grounding. It is also clearly written and insightful.

main_review: This is a strong, clearly written paper that improves our understanding of an important subject. While it is not totally original in its approach (there is a plethora of existing work on combining humans and machines for better decision making), it is extremely clear in its approach and it makes several important contributions to this literature. 

Mainly, the proposed framework generalizes existing approaches to human-machine combinations and can be used in the future for other applications and with other algorithmic solutions. Second, the empirical analysis sheds light on the sensitivity of such approaches to model calibration, and specifically to overconfidence. Deep neural networks are notoriously over confident, and understanding how this affects their usefulness as decision support systems is crucial. Finally, the theoretical analysis is insightful and relevant, shedding light on the sensitivity to model overconfidence.

However, while I appreciate the authors’ acknowledgement of the paper’s limitations, I do think that more can be made to address them. First, the reliance on conditional independence is indeed problematic and might result in the approach being less useful when estimating conditional independence is infeasible. Second, it would be interesting to see the performance of the approach on datasets which do not involve image classification. Specifically, extending to NLP, where overconfidence is even more of a problem, would also be interesting. Third, it might be interesting to explore more advanced recalibration methods, or optimize for calibration in the first place (for example see Kumar et al 2018).

Finally, it is unclear whether the results are robust to other human selection methods. The authors choose humans randomly, but it is unclear if their method is sensitive to cases where each human annotates many examples and is biased towards specific types of decisions. I might have missed that in the paper, so perhaps better explaining the sensitivity to the human selection process in the CIFAR and ImageNet datasets might be helpful.

limitations_and_societal_impact: This paper can potentially have a substantial positive societal impact, as successfully combining human and machine predictions can improve decisions made in high-stakes settings. Moreover, better-understanding the effect of model calibration on downstream decisions supported by machine learning models can surely improve their safe dissemination in the wild. I believe that the authors have properly explained the benefits in the context of high-stakes applications, though more emphasis can be made on the importance of understanding model calibration. 

I also think that the authors have adequately addressed the limitations of their work, which are the focus on computer vision applications and their reliance on the conditional independence assumption.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The current work focuses on the interesting prospect and problem of combining human and machine classification information per input, especially in the small-data regime, for the sake of ultimately improving accuracy. The authors take a probabilistic approach to motivating their strategy for combining human-machine information, leveraging a carefully chosen prior to enable inference of a reliable human confusion matrix when the number of annotators is low. They ultimately find that the resulting combination (evaluated in the context of popular image classification tasks) improves accuracy over both humans and models alone, and with drastically less data than previous work. They show more formally that this success is in part driven by both the accuracy of humans/models as well as the confidence values of each.
main_review: To my knowledge, the methods employed, formal arguments, and conclusions are generally valid. The paper is atypically clear in explaining its aims, proposal, and evaluation. While I expect that these results will be robust, I would have preferred that the authors discuss them in the context of past work as well as different interpretation of the human behavioral datasets.

For example, the CIFAR-10H dataset contains over 500,000 judgements spanning thousands of annotators having viewed different subsets of images. Under the assumption that human annotators are noisy labelers with similar classification behavior, each labeler provides a sample from a "mental class-conditional distribution". However, if classification behavior is diverse, the resulting labels are more like argmax predictions from an ensemble of humans. In the former case, we can think of this sort of data as sort of representing a model, whereas in the latter, we might instead think of it as representing an ensemble of models in itself. The implications for the proposed method and its applicability given these two interpretations is not clear. Since the authors sample from single annotators, I think this is worth discussing.

Moreover, the original CIFAR-10H work performed a sort of validation of the soft labels by evaluating models trained on them on out-of-distribution datasets and against adversarial attacks. This was effectively a kind of argument that the confidence information conveyed in this sort of data is useful for classification. Given these results, why shouldn't we expect this confidence information to be useful in the current work? Further, why wouldn't the authors evaluate their hybrid method using OOD datasets, where accuracy on validation sets can be misleading as a target of combination?

Nonetheless, I think the work provides a unique contribution, and provides a good reason to be optimistic about leveraging these sorts of human datasets without having to scale dataset collection by a factor of n_annotators_per_image in a way that is prohibitively expensive.
limitations_and_societal_impact: The authors discuss plausible possible negative impacts that I did not foresee myself.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper "Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration" proposes a method to obtain a well-performing ensemble model of both an ML classifier (which outputs a probability for every class) and a human observer (who outputs a categorical decision). This is a setting that traditional ensemble methods do not account for; e.g. majority vote would not be possible here since there are only two decision makers in total (1 model, 1 human). The problem may not be common in standard ML but since ML applications are increasingly being used in real-world settings, there are areas where this approach can be applied (such as high-stakes medical decisions). Combining model and human predictions is shown to lead to lower error rates than either of the two alone.
main_review: ## Originality

According to the authors (lines 177ff) , this is the first method that combines a hard label prediction with a probabilistic label prediction. The related work section (Section 3) references relevant prior work, and clearly delineates how the paper differs from existing publications.

Minor comment: The authors write that "This work emphasizes the point that combinations of models that have diversity in how they make predictions can systematically outperform a single model." -> I'm not convinced that this point really needs emphasis, it's rather well-known that correlated predictors are bad, right? (e.g. multicollinearity in regression).


## Quality

The paper uses a nice combination of experimental and theoretical methods, and the results are presented clearly in text, figures and tables. Disclaimer: I am not an expert in this area. That being said, the results (e.g. lines 55f) are convincing despite the simplicity of the approach - for instance, if the achieved accuracy gain were the result of an architectural modification, it would be considered a significant improvement. Of course, this comes at the cost of obtaining human decisions, which is not feasible or practical in many cases, but for specific use cases described by the authors (e.g. medical diagnostics) it may well be useful. 

Equation 2 assumes conditional independence, which is an assumption that typically does not hold in practice, at least not completely (see Table 3 in the appendix) but the numerical results are good in spite of a potential violation of this assumption, so I don't consider this to be a big problem.

While the authors focus on the setting where human and machine decisions are combined, their approach is generally applicable to any two decision makers whenever there is a combination of two models, one with confidence scores for all classes and one without (i.e. only outputting a single class). In order to increase the potential appeal and applicability of the paper, it might be helpful to additionally showcase one example where instead of combining human and machine predictions, two machine predictions are combined in this way. This might be an interesting setting e.g. for cases where there is a proprietary model for which only the output decision (but not the confidence) is known, and it would additionally be a nice example for settings where there might be more than a handful (16) of classes.


## Clarity

The introduction is well written and clearly motivates the problem. The paper is well organized and structured.

While the authors perform multiple experiments testing calibration properties, it would be helpful to explicitly state the limitations of the approach: e.g. are there any limitations on the accuracy difference between the two decision makers that need to be met? At which point of poor calibration does the combined model perform worse than e.g. a human observer alone? How "different" / uncorrelated / ... do model and human predictions need to be?

Two minor comments regarding clarity:

- Figure 1 right: it would be helpful to mention the datasets below the two bar plot groups (I'm assuming the left one refers to CIFAR-H and the right one to ImageNet-16H?).

- I was confused about the sentence in line 98: why does a single observer have a confusion matrix? Is the observer shown the image multiple times and then decides differently? Or is this an aggregated confusion matrix over multiple images? Generally, how can you parameterise a probability with an entire confusion matrix? This became a bit more clear when looking at equation (3) but it would be helpful to make the entire paragraph starting with line 98 more precise / self-contained. I'm still not entirely sure about how the values for the confusion matrix are obtained.

- In equation (3), "i" is defined twice: once on the left-hand side of the equation and also again on the right-hand side as the sum counter; this is confusing

#### Code clarity

I appreciate that code was provided. A README would go a long way in making the code more accessible; currently it's mostly a list of files and scripts. The 20+ TODOs don't really help in providing confidence in the code quality: "Do these edge cases really need to be separated out??", "This isn't the right file for this", ... plus, as a suggestion for future code submissions, I would make sure that hard-coded paths are properly anonymized.

## Significance

As explained above, the paper addresses a relevant topic, more precisely, how human and model decisions may be combined. This is probably less relevant for tasks like the investigated one (such as standard ImageNet object recognition), but might become very relevant in the context of medical imaging (where peak performance truly matters, and spending the time of obtaining a human classification decision is a fair price to pay for improved performance).

## MISC

- it would be helpful to use a citation style etc. where one can click on a number in the paper and directly end up at the reference (currently this is not possible, so looking up a reference involves a lot of scrolling down)
- typo line 39: remove "being"
- humans are refered to as "non-probabilistic labelers"; perhaps a footnote could be added to explain that this refers to the fact that humans output a category instead of a probability vector. The term "non-probabilistic labeler" could lead to confusion if understood in the sense that humans would be deterministic and always choose the same label, i.e. that humans wouldn't have internal noise, which is not the case.
- typo line 146: remove "that"
- Table 2: for readers not familiar with the metrics, indicate whether lower=better or higher=better (and perhaps make the leading entries bold?)
- the paper sometimes refers to "human labelers", which may be confusing since usually "labelers" are those who generate labels for the dataset (rather than classify images). Perhaps rephrase accordingly?
limitations_and_societal_impact: In Section 7 of their paper, the authors discuss limitations and potential societal impact. In particular, I appreciated that the authors took the time to think about potential _negative_ societal consequences as well, which can often be difficult at first (especially for standard "pure research" papers). Well done!
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors introduce a simple method for combining predictions from a human and a model (or more generally a non-probabilistic and probabilistic labeler).  The resulting combination model is more accurate than either models or humans alone, and better calibrated than models alone.
main_review: Note: I am not familiar with this area of work.  

Overall, I think the problem statement is well-motivated and seems to be under-researched.  However, I would've liked to see more thorough experiments on the methods.

I think it would be natural to have baselines where: the model m(x) is finetuned on the labels h(x).  another variant is having m(x) re-trained on ground truth but where it can condition on h(x) for the portion of inputs h(x) is available.  I expect this should be worse in the low data regime but it's less clear in the 5000 datapoint regime.  (It would also be interesting to have humans that get to condition on model predictions, although that is understandably much more difficult to try.)

The combining method at its core is simple, but I would've liked to see ablations on the functional forms for confusion matrix estimation and calibration parameter estimation.  For example, learning K^2 parameters doesn't seem that bad in the 5000 data point regime.  It seems natural to parametrize with a low rank factorization as well.


Other Qs:

- In Figure 2, it's not clear to me why the original CIFAR models aren't already well calibrated on NLL, given that there is no distortion.  

- I might have misunderstood, but did Theorem 2 require the conditional independence assumption?  that seems pretty important to state, even if it's kind of obvious after thinking about it."
limitations_and_societal_impact: seems good
review_title: null

Meta-review: Strengths:
- Simple, well-executed approach
- Clever combination of machine and human strengths (probabilistic vs categorical predictions)
- Clear idea, clear writing

Weaknesses:
- Some unclarity regarding robustness of empirical results
- Needs discussion regarding individual vs. collective human-labeling behavior
- Authors should be more precise about limitations, and address them

Summary:

After some helpful discussions between the reviewers and the authors (including detailed responses by the authors, which the reviewers appreciated), reviewers are mostly in agreement that this is a strong submission which should be of interest to the community. Concerns mostly regarded the experimental setup. Here, reviewers were concerned about whether the empirical results would prove to be results under reasonable variations of the setup. Relatedly, reviewers also pointed out that results could be specific to how the authors chose to select labelers (at random), and asked the authors to clarify their reasoning behind this and they way in which they present their results in light of this experimental design decision. Authors are encouraged to clarify the issues regarding diversity in individual user behavior vs. collective user behavior, and how it relates to their experiments.
Finally, two reviewers were glad to see that the authors acknowledged some of the limitations of their approach, but at the same time, urged the authors to discuss them with more precision, and – more importantly – take steps to address them in the final version of the paper.

Decision: Accept (Poster)