Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a new method to identify "regions of disagreement" in a setting with human decision-makers assigned to individuals (eg. judges assigned to court cases). The "regions" are framed as subgroups of the data (eg. males over 30 years old) which can be interpretable (based on the chosen model). The paper formalizes this task as that of identifying the subgroup where the identity of the specific decision-maker has a causal effect on the decision, proposes an optimization algorithm to find this subgroup, develops generalization bounds (under some simplifying assumptions), and demonstrates compelling performance on semi-synthetic and real-world datasets.
main_review: ## General Points

Finding regions of "disagreement" or "heterogeneity" in the decisions made by human experts is important to diagnose decision-making issues. For example, if we find that judges are highly variable in the decisions they make for young males, it might be worth creating a uniform judgement policy for such males to improve judgement consistency, and reduce the potential impact of personal biases. If we find that teachers grading essays tend to disagree on the scores assigned to female students, it might be worth hiding student identities. In both cases, before enacting any policy changes, **we need a principled method to identify such regions of heterogeneity, which this paper provides.**

This specific task falls under the broader umbrella of *subgroup discovery*. One of my favorite recent examples of work in this space is [this paper](https://arxiv.org/pdf/1707.00046.pdf) on identifying regions/subgroups where 2 machine learning models (not humans) disagree (with a specific focus on disagreement  in fairness). This paper is also related to work on *heterogeneous treatment effect estimation* (the paper on [causal forests](https://www.tandfonline.com/doi/abs/10.1080/01621459.2017.1319839) is a canonical example of this space). **Unlike the aforementioned 2 papers, this paper does not use recursive partitioning and has a clever, novel formulation of the subgroups** (those where the causal effect of the assigned decision-maker is non-zero). I believe this formulation itself will be useful for the fairness literature, and that this paper can benefit from some of the ideas in the aforementioned 2 papers.

I have summarized the key strengths and weaknesses of this paper below.

## Key Strengths

**Important problem and clever, novel formulation.** I have stressed the problem importance and novelty above. I want to note that the proposed formulation and optimization method is fairly flexible and can be made interpretable and tailored to specific domains. This helps expand the proposed method's applicability to a wide range of real world settings.

I must also stress that the proposed causal estimand in eq. 2 is non-trivial to come up with: it is typically hard to come up with a causal estimand for a new task that makes intuitive sense, that is identifiable from observable data, and that is estimable from data in an efficient manner.

In addition, Theorem 1 transforms eq. 3 which contains the propensity scores P(a|s) into eq. 4 that does not require knowing these propensity scores. What this means is that, as long as the (conditional) randomization/no unobserved confounders is satisfied, we do not need to know the specific randomization protocol! This enables applying this method to settings like judges assigned to cases, and physicians assigned to patients, where we know that there is randomization but the specific assignment mechanism is proprietary/unknown. Without Theorem 1, this method would probably be restricted to randomized control experiments where the decision-maker assignment probabilities are known.

**Clear and complete description of the causal task.** This paper states the causal estimand, the identification assumptions, and derives an identification proof to link unobservable counterfactual quantities to observable ones. As such, the description of the causal task is clear and complete. Importantly, the mathematical quantities are also described intuitively and linked to real-world quantities, which aids readers interested in applying this method to their setting.

**Compelling synthetic and real world performance.** The paper empirically demonstrates performance in a well-described collection of experimental settings, one synthetic and two real world. The method is evaluated along several dimensions, and recipes are provided for essential real-world tasks such as interpreting and assessing the significance/stability of the identified region

## Key Weaknesses

**The proposed method finds the one region with maximum heterogeneity.** In practice, we can expect there to be multiple regions where the agents disagree; the proposed method would not help identifying all such regions. While it might be tempting to simply tweak the proposed method to find all such regions, it would be very difficult to address issues of multiple testing / false discovery (if we test enough regions of a large feature space, we will likely find at least a few that appear to have significant heterogeneity). The paper might want to clarify this in the problem definition, since the title is a bit misleading in this regard.

**It is unclear how the method would perform and scale as the dimensionality of the feature space increases.** I expect that, with high dimensional $X$ and several irrelevant features, it would be harder to find valid regions of heterogeneity. In addition, I wonder how the running time / no. of iterations of the optimization algorithm scale with the size of the feature space, no. of decision makers, and no. of data samples.

**While the no unobserved confounders and consistency assumptions are stated, its real world implication need to be clarified up front.** What would violate these assumptions in the real world (provide concrete examples)? It is very important to clarify, for future readers of this method, that you **need (conditional) randomized assignment of decision-makers for this method to work.** This is not a dealbreaker, but I can imagine future readers brushing over the no unobserved confounders assumption without realizing how restrictive it is.

**While recipes to assess significance/stability are provided, it is unclear if these are statistically valid.** Ideally, one would have a way to construct "confidence intervals" around the identified region of heterogeneity. While this paper provides a recipe to test significance, I am not sure how statistically valid that is (can validity be demonstrated empirically?). The paper needs to caution readers about the potential for false discoveries without a statistically valid inference procedure.

**Concerns with estimating the f(x) and h(x) models on the same data where the causal effect is estimated.** In the "double machine learning literature" where ML models are using along with causal inference, the ML models are estimated on a *separate sample* of the data. This is to disentangle the errors of the causal effect estimation and the errors arising from (i) overfitting the ML models, and (ii) underfitting the ML models. This paper does not consider what happens if f(x) and h(x) are poor models (does it lead to an increased or lowered risk of false discovery of regions of heterogeneity?).

**Intuition behind the grouping objective in eq. 3.** The grouping mechanism was introduced without much intuition; could the equation be supplemented with a few descriptive lines? I believe because of the grouping, there is no need to place an absolute value around eq. 3, but I could be wrong.

**Optimization guarantees.** Does the optimization method find a global optimum (I know most subgroup discovery methods are greedy and do not find the globally optimal region of heterogeneity, but it might be good to state this clearly to readers)?

**Potential negative societal impact.** As a method of "discovery", there is is always the potential for false discoveries with bad consequences (especially when no statistical inference safeguards are provided). In addition, the models f(x) and h(x) could propagate biases in the data they are trained on, which might spill over in to the regions discovered in unexpected ways. The authors don't actually talk about the potential negative societal impacts in Section 1. It is essential that these concerns be mentioned early in the paper.
limitations_and_societal_impact: **Potential negative societal impact.** As a method of "discovery", there is is always the potential for false discoveries with bad consequences (especially when no statistical inference safeguards are provided). In addition, the models f(x) and h(x) could propagate biases in the data they are trained on, which might spill over in to the regions discovered in unexpected ways. The authors don't actually talk about the potential negative societal impacts in Section 1 (they only state that Assumption 1 is a limitation). It is essential that these concerns be mentioned early in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary:

This paper considers settings where decisions for individuals are each made by different decision makers (agents). The paper defines the problem of finding a region in feature space in which the agents administer highly varying decisions. For example, perhaps judges mete out different sentences for misdemeanors by young men. The paper defines new causal criteria to characterize such regions in the feature space, where the choice of agent has an effect on the decision. The paper introduces an iterative algorithm to maximize this criteria to identify regions of heterogeneity, and provides theoretical results about the optimality of the proposed algorithm. The paper empirically demonstrates the algorithm on a semi-synthetic dataset and real clinical data about doctor recommendations for diabetic patients.

Contributions:

+ A new criteria for defining regions where there is heterogeneity in decisions administered across agents.
+ Results for estimating the criteria from observational data and an iterative algorithm for optimizing the criteria.
+ Theoretical results about the algorithm and empirical validation of the algorithm.
main_review: Originality:
This paper presents an original estimation problem and algorithm to solve it. To the best of my knowledge, this paper provides a new formalization of the problem of wanting to learn about decision making heterogeneity and proposes a new criteria for measuring this heterogeneity based on regions of input space in which agents differ in their decision making.

The iterative algorithm is a reasonable optimization procedure and the theoretical results are obviously new for this algorithm, but probably borrow ideas from the optimization literature (I haven't checked the proof.)

Quality:

Although I haven't checked the proofs for the theorems, the results pass the sniff test. I looked at the algorithm closely to follow along, and there are again no surprises there, which is good.

At a high level, my one complaint is about the beginning about section 2 and Eqn 1. I feel that causality is almost incidental to the paper because of the way the estimand in Eqn 1 is defined. Indeed, as the paper says, the challenge of causal inference is if we needed to evaluate Y(a') for a person who was assigned A=a and got Y(a) = Y|A=a. But in Eqn 1, we don't consider the impossible counterfactual Y(a') because we contrast the observed outcome with a different quantity, the average decision over agents.

Eqn 1 could be rewritten as:
$$\mathbb{E}[Y|A=a, X \subset S] - \mathbb{E}[Y|X \subset S]$$
The interpretation of the agent bias criteria still holds: we can think of it as the difference between a's decision and the expected decision across agents for that region of feature space. This formulation also connects the algorithm and criteria to the "direct" baseline.

Mainly, I wonder if invoking causality is an unnecessary distraction in this paper. Can the authors explain why we need to consider counterfactual notion from their view?

Another technical point where I have a doubt is: how does this algorithm fare when we have a high dimensional feature space? In the empirical studies, both settings consider just a few features. Is that because the algorithm fails with high dimensional data, or is it for ease? Theorem 2 does show a positive dependence between the "size" of S and the error term -- does this turn out to be a practical hindrance, restricting the algorithm to small feature sets?

I think the empirical studies are thorough, with both a semi-synthetic study and a real world case study where we have validation based on domain knowledge. I think especially given that the authors had to come up with their own baselines, they did a good job of designing comparisons.

However, I would've liked a better explanation about the motivation behind using TARNet baseline as its used. That is, what's the significance of predicting E[Y|X,A] and then calculating Var[E[Y|X,A]]?

Finally, do the authors have a hypothesis for why Direct performs comparably with Iterative in the misdemeanor setting?

Clarity:
The paper is reasonably well written, with enough detail around the algorithm and experimental evaluation for a reader to implement the algorithm themselves or to understand the studies.

The authors also provided good plain English explanations about the agent bias criteria that they defined, which helped build intuition. There was also exposition to give intuition about the algorithm. 

However, I think that the notation of the form $\mathbb{E}[Y | X \subset S]$ is unconventional and needs a clear, mathematical explanation prior to equation 1. Initially, I interpreted it as E[Y|X], which is a random variable in X, and I got extremely confused with the equivalence in Eqn 2, which clearly evaluates to a scalar. 

I might even give Eqn 2 right away without Eqn 1 with more explanation about the notation. The intuitions that came after Eqn 2 were important for understanding the interpretation of the quantity.

Significance:

The paper formalizes the problem of decision making heterogeneity and offers a practical procedure to find regions where agents make different decisions. I think this offers a significant contribution to domain experts as well as researchers in causal inference who will want to build upon this work.

limitations_and_societal_impact: There isn't much a discussion about the limitations of this method, which I would've liked to see. This is a room for improvement.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors develop an algorithm for identifying cases (e.g., defendants or patients) on which decision makers (e.g., judges and doctors) "disagree" on the decision. 

## Identification
The authors consider a setting in which cases are summarized by features $X$. Each case is associated with a decision maker $A$ and a binary decision $Y$. In the pretrial release setting, each case is a defendant, the features $X$ are observable characteristics of the defendant (such as demographics or charge information), the decision maker $A$ is the assigned judge and the binary decision $Y$ is to either release or detain the defendant. The authors define the potential decision Y(a), which describes the decision that would be made by decision maker $a$. 

The authors propose a summary measure of agent disagreement using average contrasts of potential decisions across agents given sets of the observable features. In particular, the authors focus on the "conditional relative agent bias" as 
$$ \mathbb{E}[Y(a) - Y(\pi(x) | A = a, X \in S], $$
where $Y(\pi(x))$ denotes the expected decision under the assignment mechanism of decision makers to cases. The authors key identification assumption is that decision makers are as-if randomly assigned to cases conditional on the features (i.e., $A \perp Y(a) \mid X$). This measures the extent to which decision maker a's average decisions over $X \in S$ differs from the average decision that is made under the assignment mechanism. In other words, this measure of disagreement depends on the assignment mechanism of agents.

The conditional relative agent bias measures the extent to which a single agent a's decision disagrees relative to the average decision over $X \in S$. The authors then propose aggregating this measure across agents. For some grouping of agents G, the authors define the estimand
$$ Q(S, G) = \sum_{a : G(a) = 1} P(A = a \mid X \in S) \mathbb{E}[Y(a) - Y(\pi(x) | A = a, X \in S]. $$
The authors main identification result (Theorem 1) is to show that if agents are as-if randomly assigned to cases, then this estimand can be identified as the conditional covariance between the observed decision and the agent assignment
$$ Q(S, G) = \mathbb{E}[Cov(Y, G \mid X ) \mid X \in S]. $$

## Estimating Regions of Heterogeneity.
Using this identification result, the authors goal is to search over regions of the feature space $S$ and groupings over agents $G$ to find the combination that maximizes this proposed measure of disagreement. To do so, the authors propose to solve 
$$ \max_{S, G} Q(S,G) \mbox{ s.t. } P(X \in S) \geq \beta, $$
which requires that the region have probability at least $\beta$ (a tuning parameter to be specified). The authors develop an iterative algorithm to optimize this objective function, and provide some conditions under which they are able to derive a generalization bound on the solution return after the first iteration (Assumptions 2-4 and Theorem 2).

## Empirical Applications
The authors apply their proposed algorithm in two empirical applications. 

First, the authors analyze the behavior of their proposed algorithm in a semi-synthetic experiment that is calibrated to data from Lin et al. (2020) on recidivism risk predictions made by MTurk workers. The authors simulate data from two types of agents that follow different decision rules for detention vs. release. The authors compare their algorithm against direct models that fit logistic regressions for the decision on agent identifiers and observable features (direct models) and TARNet. The authors report that their algorithm does better at recovering the "region AUC."

Second, the authors analyze an observational dataset on medical treatment decisions for diabetes. The data contain information on 3,956 patients and 458 group practices, and the authors search for the region of disagreement across these group practices.

main_review: Overall, I found this paper to be interesting. The authors study a potentially important, policy relevant problem -- in many empirical settings such as the criminal justice and health care, it is useful to understand how much disagreement there is across decision makers and what types of cases have the most disagreement.

## Significance

My main concern with the paper surrounds the authors' proposed measure of disagreement across agents. 

1) As mentioned above, the authors' proposed measure of disagreement ("conditional relative agent bias") measures the difference in the decisions of some fixed agent a against the average decision taken under the agent assignment mechanism.

This leads to strange properties when overlap fails. Consider a simple example where $X = \{0, 1\}$ and there are only two agents. Suppose the agents always make opposite decisions $Y(a_1) \neq Y(a_2)$ and $P(A = a_1 | X = 0) = 1, P(A = a_2 | X = 1) = 1$. In this case, the conditional relative agent bias is zero. This occurs because there is no overlap across the two agents.

Of course, as the authors point out an overlap assumption, this is not identified. But clearly this is a DGP in which we would naturally say there is a lot of disagreement across agents. However, their estimand would there is no disagreement across agents.  I think this deserves more discussion in the paper. 

2) The authors' proposed measure of disagreement is also not signed. It would appear to be more natural to consider an absolute conditional relative agent bias? In this way, disagreement in any direction contributes to overall disagreement. In contrast, with this current definition, it seems like you could have a setting where one group of agents is more likely to choose Y = 1 on average, and a second group of agents is less likely to choose Y = 0 on average but these positive vs. negative biases of the groups could cancel each other out. Is this not possible? If it is possible, why should we not be worried about understanding disagreement with this objective? 

3) The authors aggregate their proposed measure of disagreement into the estimand $Q(S, G)$. Again, the fact that the authors use the agent assignment mechanism leads to some strange behavior here. In particular, it implies that for any $S$, $Q(S, G) = 0$ if $G$ is equal to the set of all agents. This can be most clearly seen from their identification result in Theorem 1. If $G$ always equals 1, then 
$$E[(Y - E[Y | X]) G \mid X \in S] = E[(Y - E[Y | X]) \mid X \in S] = E[Y \mid X \in S] - E[Y \mid X \in S] = 0$$.
This is also strange since it implies that disagreement is zero whenever G is chosen to be all possible agents. That's not an intuitive property of a measure of disagreement and deserves some discussion. 

Since this estimand is the core object of the paper, it is essential for the authors to clearly motivate it, and discuss its limitations. The authors have not sufficiently done so in this paper. I have other comments about the significance of these results as well:

4) The authors assume that agents are assigned to cases as-if randomly conditional on the features $X$. In many empirical applications of interest, such as the cited examples in the pretrial release system (e.g., Arnold, Dobbie & Yang 2018; Kleinberg et al. 2018 which the authors cite), decision makers are \textit{quasi-randomly} assigned to cases conditional on some location-by-time cells (i.e., judges working in a given courthouse/county on a given day of the year are as-if randomly assigned to cases). Denote these location-by-time cells by $T$. The typical assumption in empirical applications is $A \perp (Y(a), X) | T$, which of course implies that $A \perp Y(a) | T, X$. 

Under this assumption, the authors methods search for the region of heterogeneity over both (T, X). Can this be extended to only search for a region of heterogeneity that only depends on X? For example, policymakers may be interested in only measuring heterogeneity across defendant characteristics but not across the court-by-time cell itself.  Discussing this case would enhance the empirical relevance of these methods.

5) The authors emphasize that avoiding the overlap/positivity assumption is a key contribution of the paper. 

Is this a common problem in practice? For example, if decision maker's are totally randomly assigned (for example, in my comment #3 imagine focusing on a single court-by-time cell), then A \perp (Y(a), X). If P(X) > 0, then overlap is satisfied. It would be useful for the authors to provide some concrete examples (perhaps in their second empirical application?), where the failure of overlap is a serious problem.

6) The authors require strong assumptions for their derivation of the generalization error in Section 3.4 -- e.g., only two groups of agents that follow distinct policies in Assumption 2 and a positivity type condition in Assumption 4. It would be useful for the authors to succintly summarize the roles of these assumptions in the paper. Could they be relaxed in any way? Moreover, it is strange that the authors introduce a positivity type condition in their error analysis but spend much of Section 2 emphasizing that they don't need to make this assumption for the identification result. If I need to make a positivity type condition for estimation, why wouldn't I be willing to make that same assumption when I determining what is identified from the DGP? 

7) Is the conditional random assignment assumption a reasonable assumption in the second empirical application to diabetes treatment? For example, if this is a primary care setting, there are many reasons we would suspect that patients are not randomly assigned to doctors -- e.g., patients may self-select to particular doctors based on information that we do not observe, doctors may have closed practices that limit the number of patients that they may see. The authors should discuss why this assumption is reasonable in this setting. 

## Originality 

The authors study an important question about measuring heterogeneity in the choices of decision makers. I agree with the authors assessment that this question is often overlooked in statistics and computer science. However, there is an enormous and active literature studying heterogeneity across decision makers in economics (some of which they cite such as Arnold, Dobbie, Yang 2018 and Kleinberg et al 2018).

For example, recent work such as 
* Abaluck et al. (2016) ("The Determinants of Productivity in Medical Testing: Intensity and Allocation of Care")
* Currie & Macleod (2017) ("Diagnosing Expertise: Human Capital, Decision Making and Performance Among Physicians")
* Arnold, Dobbie and Hull (2020) ("Measuring Racial Discrimination in Bail Decisions)
* Chan, Gentzkow and Yu (2020) ("Selection with Variation in Diagnostic Skill: Evidence from Radiologists")
* Norris (2020) ("Examiner Inconsistency: Evidence from Refugee Decisions")
* Ulrich & Ribers (2020) ("Machine predictions and human decisions with variation in payoffs and skill") 
study this same question in the same empirical settings studied by the authors (i.e., criminal justice and health care). 

These papers typically place behavioral/structural assumptions on what the authors call the "potential decision" and estimate heterogeneity across decision makers using discrete choice modelling. This additional structure is often useful for domain experts since it can be leveraged to ask and answer interesting policy counterfactuals ("how would decisions change if all decision makers have the same threshold, etc.). In this sense, the authors approach is more "nonparametric" by not modelling the potential decision, but that has a cost since it is unclear how the resulting estimand they focus on is directly useful for domain experts.  Since this question is not novel relative to this large literature in economics, it would be useful for the authors to carefully discuss how their methods and approach differ from this existing research. 

## Clarity

There are two places where I found the paper to be unclear:

1) In their empirical application to recidivism prediction, the authors report the "Region AUC." I think this compares the AUC of region classifier S returned by their own model (which classifies whether X is in the region) vs. the ground truth region. Is that right? If so, it would be useful to explicitly define this in the main text.

2) In Section 3.4, the authors introduce notation $S^*, G^*$ to describe the optimal $S^*, G^*$ that maximize the objective. Are these assumed to be unique?

3) I was confused why there is some notion of a true parameter $\beta$? I would have thought that $\beta$ is a tuning parameter to be set by the user, not something that would be estimated/calibrated by the data. In other words, it is up to the user to specify what the appropriate region size is. 

## Update:
I read the other reviews and the authors' responses. I thank the authors for clarifying some points that I raised on the interpretation of their proposed measure of disagreement. Provided the authors make these additional clarifications in the main text, I'd like to revise my score from 4 to 6.
limitations_and_societal_impact: Please see above for my comments on limitations and suggestions for improvement.

I do not see any potential negative societal impact from this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors are concerned with identifying regions in a covariate space. A region is characterised by having high variation in (potential) treatment outcome, as compared to the expected (potential) outcome. A method such as this could be used to identify variation of practice in the legal or medical domain (as indicated by the authors). 

This is a relevant topic which the authors rightfully relate to the causal literature. Specifically, the authors chose the potential outcomes model, where the treatment is now a clinician or judge; the outcome is binary (amounting to a decision, e.g. bail or no bail); and the condition is an individual. Contrasting potential outcomes, the authors are interested in learning subsets of X, rather than Y(a).
main_review: I found the paper clear and well written, the topic relevant, and the presented method interesting. However, there are a few concerns which I have listed below.

### Major
* On overlap. In Assumption~1 (line 68), the authors do not assume overlap (0 < p(A | X =x) < 1 for all x). They motivate this in the remark starting at line 104. The motivation seems flawed to me; line 108 "If judge a makes bail decisions primarily for violent crimes, but some other judges never see violent crimes, then we cannot hope to compare Y(a) to the decisions of those alternative judges.", this is correct. However, what if we want to know the potential outcome of placing a violent crime case to those alternative judges? The reason we make the overlap assumption is such that we can ask the model for an estimate of these edge-cases. By using a propensity (such as the authors suggest on line 111), the contribution of those alternative judges is minimal, but if we wish to make an estimate for these judges, we do want some data which we can consider. If those propensities were 0, we would have a biased estimate.
* There exists literature on partitioning for heterogeneous treatment effects which remains undiscussed [1, 2, 3, 4, 5]. This literature contains important discussion, as well as important benchmarks for the authors' experiments. 

### Minor
* line 94, "[...] if the agent-specific bias is non-negative on S.", I'm guessing this should be non-zero? If the bias is negative, then there is still bias, correct?

[1] HS Lee et al. "Robust Recursive Partitioning for Heterogeneous Treatment Effects with Uncertainty Quantification" NeurIPS 2020

[2] K Imai et al. "Estimating treatment effect heterogeneity in randomized program evaluation." Annals of Applied Statistics 2013

[3] JC Foster et al. "Subgroup identification from ran- domized clinical trial data." Statistics in Medicine 2011

[4] LL Doove et al. "A comparison of five recursive partitioning methods to find person subgroups involved in meaningful treatment– subgroup interactions." Advances in Data Analysis and Classification 2014

[5] S Athey & G Imbens "Recursive partitioning for heterogeneous causal effects." PNAS 2016
limitations_and_societal_impact: The work presented in this paper taps into an active area of research (subgroup analysis of HTE). The biggest limitation, in my opinion, is (largely) not acknowledging this existing brand of literature. Furthermore, I believe the authors should benchmark against (at least some of) these existing methods (cited above).
review_title: null

Meta-review: In this work the authors propose a method for identifying regions (types of instances) where there is a high level of heterogeneity in in human decision making.  Their approach is grounded in a causal inference framework that models high-heterogeneity regions as ones where the choice of decision maker ("treatment") has a large effect on the resulting decision.  

This paper received a number of excellent high-quality reviews that raised important concerns and requests for clarification.  The authors responded well to the major concerns raised by the different reviewers and have clearly articulated how they will revise the manuscript to improve clarity and further contextualize the work within a broader set of relevant literature.  I agree with the reviewers that this paper tackles an important problem and that the work meets the bar both in terms of technical contribution and exposition.  In revising the manuscript, the authors should seek to respond to as many of the reviewer suggestions as possible, with an emphasis on:

- Adding and briefly discussing the related work suggested by reviewers
- Making assumptions more tangible by giving clear examples of when they may or may not hold in practice
- Clarifying the particular type of overlap assumption upon which the work relies, and contrasting with all-agent overlap
- Clarifying why agent-specific biases do not cancel out in the aggregate objective
Decision: Accept (Poster)