Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance
Authors: Justin Lim
Christina X Ji
Michael Oberst
Saul Blecker
Leora Horwitz
David Sontag
Venue: NeurIPS 2021 Poster (2021)
Keywords: heterogeneity
causal inference
decision making
criminal justice
healthcare

Abstract: Individuals often make different decisions when faced with the same context, due to personal preferences and background.  For instance, judges may vary in their leniency towards certain drug-related offenses, and doctors may vary in their preference for how to start treatment for certain types of patients.  With these examples in mind, we present an algorithm for identifying types of contexts (e.g., types of cases or patients) with high inter-decision-maker disagreement.  We formalize this as a causal inference problem, seeking a region where the assignment of decision-maker has a large causal effect on the decision.  Our algorithm finds such a region by maximizing an empirical objective, and we give a generalization bound for its performance. In a semi-synthetic experiment, we show that our algorithm recovers the correct region of heterogeneity accurately compared to baselines. Finally, we apply our algorithm to real-world healthcare datasets, recovering variation that aligns with existing clinical knowledge.

Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance
Justin Lim'
Christina X JI'
Michael Oberst"
MIT CSAIL and IMES
MIT CSAIL and IMES
MIT CSAIL and IMES Cambridge, MA moberst@mi edu
Cambridge, MA justinl@mit.end
Cambridge, MA cji@mit.eet edu
Saul Blecker
Leora Horwitz
NYU Langone
NYU Langone New York, NY
New York, NY
saul. wweeckkr@@nyllaggne.oorr org
leora. leora. hooor..oooitteeelylaggnn...o
David Sontag MIT CSAIL and IMES
Cambridge, MA
dsontag@csail edu
Abstract
Individuals often make different decisions when faced with the same context, due to personal preferences and background. For instance, judges may vary their leniency towards certain drug-related offenses, and doctors may vary in their preference for how to start treatment for certain types of patients. With these examples mind, we present an algorithm for identifying types of contexts (c.g., types of cases or patients) with high inter-decision-malee maker disagreement. We formalize this a causal inference problem, seeking region where the assignment of decision-maker has a large causal effect on the decision. Our algorithm finds such region by maximizing an empirical objective, and we give generalization bound for its performance. In a semi- remi-synthetic experiment, we show that our algorithm recovers the correct region of heterogeneity accurately compared to baselines. Finally, we apply our algorithm to real-world healthcare datasets, recovering variation that aligns with existing clinical knowledge.
1 Introduction
Understanding heterogeneity in decision-making is an established problem in medicine (Birkmeyer
al. 2013) Corallo et al. 2014 De Jong al.] 2006), consumer choice (Ortega et al.) 201 Scarpa et al. 2005). and law (Kang et al. 2012 Kleinberg et al.] 2018 Arnold et al. (2018). 3. In the context of medieine, this is referred as the study of practice variation (Atsma al. 2020 Cabana al.) 1999), where it is often observed that doctors, facing the same clinical context, will make different decisions. Likewise, in legal context, judges often differ in their leniency in their decisions regarding bail
(Kleinberg et al. 2018) juvenile incarceration (Aizer and Doyle Jr] 2013), the use of alternatives to incarceration (Di Tella and Schargrodsky (2013), and incarceration length (Kling) (006). In some scenarios this variation may be justified: The best medical treatment may not be obvious. In others, it
may be grossly unfair, as the case of racial bias in bail decisions (Arnold et al. 2018).
Equul contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this work, we tackle the question of how to find and characterize this variation in the first place. In particular, we present a learning algorithm for identifying a "region of heterogeneity", defined as a subset of all contexts (e.g., patients, cases) for which the identity of the catiiionnmmeerr substantially affects the decision. In medicine, a better understanding of treatment variation can inform the development and dissemination of clinical guidelines. In the legal domain, characterizing the cases where judges vary most in their leniency may help with investigating potential issues of fairness.
We want to characterize examples where changing the decision-makes would have resulted in a different decision. The challenge is two-fold: First, we only observe single decision-make directly observe how (for instance) multiple judges would have decided the same case, Second, our data on individual decision crdision-makerrs makers is often scarce. For instance, in Section [] we consider a medical dataset with more than 400 doctors, each of whom has fewer than samples on average.
We formalize characterizing the region of heterogeneity as a causal inference problem:
per example, so we cannot
We will refer to decision-maker as "agents"., and our contributions are as follows: In Section 2 we propose an objective defined in terms of counterfactual decisions across different agents, and show that this objective can be identified from observational data. Moreover, this objective does not require the use of agent-specific statistical models, making it amenable to our sparse setting. In Section D we give an iterative algorithm to identify regions of disagreement by maximizing this objective and provide intuition (in the form of a generalization bound) for the factors that drive its performance. In Section 4
we use a semi-synthetie dataset, derived from rowd-sourced recidivism predictions, to demonstrate that our algorithm recovers the correct region of heterogeneity accurately, even when there are many agents. Finally, in Section 5 we apply our algorithm to a real-world healthcare dataset and confirm that it recovers intuitive regions of variation in first-line diabetes treatment. We conclude with a discussion of related work and implications. Our code is available at https://gi http:/ggttuu....m com/ www///llniii
-ndecisiio heterogeneity terioneeittririinnnnen
Our algorithm does not determine whether variation is inherently good or bad or how it should be addressed. Rather, more careful study with domain experts would be required to determine if variation can (or should be) reduced and how. In addition, false discovery of variation is possible and could have negative impact. We expect that validation on independent datasets would be required in real-world applications, using the regions identified by our method as plausible hypotheses to test.
2
Characterizing Heterogeneity from a Causal Perspective
2.1
Notation
Let the data be drawn from a distribution P(X, A, Y), where X is random variable representing context (or features), A is a discrete agent, and V E (0, 1) is the binary decision. The spaces of all X and A are denoted as X and A with realized values lower case x and a, respectively. Indicator variables 1 [-] are one if the statement inside the brackets is true and zero otherwise. For subset S X, [x S] is sometimes written as a function Sxx), where S X 10, 1). A subset S may have several disjoint regions. B [ E S] denotes the average Y across samples in S. For instance, if (X: : X, 10), then E [Y]X,- 10] is a scalar average of Y among samples with Xg- 10.
2.2 Heterogeneity as a Causal Contrast
Our conceptual goal is to identify a region S C X where different agents tend to make different decisions even when faced with the same context. We can formalize this in the language of potential outcomes from the causal inference literature (Pearl) 2009, Hernán and Robins, 2020). which for clarity we will refer to as potential decisions: In particular, we denote Y(a) to be the potential decision of agent a. The fundamental challenge of causal inference is that we do not observe all potential decisions (Y(a): : a e A) for each sample, but only a single decision Y. With this in mind, we will make the following assumptions, standard in the literature causal effect estimation.
Assumption 1 (Causal Identification Assumptions). (i) Consistency: Y = y,A = Y(a) yy, and (ii) No Unmeasured Confounding (NUC): For all a E A. Y(a) 1 1AI X.
Consistency links the potential y(a) to the observed Y, and NUC says that there are no unobserved factors that influence both the assignment of agents and the decision itself. For instance, the quasi- random assignment of cases to judges conditioned on features X satisfies NUC (Kleinberg al. 2018).

---

NUC may be violated if key aspects of the case (e.g., misdemeanor vs. felony) are omitted as features. For instance, misdemeanor and felony cases may be seen by different judges and have different decision processes, but this variation is not due to agent preferences. Given these assumptions, we propose a causal measure of agent-specific bias, defined as contrast between potential decisions. and subset X, the
Definition 1 (Conditional Relative Agent Bias). For agent conditional relative agent bias is defined as
B[Y(a) Y(T()) A a, [J
(1)
where Y(a) is the potential decision of agent and Y(T(x)) E. B[Y(a') x]m(a' | x) denotes the expected potential decision under the agent assignment distribution A'a' xxx P(A a' X x).
Note that under Assumption 1 Y(π(x)) = B|Y _ but here we emphasize its causal interpretation as
the expected decision of a random agent. Equation T represents the relative difference between the decision of an agent (on their particular distribution of cases in the region) and the potential decision of a random agent. In particular, Equation (I] can be written as follows under Assumption T
J.es [ B[Y(a) - Y(x) X = x]p(x A = a, X S)dx,
B[Y(a) Y(x) A = a, X
(2)
where we shorten ((x(x)) to Y(x). This is the average difference (over p(x a), restricted those in the set S) of the conditional expected difference between Y(a) and V(x). For example, suppose that the agent a is judge who is particularly lenient bail decisions for felony arrests (the region ) and Y = denotes granting bail. Then imagine the following counterfactual: Take the felony cases that are assigned to this judge and reassign each individual case, described by x, to a random judge a', proportionally to p(a' x). We may then observe, on average, that the bail rate would decrease, because most judges are less lenient than judge a, corresponding
positive value of Equation (I)
Equation T has the additional advantage of being easy to estimate: Under Assumption [T it can be rewritter as B[Y B[Y X] A a, E S1, the expected residual in predicting (using the conditional expectation B[Y X]) the decision of an agent a across the context x typically seen by that agent.
2.3 Formalizing a Causal Objective
Our primary goal is to discover region S where substantial heterogeneity exists across agents. To do so, we define an aggregate objective across a group G of agents, where G(a) e (0, 1J is an indicator function for membership.
Σ X acG(a==1
@(S,))=
Y(r) | A a, X S).
(3)
We now show that this quantity can be identified and estimated from observational data without requiring agent-specific statistical models, before discussing the interpretation of this objective.
Theorem (Causal Identification). Under Assumption 7 Q(S. G) can be identified as
Q(S,G) Es|Cov(Y, X)] I(Y B[Y XI)G],
(4)
where BsI.] = B[- X E S] and Cov(Y, G X) is the conditional covariance.
Theorem and other theoretical results are proven in Appendix A The result follows from proving that the agent-specific bias (Definition]) M is identifiable using the expected conditional covariance between Y and the binary indicator 1 [A a]. With this in mind, optimize the following objective, where the set S is constrained to be at least a certain size B and S is a hypothesis class functions S.
Q(S G) s.t., P(S) 2 B. SES.G
(5)
Intuitively, this objective measures the disagreement between the agents in the group G(a) I and the overall average B[Y X] on the region S. Hence, the choice of group is important for interpreting the objective: If G(a) for all agents, the objective will be zero for any set S, as can be seen from Equation 4 applying the definition of the conditional expectation.
Interpretation:
2See Proposition A1
for short proof, and Proposition A2 for the derivation of Equation
*See Proposition A3 in Appendix A.1
3

---

Accordingly, we seek region S for which the partially maximized objective L(S) := maxe Q(S, G) large: This partial maximization is obtained by taking G(a) I whenever the conditional relative agent bias of agent a (on the set S) is non-negative. Thus, Equation (5) can be re-written as
max Q G) = Σ P(A- - a X () - Y(T) | A = a, 5]
(6)
where (al., max(x, 0), and this objective becomes an average over agents who have a positive bias. This population objective is also equivalent (up to a constant factor) to the (weighted) sum of the magnitude of each agent's conditional relative agent bias. See Proposition (4 in Appendix
Lack of Overlap: We have not made the overlap or positivity assumption that F(A = a x) > 0 for all x.a. While this assumption is required to identify conditional causal effects B[Y(a)- Y(a') X] (Nie) and Wager 2017; WWager and Athey] 2018 Shalit et al.) 2017). it is not required for identifying our causal contrast. Our problem only requires each context has positive probability of being seen by more than one decision maker. For instance, suppose that S contains both misdemeanors and felonies and there are four judges do.....a.. If judges ao and aj make bail decisions exclusively for felonies while judges and a3 make bail decisions exclusively for misdemeanors, our measure captures disagreement between ag and aj and between dg and a3 even though comparisons between dg and an or other pairs are impossible to make. Thus, we have chosen to compare Y(a) to the decisions of viable alternative agents, weighted by their probability p(a' x) of being selected for such a case.
3 Identifying Regions with Heterogeneity
In Section 3.1 we introduce an iterative optimization algorithm for a finite sample version of Objec- tive (5) that alternatingly optimizes S and G. In Sections 3.2 and [3.3] we discuss practical heuristics for choosing the region size parameter / on training data and validating if the resulting region generalizes to held-out data. Finally, we build intuition for the factors that influence performance of this algorithm via a generalization bound in Section 3.4 under simplifying assumptions.
3.1 Iterative Optimization Algorithm
We let Q(S, G) be the empirical analog of Q(S, G) (Equation 4 which we can write as follows.
Q(S,G)= En, 1 May E ES]
Σ (aa f(xa)) G(a) 1 |xa esJ.
n.j
where f(x) denotes a model of the conditional expectation f(x) x B[Y X x]. For simplicity of notation, we assume that there are R samples (indexed by j) for each of finite set of N agents (indexed by a). giving N R samples in total.
Our algorithm (Algorithm [ takes input the data ((Xaj, Yaj)) and a minimum region size B. and outputs a model h(x) and a threshold value that describe a region of heterogeneity S = (x e X: h(xx Z b). Starting with S X (the entire space), the algorithm identifies the grouping that maximizes Q(S, G), then uses that grouping to identify the region maximizing the same quantity, repeating this process until convergence. The algorithm uses a classifier f(x) to estimate B[Y X = x] and a regression model h(xx) to estimate the conditional covariance of the decision Y and the grouping G at X x. Note that we can use any supervised learning algorithms for f and h, allowing us to learn interpretable regions as part of the algorithm if h(x) is interpretable (c.g., decision trees). If sufficient data available, samples can be split into three parts for estimating f(x) in line 2, computing G(a) in lines 5-8, and training h(x) and estimating the (1 (1-B)-th quantile in line 10. We do not perform this sample splitting because our sample sizes are already small. under-fitting f(x by further restricting the sample size could lead to false discovery if f(x) does not capture the variation explained by X.
Optimizing over G given S. Given region S. our first result identifies the grouping G: F (0, 1) that maximizes Q(S. G) and shows that it can be expressed in terms of Q(S, al).
Proposition 1. Given S C X. Ō(S, G) is maximized over the space of functions G : A - (0. 1) at
Gs. where G5 (a) 1 @(S. 1 [A- a)) 2 0..
Intuitively, this proposition states that to maximize the empirical expected conditional covariance of the decision and grouping on a region, we must group agents by whether their residuals yaj - [(xj) are (on average) positive negative on S.
4

---

Algorithm Identifying regions with variation
1: Input: Data Yajl 1 minimum region size B.
a=1" 2: Fit a model f(x) to B(Y X =x x).
3: Initialize S = X.
4: repeat
5: for a==a=...... do
Compute Ô(S, [A = a)), where Q(S, 1 =a)= Σ, [ 2,0as f(xai)) Xas E Es}.
6:
7: Set G(a) = if ((S. [A a)) 2 0 and otherwise.
8:
end for
9:
Compute baj (Yaj ((xx)))G(a), a I.....N, N. j= I....,
Fit a model h(x) to predict baj from Xajs and let be the (1 - B)-th quantile of h(Kaj).
11:
S' S.
12:
S - (Xayi h(xa,) 2 b).
13:
until S = S" iteration limit reached.
14: Output: Model h and threshold b, defining a region (x E X;h(x) 2 b).
Optimizing over given G. To optimize Ō(S, G) for fixed grouping G over the hypothesis class S. we train a model h(x)) to predict (aa- f(xaj))((a) given Xaj. where h E H. Using h as an estimate in Eq. we find a set S to maximize the quantity Σ, H[xE] Σxes h(x), the empirical conditional expectation of /xxx over S. This quantity is maximized (subject to our constraint) by including the largest f-fraction of the h(xaj) in S. Hence, we pick b as the (1 B) quantile of h(Xaj) and choose our region as 6G = [x E h(x) 2 b).
3.2 Tuning the Region Size Parameter
For real datasets, we need to choose & without knowledge of the true value. Given that our objective be calculated on held-out data using the functions S, G, a seemingly obvious approach would be to compute Q(S, G) on a validation set and select the B that leads to the highest Q(S, G). However, for fixed data distribution, smaller values of ß tend to produce higher values of Q(S, G), and there a trade-off between finding smaller region of higher variation and a larger region that may include areas of lower (but still meaningful) variation. This motivated our original constraint P(S) 2 p.
To select B, we propose a heuristic inspired by permutation- based hypothesis testing (Wasserman 2004). We compare the training objective that might be seen even if all agents followed the same policy. For each candidate B. we (i) run our algorithm and compute the objective on training data gobs = Q G). (ii) For T iterations, randomly shuffle the agents and re-run the algorithm to get new objective value. This gives us distribution Pmai over Q(S, G) from a distribution where P(X, Y) and P(A) are unchanged but Y,X IL A. (iii) Finally, we compute a paluuu Pş f ( > qubs) and choose the / with the smallest p-value. In Section 4.2
reference distribution of values (for the same B)
we find that this heuristic empirically recovers the true B value in semi-synthetic settings.
3.3 Validation of the Region
We may wish validate the learned region S independently of the grouping G. In particular, finding & is not our main goal, and we observe in our semi-synthetic experiments that our algorithm can find the true region even when the grouping & fairly poor (due to few samples per agent), as shown in Appendix B.2 We can optimize over G in Q(S, G) to obtain an objective that depends only on S and we obtain an empirical analog of Equation (6) as
can be used to compare regions. By Proposition
I
L(S) == max Ô(S, G) = En,, [|aof ES)
Σ
oor f(xx()) Kaj E ES]
(8)
G
where |xl, equal to the positive part [x], max(xx, 0) as before. We then use this objective i(S) to answer the following question: Does our chosen region S yield significantly higher objective value on test data than randomly selected region of the same size? An example of this analysis given in
Table for the real-data experiment in Section 5
5

---

3.4 Generalization Error
We give a generalization bound for Algorithm | to build intuition for the factors that influence performance. To derive this bound, we consider a simplified setting, where there exists a set , such that the following set of assumptions hold.
Assumption 2 (Group-basee based variation). For all x S', B[Y | X = x,A a =|= (a))
for all x $ S', B[Y X x, a] B[Y [x
Assumption 3 (Non-zero relative biases). There exists a constant a > such that for all x S',
B[Y x, G' (A)= 1] E[Y x= > a, and B[Y X x, (A) 0] 00-E[Y B[Y X x] < aa,
Assumption 4 (All agents see samples in S'), There exists a constant e > 0, such that for every E A, P(X e S' A a) wP(X S*).
Note that under these assumptions, G maximize the objective Q(S, G) (see Appendix A.4
will refer to them as S for the remainder of this section. Assumption 2 says that there are two groups of agents, who follow two distinct decision policies within region S but follow an identical decision policy outside of S Assumption 3 says that one group has a positive bias across all of S", relative to the average over both groups, and the other group has negative bias. To simplify the analysis, we also make Assumption 4 that every agent has some non non-zero chance of observing some contexts in the region, but note that we do not require that p(x a) 0 for all x E S'. Under these assumptions, we demonstrate that the first iteration of Algorithm I will find, with high probability, a region S whose value Q G²) (for the same grouping G defined above) is close to that of the optimal S Note that we do not claim that the iterative algorithm finds the globally optimal solution. For simplicity, we assume that f(x) perfectly recovers B[Y | X]. This can be relaxed at the cost of additional terms in the bound that go zero as the overall sample size increases. Under Assumptions
2 3 and assume that P(S') = B. For the informal version presented here, we assume that P(S") P(S) ) = B. where S is returned by our algorithm, and that exactly a f-fracction of our samples fall into S and $ (in the Appendix, we give a version without these simplifications).
Theorem 2. Under the assumptions above, if E S and R > a-P
2lm2 the first iteration of Algorithm !
returns S such that, with probability at least 1 - 5, (QS",G") (GS) "SS.G)= S E, where
2 2m(11))
+ 2
31 m)
2 2l1(11/0)
=
B 77
+ 2R(S, R) +4 4
BN R
6.N
N
where R(S,N N R) is the Rademacher complexity of S, and 17 exp (-Ro2
The term 17 plays important role: It bounds the expected misclassification error P()(a) # Ga)).
For sufficiently large R, we have that 17 < 1/2 with high probability, i.e., we have a better-than-random chance of identifying the group for an individual agent. Moreover, 17 decreases as we increase the number of samples R for each agent, the separation a between the two groups on S", the region size B, and the constant w. For sufficiently small 17, our algorithm discovers region whose value (in terms of Q(S.G")) is close to that of the optimal region. The generalization bound improves as the number of agents N increases, the number of samples for each agent increases, or the complexity of the hypothesis class decreases. The latter is measured here by the Rademacher complexity R(S, R) of our hypothesis class S, which can be bounded by standard arguments. In conclusion, under some additional assumptions, Algorithm]] T identifies an approximately optimal solution with high probability after one iteration. We show via soni-eynteetic experiments in Appendix B.3 3 that convergence is generally fast in practice.
4 Semi-Synthetic Experiment: Recidivism Prediction
For conceptual motivation in the introduction, we discussed the legal system: As a potential applica- tion of our method, one could determine types of cases for which the idiosyncratic preferences of judges have significant impact on their decisions. Lacking data on judge decisions with sufficient context, we turn to the more controlled setting of human predictions of recidivism.
Dataset: We use publicly available data from Lin et al. (0220) who ask participants on Amazon's 's Mechanical Turk platform to make recidivism predictions based on information present in the
Available

---

Drug possession
Misdemeanor under 35
1.0
0.8
0.7
0.5
Direct
0.4
TARNet
Iterative (Ours)
Num gents
Num agents
Figure 1: Comparison of our method and best baselines at identifying region of heterogeneity, as measured by the held-out test AUC for classifying samples into the true region of heterogeneity. Total number of samples is fixed. Baselines are described in Section 4.1 Uncertainty bands give 95% intervals for the mean derived via bootstrapping over 10 random seeds using seaborn (Waskom) 202T). Left: Region modelled using a ridge regression in the drug possession semi-synthetic set-up.
Right: Region modelled using random forest for the misdemeanor under age 35 set-up.
"Correctional Offender Management Profiling for Alternative Sanctions" (COMPAS) dataset for Broward County, FL. (Dressel and Farid, 2018). Participants (or "agents") are shown risk factors: age, gender, number of prior convictions, number of juvenile felony charges, and number of juvenile misdemeanor charges. The charge in question is also given, as well whether the charge misdemeanor or felony. The dataset contains 4550 cases evaluated by 87 participants.
To benchmark where we have access to a "ground truth" region of heterogeneity. We retain the features presented to the original participants and construct two policies. which we refer to as the "base" and "alternative" policies: For the base policy, we learn logistic regression model on the binary decisions across the whole dataset. For the alternative policy, an extra positive term added to the logistic regression for samples within the region. We construct two scenarios with different regions of variation: (1) all drug possession charges, and (2) all misdemeanor charges where the individual is 35 years old or younger. These make up 22% and 21% of the data, respectively. Then, we generate synthetic agents (randomly assigned cases) and assign half of the agents the base policy and half to the alternative. Synthetic decisions are then sampled from the logistic regressions. For each scenario, the two groups of agents follow the same stochastic policy outside of the region, and one group systematically prefers within the region. More details can be found in Appendix B.
Semi-Synthetic Policy Generation:
method, we generate semi-synthetic data
4.1 Performance versus Baselines
Baselines: We compare how well our approach identifies the true region heterogeneity with several baselines. To our knowledge, the problem of finding regions of heterogeneity with large number of agents has not been studied before. Many causal inference methods for treatment effect estimation designed for single, binary treatment. However, naively estimating the treatment effect between each pair of providers would scale 0 (199P). Therefore, we develop new baselines. Some (including the causal forest and U-learer adaptations described in Appendix [.2) are based causal inference methods augmented to identify a region of heterogeneity and grouping of agents where possible.
Direct models: This bascline measures how much adding the agent to the feature set improves prediction of decisions. We fit logistic regressions with and without the agent feature to estimate E|Y A, X] and B[Y X]. For each sample (x, a), we compute by B[Y X x]] by E[| a,X = x]] to quantify how much the model with agents outperforms the model without agents. Then, fit a "region model" to predict this quantity from X. This region model is either ridge regression, decision tree, or random forest model. Finally, we compute the top 8 quantile of predictions from the region model in the training and validation sets and this cut-off select points in the test set.
TARNet: agnnsict- representation network (Shalit et al., 2017) models the outcomes of all treatments for each sample by learning shared representation of the sample features and then
7

---

Chosen value of B
Avg. p-value by
0.5
Density/ Count
0.4
0.3
0.2
0.1
0.0
0.1
0.2
0.4
0.0
0.2
0.4
0.6
B
(b)
Figure 2: Results of tuning f over 30 semi-synthetic datasets. (a) Average p-value for each candidate B. with 95% uncertainty intervals for the mean generated by bootstrapping. The dotted vertical line represents the true value of B. (b) Distribution of B with the smallest p-value over the datasets.
having separate prediction head for each treatment. We implement the shared representation model using a 2-layer neural network with ReLU and dropout. Each prediction head is linear layer with sigmoid function. TARNet predicts B[Y X, A] for every agent for each sample. Var [B[Y X,] measures the variation across agents if they had seen context X and analogous our objective without grouping. We predict this quantity with the region models in the direct model baseline.
Results: We evaluate how well the algorithms identify the samples within the region of heterogeneity when we vary the number of agents among 2, 5, 10, 20, 40, and 87, where 87 is the number of agents in the original dataset. Figure shows the best overall region models for each set-up, with the other models deferred to Appendix [B2 The metric in Figure iis the region AUC, defined as how well the model classifies whether samples belong in the region of heterogeneity when compared the consistently performs well for both semi-synthetic set-ups, especially when the number of agents increases realistic level (and the number samples per agent decreases).
true region. Algorithm| [
The direct baseline deteriorates very rapidly as the number of agents increases in the drug possession set-up, while the TARNet baseline deteriorates rapidly in the misdemeanor under age 35 set-up. Refer to Appendix B.2 for additional baseline details, region models, and evaluation metrics, We also show that our method is robust set-up with more than agent groups in Appendix [3.4]
4.2 Tuning the Region Size
We validate the proposed approach of tuning & (discussed Section 3.2) by applying the methodology to our semi-synthetic setting here. We sample 30 semi-synthetic datasets and consider candidate values B in [0.02, 0.42] in increments of 0.04. For each proposed value of ß we use T = 40 random permutations of the agents. presents results for the misdemeanor under age 35 set-up. As the candidate value of increases (up to the value of B), the p-value decreases, and the distribution of selected ß values are centered on the true value of
5 Real-data Experiment: First-Line Diabetes Treatment
We apply algorithm real-world dataset consisting of first-line (initial) treatment for type diabetes and examine how the treatment variation we discover aligns with clinical knowledge. We present an additional real-world experiment (on Parkinson's 's discase) in Appendix D
We use an observational, de-identified, dataset provided by large health insurer in the United States, spanning from 2010 to 2020. The task is to classify first-line treatment decisions between metformin (Y 0)-the typical recommendation from the American Diabetes Association- and other common first-line treatments such as sitagliptin, glipizide, glimepiride, glyburide (Y (American Diabetes Association 2010; Hripesskk et al., (016). relevant clinical features, we include the patients' most recent eGFR (mL/min/ 1.amm....7 1.73m2) and creatinine (mg/dL) measurements,
Data and Setup:

---

Table Objective values L(S) ) for the learned region on the training and test datasets, along with the distribution of values for randomly generated regions S rand given as mean (standard deviation).
Metric
Subset
Value
L(S)
Train
0.1029
L($)
Test
0,0924
L(S mad)
Test
0.0507 (0.0073)
incidence of heart failure, and treatment date. Because treatment date does not define a type of patient, we omit from the region model. However, including in the outcome model is essential because of increasing use of metformin over time. The agent indicator A is the group practice of the doctor responsible for the patient's 's treatment. 3,980 patients and 447 group practices are included in our cohort. After requiring at least 4 patients per agent, 3,576 patients and 176 group practices are included. This filter ensures each group practice has at least sample in the training and validation sets and at least 2 samples in the test set. In this experiment, we choose B 0.25 as input to our algorithm. See Appendix C for additional cohort definition and set-up details.
To interpret the region, we use decision trees as our region model h(xx) The tree is visualized in Appendix C The decision tree identifies the region of heterogeneity as the union of (i) eGFR below 71.5 and (ii) eGFR above 98.5 and creatinine above 0.815. These regions align with clinical intuition. In the first region. low eGFR values indicate impaired renal function (Group et al. 2009). which is a contraindication for metformin since it is traditionally associated with increased risk of lactic acidosis (Tahrani et al. 2007). Still, treatment decisions can vary here because guidelines for managing patients with eGFR below 45 are lacking (Group et al. (015). Note that this region provides an example of how our algorithm works when overlap not satisfied. Although 33 of 176 group practices do not see patients with these features, we can still conclude that this is region of heterogeneity among the 143 agents with cases. In the second region, there are no obvious contraindications for metformin. Thus, understanding why some doctors on average only prescribe metformin 78% of the time to patients this region may help us identify whether this is region in which we can standardize practice.
Interpretation of Results:
Assessing Significance: In Table [ we perform sanity check, assessing whether our algorithm identifies region S whose variation in held-out data is higher than that of a randomly selected region, using the partially optimized objective L(S)) = max; Q(S. G) laid out in Section 33.3 to compare regions directly. T shows that L($) is similar on the training and test data. Furthermore, we compare L($) on the test set to the distribution of L(S raa)) where rand are random subsets of the test data of the same size as S. We compute the latter distribution using 100 random subsets and observe that the test value of L(S) is more than two standard deviations above the mean of the latter. This gives confidence that the discovered region S generalizes to region of heterogeneity beyond the training set. We direct the reader to Appendix for additional analyses, such as evaluating the stability of the region over multiple folds of splitting the data.
6 Related Work
Beyond previously mentioned connections to causal effect estimation, we highlight a few areas of research that share similar goals to our own.
Prior works have modeled agent-specific decision-making processes by estimating separate model for each agent. Abaluck et al. (2016) model heterogeneity in physician tendency to run diagnostic tests. Chan Jr et al. (2019) estimate radiologist skill based on diagnosis and miss rates. In our setting, unlike in diagnosis, there no "correct" decision that can be incorporated into the model. Ribers and Ullrich (2020) assume there is provider-speciff noise in determining patient type, which affects the pay-off functions for deciding whether to prescribe antibiotics. When only a few decisions are observed per agent, these agent-specifis models cannot be estimated reliably. Currie and MacLeod (2017) also incorporate physician beliefs and skill into pay-off function. They estimate an aggregate logistic choice model (for C-sections) across all physicians and then learn how individual physicians deviate from this model. They do not learn the regions where this deviation occurs, as they focus on how heterogeneity is associated with
Agent-specific Models of Decisions:

---

downstream outcomes. Nomis 2020) looks for disagreement between agents but relies on some cases being seen by multiple agents. We assume each case is seen by only one cerision-makess
While our objective maximizes a causal notion of dependence, one could instead ask if Y is conditionally independent of A given X. Many metrics exist for testing conditional independence. such as the Hilbert-Schmidt independence criterion (HSIC) (Fukumizu al. 2007 Zhang al., 2012), conditional mutual information (Runge) 2018), conditional correlation [Ramsey 2014). and expected conditional covariance (Shah and Peters 2020). We give the last a causal interpretation under some assumptions and seek a region that maximizes it, in lieu of testing.
Conditional Independence Testing:
Hierarchical / Mixture Models: Bayesian methods are often used to estimate group-level effects, such as per-physician effect on patient outcomes (Tuerk et al., 2008), where group identifiers are included as a categorical feature in a multi-level generalized linear model (Gelman and Hill 2006). Alternatively, one could assume conditional mixture model (Bishop) 2006), where agents belong to latent clusters that each have their own policy. However, both of these methods require parametric assumptions on the distribution of P[Y x, a], and even so, the optimal mixture model is not necessarily identifiable when both the clusters and policies are unknown (Dasgupta and Schulman 2007). By contrast, our method does not require making particular parametric assumptions about B[ x, a] and seeks to learn the region of heterogeneity directly.
Checking for heterogeneity can also be framed as feature evaluation, where would like to evaluate whether adding the agent identifier will increase predictive power. Feature evaluation methods typically maximize dependence between selected features and labels, utilizing measures similar to those in conditional independence testing, such as the HSIC (Song et al. (007). Other methods use the correlation of the new feature with the loss gradient of the current predictor as a measure of utility (Koepke and Bilenko, 2012). In contrast, we focus not on checking marginally for the predictive power of agent identifiers, but rather identifying a region.
Feature Evaluation:
Crowdsourcing: Our work conceptually related to identifying which samples are difficult to label via crowdsourcing annotations (Karger et al. 2014; Whitehill et al., 2009). Most crowdsoureing models are generative with latent variables for the correct sample labels, sample difficulty, agent expertise, and agent bias. They then optimize for the likelihood of the observed labels. The set of difficult samples is analogous to our region of heterogeneity. The main difference with our problem is that we do not require any notion of the "correct" label.
7 Discussion
In this work, we take a causal perspective on finding regions where agents (i.c., heterogeneous preferences, formalizing this heterogeneity terms of counterfactual (or "potential") decisions across agents. We propose a causal measure of agent-specific biases and give an objective that aggregates this bias over agents. This objective can be identified from observational data and written in terms of an expected conditional covariance. Importantly, for our applications, this does not require building agent-specific models or assuming overlap across all agents.
have
We give an iterative algorithm to find a region that maximizes this objective. Then, we demonstrate in semi-synthetic experiments that our algorithm accurately recovers the region of heterogeneity and scales well with the number of agents. In contrast, performance of baslines deteriorates when the number of agents increases. Although our experiments have low-dimensional spaces, we hypothesize our algorithm would scale well to high-dimensional feature spaces since the average policy and region models can handle nigh-dimensional input spaces. Finally, on a real-world medical dataset, we show that our algorithm can yield insights that align with clinical knowledge.
Our work is motivated by understanding variation in human decision-making In the judicial domain, our method may help unearth types of cases where decisions are highly dependent on the judge assigned the case. In the medical domain, our approach may identify types of patients where new guidelines may be required to help doctors make decisions more consistent with standard of care. Domain expertise is required determine the implications of the regions discovered by our method. Beyond these applications, we see our approach as useful data science tool for understanding heterogeneity in decisions that appears be driven by individual-level preferences.
10

---

Acknowledgements: We would like to thank Aaron Smith-MeLallen, James Denyer, Luogang Wei. Johnathon (Kyle) Armstrong, Neil Dixit, Aditya Sai, and the rest of the data science group at Independence Blue Cross, whose expertise, data, and support enabled the diabetes experiment. We would also like to thank Rebecca Boiarsky for converting the diabetes data to the OMOP common data model format, Monica Agrawal for her helpful comments, and other members of the lab for insightful discussions. We are also grateful to Charles Venuto and Monica Javidnia for their advice on Parkinson's. s. This work was supported in part by Independence Blue Cross, Office of Naval Research Award No. N00014-17-1-2791, an Abdul Latif Jameel fellowship, and a NSF CAREER award.
References
Jason Abaluck, Leila Agha, Chris Kabrhel, Ali Raja, and Arjun Venkatesh. 2016. The determinants of productivity in medical testing: Intensity and allocation of care. American Economic Review 106, 12 (2016), 3730-64.
Anna Aizer and Joseph J Doyle Jr. 2013. Juvenile Incarceration, Human Capital and Future Crime: Evidence from Baaddomly-Asignann Judges. Technical Report w19102. National Bureau of Eco- nomic Research.
American Diabetes Association. 2010. Standards of medical care in diabetes 20100 Diabetes care 33, Supplement (2010), S11-S61.
David Arnold, Will Dobbie, and Crystal S Yang. 2018. Racial Bias in Bail Decisions. The Quarterly Journal of Economics 133, 4 (May 2018), 1885-1932.
Femke Atsma, Glyn Elwyn, and Gert Westert. 2020. Understanding unwarranted variation in clinical practice: a focus on network effects, reflective medicine and learning health systems. International Journal for Quality in Health Care 32, (jun 2020), 271-274. https://doi. org/10...rg11.. /ttgg//..991...
intqhc/mzaa023
John D Birkmeyer, Bradley N Reames, Peter McCulloch, Andrew J Carr, W Bruce Campbell, and John E Wennberg. 2013. Understanding of regional variation in the use of surgery. The Lancet 382, 9898 (2013), 1121-1129.
Christopher M Bishop. 2006. Pattern recognition and machine learning. springer.
Michael D Cabana, Cynthia S Rand, Neil R Powe, Albert W Wu, Modena H Wilson, Paul-André C Abboud, and Haya R Rubin. 1999. Why don't 't physicians follow clinical practice guidelines?: framework for improvement. Jama 282, 15 (1999), 1458-1465.
David C Chan Jr, Matthew Gentzkow, and Chuan Yu. 2019. Selection with variation in diagnostic skill: Evidence from radiologists. Technical Report. National Bureau of Economic Research.
Ashley N Corallo, Ruth Croxford, David C Goodman, Elisabeth L Bryan, Divya Srivastava, and Therese A Stukel. 2014. A systematic review of medical practice variation in OECD countries. Health Policy 1 14, 1 (2014), 5-1
Janet Currie and Bentley MacLeed. 2017. Diagnosing expertise: Human capital, decision making, and performance among physicians. Journal of labor economics 35, I (2017), 1-43.
Sanjoy Dasgupta and Leonard J Schulman. 2007. A probabilistic analysis of EM for mixtures of separated, spherical Gaussians. Journal of Machine Learning Research 8 (2007), 203-226.
Judith D De Jong, Gert P Westert, Ronald Lagoe, and Peter P Groenewegen. 2006. Variation in hospital length of stay: do physicians adapt their length of stay decisions to what is usual in the hospital where they work? Health Services Research 41, (2006), 374-394.
Rafael Di Tella and Emnssoo Schargrodsky. 2013. Criminal Recidivism after Prison and Electronic Monitoring. The Journal of Political Economy 121, 1 (2013), 28-73.
Julia Dressel and Hany Farid. 2018 The accuracy, fairness, and limits of predicting recidivism. Science advances 4, 1 (Jan. 2018), caao5580.

---

Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. 2007. Kernel measures of conditional dependence.. In NeurIPS, Vol. 20. 489-496.
Andrew Gelman and Jennifer Hill. 2006. Data Analysis Using
Regression
and
Multileve//Hierarchical Models, Cambridge University Press.
Christopher G Goetz, Stanley Fahn, Pablo Martinez-Martin, Werner Poewe, Cristina Sampaio, Glenn Stebbins, Matthew B Stern, Barbara C Tilley, Richard Dodel, Bruno Dubois, et al. 2007. Movement Disorder -ociety-sponsooed revision of the Unified Parkinson's 's Disease Rating Scale (MDS-UPDRS); process, format, and clinimetric testing plan. Movement disorders 22, (2007), 41-47.
Guideline Development Group, Henk Bilo, Luis Coentrão, Cécile Couchoud, Adrian Covic, Johan De Sutter, Christiane Drechsler, Luigi Gnudi, David Goldsmith, James Heaf, et al. 2015. Clinical practice guideline on management of patients with diabetes and chronic kidney disease stage 3b or higher (eGFR< 45 mL/min). Nephrology Dialysis Transplantation 30, suppl_2 (2015), iil-ii122.
Kidney Disease: Improving Global Outcomes (KDIGO) CKD-MBD Work Group et al. 2009. KDIGO clinical practice guideline for the diagnosis, evaluation, prevention, and treatment Chronic Kidney Disease- and Bone Disorder (CKD-MBD). Kidney international. Supplement 113 (2009), S1-S130.
PD Med Collaborative Group et al. 2014. Long-term effectiveness of dopamine agonists and monoamine oxidase B inhibitors compared with levodopa initial treatment for Parkinson's 's disease (PD MED); a large, open-label, pragmatic randomised trial. The Lancet 384, 9949 (2014), 1196-1205.
M A Hernán and J M Robins. 2020. Causal Inference: What If. Chapman & Hall/CRC, Boca Raton.
George Hripesak, Patrick B Ryan, Jon D Duke, Nigam H Shah, Rae Woong Park, Vojtech Huser, Marc A Suchard, Martijn J Schuemie, Frank DeFalco, Adler Perotte, et al. 2016. Characterizing treatment pathways at scale using the OHDSI network. Proceedings of the National Academy of Sciences 113, 27 (2016), 7329-7336.
Jerry Kang, Judge Mark Bennett, Devon Carbado, Pam Casey, Nilanjana Dasgupta, David Faigman, Rachel Godsil, Anthony G. Greenwald, Justin Levinson, and Jennifer Mnookin. 2012. Implicit bias in the courtroom. UCLA Law Review 59, 5 (2012), 1 1124-1186.
David R Karger, Sewoong Oh, and Devavrat Shah. 2014. Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research 62, 1 (2014), 1-24.
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovee, Jens Ludwig, and Sendhil Mullainathan. 2018 Human Decisions and Machine Predictions. The Quarterly Journal of Economics 133, (Feb. 2018), 237-293.
Jeffrey R Kling. 2006. Incarceration Length, Employment, and Earnings. The American economic review 96, (June 2006), 863-876.
Hoyt Koepke and Mikhail Bilenko. 2012. Fast prediction of new feature utility. arXiv preprint
arXiv:1206.466. (2012).
Zhiyuan (Jerry) Lin, Jongbin Jung, Sharad Goel, and Jennifer Skeem. 2020. The Limits of Human Predictions of Recidivism. Science Advances 7 (2020). https: org/10.....1.... 10..0.11.66 2/26/ccidd
aaz0652 ar
Kenneth Marek, Danna Jennings, Shirley Lasch, Andrew Siderowf, Caroline Tanner, Tanya Simuni, Chris Coffey, Karl Kieburtz, Emily Flagg, Sohini Chowdhury, et al. 2011. The parkinson progres- sion marker initiative (PPMI). Progress in neurobiology 95, 4 (2011) 1), 629-635
Sharon Muzerengi and Carl E Clarke. 2015. Initial drug treatment in Parkinson's 'ss disease. bmj 351 (2015).
Xinkun Nie and Stefan Wager. 2017. Quasi-oracle estimation of heterogeneous treatment effects. arXiv preprint 1712:1112.0091 (2017).
12

---

S Norris. 2020. Examiner inconsistency: Evidence from refugee decisions. Working paper.
Technical Report.
David L Ortega, H Holly Wang, Laping Wu, and Nicole Olynk. 2011. I. Modeling heterogeneity in consumer preferences for select food safety attributes in China. Food Policy 36, (2011), 318-324.
https://www www.scceieeecccccccccccceeeeeeee http:///ecceeeiree..ooo/oocaiii www.c//w....////...cc..oooocccee
perhhperformacce deep learning library. arXiv preprint ar http://w.......cccccccc.c.ccccaa
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, (2019).
Judea Pearl. 2009. Causality: Models, Reasoning, and Inference (2nd ed.). Cambridge University Press.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825-2830.
Joseph D Ramsey. 2014. scalable conditional independence test for nonlinear, non-gaussian data. arXiv preprint arXiv::401 140,50331 (2014).
Microsoft Research. 2019. EconML: A Python Package for ML-Based Heterogeneous Treatment Version 0.x.
Effects Estimation.
Michael A Ribers and Hannes Ulirich. 2020. Machine Predictions and Human Decisions with
Variation in Payoffs and Skill. (2020).
Jakob Runge. 2018. Conditional independence testing based on a nearest-neighhor estimator of con- ditional mutual information. In International Conference on Artificial Intelligence and Statistics. PMLR, 938-947.
Riccardo Scarpa, George Philippidis, and Fiorenza Spalatro. 2005. Product-country images and pref- erence heterogeneity for Mediterranean food products: discrete choice framework. Agribusiness 21, 3 (2005), 329-349.
Rajen Shah and Jonas Peters. 2020. The Hardness of Conditional Independence Testing and the Generalised Covariance Measure.
[math.ST]
Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding Machine Learning: From Theory to Algorithms, Vol. 9781107057. Cambridge University Press. hhttps://doi.
CB09781107298019
Uri Shalit, Fredrik D Johansson, and David Sontag. 2017. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference Machine Learning. PMLR, 3076-3085.
Le Song, Alex Smola, Arthur Gretton, Karsten M Borgwardt, and Justin Bedo. 2007. Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning.
AA Tahrani, GI Varughese, Scarpello, and FWF Hanna. 2007. Metformin, heart failure, and lactic
acidosis: is metformin absolutely contraindicated? Bmj 335, 7618 (2007), 508-512.
Peter W Tuerk, Martina Mueller, and Leonard E Egede. 2008. Estimating physician effects on glycemic control in the treatment of diabetes: methods, effects sizes, and implications for treatment policy. Diabetes care 31, 5 (May 2008), 869-873.
Stefan Wager and Susan Athey. 2018. Estimation and inference of heterogeneous treatment effects using random forests. J. Amer. Statist. Assoc. 113, 523 (2018), 1228-1242.
Michael L. Waskom. 2021. seaborm: statistical data visualization. Journal of Open Source Software 6, 60 (2021), 3021. https://doi. htt.::/o...rr..11111111/1/11./// 03021
13

---

Larry Wasserman. 2004. All of Statistics: A Concise Course in Statistical Inference. Springer, New York, NY.
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier Movellan, and Paul Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22 (2009),
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. 2012. Kernel-based conditional independence test and application in causal discovery. arXiv preprint arXiv:1202.3775 (2012).
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We present the formalized problem and solution we describe in the abstract. We also run experiments that support the application examples
we mention in the abstract.
(b) Did you describe the limitations of your work? [Yes] Assumption must hold.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section I
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In Theorem we collect the full set of assumptions in Assumption D and for Theorem 2 we give the relevant assumptions immediately before the statement of the theorem in Section 3.4
(b) Did you include complete proof' all theoretical results? [Yes] All proofs can be found in Section of the Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either the supplemental material or as URL)? [Yes] We will submit code and instructions with the supplement. The recidivism and Parkinson's ' data are publicly available. The only exception is the diabetes data is proprietary from an insurance company collaborator.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2 for how the region size hyperparameter is tuned, Appendix B.2 for how other hyperparameters are tuned, and other details in the paper. (c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See our figures.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We discuss this in the upfront overview section of the Appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We cite Lin et al. (2020) for the recidivism dataset. We cite Marek et al. (201T) for the Parkinson's dataset. The insurance data is proprictary.
(b) Did you mention the license of the assets? [Yes] For the recidivism dataset, the data repository does not have license at httttp:://githe com/ ctmmcoomoiiitee htt/staaffodd-ao stanford-pollcyl poldlcllllll/////oo rrcidivism-preede wwwpppriiioonnnee For the Parkinson's dataset, we include the disclaimer from PPMI.
(c) Did you include any new assets either in the supplemental material [N/A] We do not curate/release any new assets, other than code for replicating our experiments; This code will be available in the supplementary material, as discussed above.
URL?
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Appendix C regarding consent for the proprietary health insurance claims dataset. All other data we use is publicly available.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] See Appendix regardding de-identification of the health insurance dataset. All other data use is publicly available.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] The crowdsourced dataset was collected by another paper and publicly released.
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "sc9DIpJngdw", "reviewer": "Reviewer_Y7AH", "summary": "The authors are concerned with identifying regions in a covariate space. A region is characterised by having high variation in (potential) treatment outcome, as compared to the expected (potential) outcome. A method such as this could be used to identify variation of practice in the legal or medical domain (as indicated by the authors). \n\nThis is a relevant topic which the authors rightfully relate to the causal literature. Specifically, the authors chose the potential outcomes model, where the treatment is now a clinician or judge; the outcome is binary (amounting to a decision, e.g. bail or no bail); and the condition is an individual. Contrasting potential outcomes, the authors are interested in learning subsets of X, rather than Y(a).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper clear and well written, the topic relevant, and the presented method interesting. However, there are a few concerns which I have listed below.\n\n### Major\n* On overlap. In Assumption~1 (line 68), the authors do not assume overlap (0 < p(A | X =x) < 1 for all x). They motivate this in the remark starting at line 104. The motivation seems flawed to me; line 108 \"If judge a makes bail decisions primarily for violent crimes, but some other judges never see violent crimes, then we cannot hope to compare Y(a) to the decisions of those alternative judges.\", this is correct. However, what if we want to know the potential outcome of placing a violent crime case to those alternative judges? The reason we make the overlap assumption is such that we can ask the model for an estimate of these edge-cases. By using a propensity (such as the authors suggest on line 111), the contribution of those alternative judges is minimal, but if we wish to make an estimate for these judges, we do want some data which we can consider. If those propensities were 0, we would have a biased estimate.\n* There exists literature on partitioning for heterogeneous treatment effects which remains undiscussed [1, 2, 3, 4, 5]. This literature contains important discussion, as well as important benchmarks for the authors' experiments. \n\n### Minor\n* line 94, \"[...] if the agent-specific bias is non-negative on S.\", I'm guessing this should be non-zero? If the bias is negative, then there is still bias, correct?\n\n[1] HS Lee et al. \"Robust Recursive Partitioning for Heterogeneous Treatment Effects with Uncertainty Quantification\" NeurIPS 2020\n\n[2] K Imai et al. \"Estimating treatment effect heterogeneity in randomized program evaluation.\" Annals of Applied Statistics 2013\n\n[3] JC Foster et al. \"Subgroup identification from ran- domized clinical trial data.\" Statistics in Medicine 2011\n\n[4] LL Doove et al. \"A comparison of five recursive partitioning methods to find person subgroups involved in meaningful treatment– subgroup interactions.\" Advances in Data Analysis and Classification 2014\n\n[5] S Athey & G Imbens \"Recursive partitioning for heterogeneous causal effects.\" PNAS 2016", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work presented in this paper taps into an active area of research (subgroup analysis of HTE). The biggest limitation, in my opinion, is (largely) not acknowledging this existing brand of literature. Furthermore, I believe the authors should benchmark against (at least some of) these existing methods (cited above).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "Q9Pb0Gq80ME", "reviewer": "Reviewer_v34w", "summary": "Summary:\n\nThis paper considers settings where decisions for individuals are each made by different decision makers (agents). The paper defines the problem of finding a region in feature space in which the agents administer highly varying decisions. For example, perhaps judges mete out different sentences for misdemeanors by young men. The paper defines new causal criteria to characterize such regions in the feature space, where the choice of agent has an effect on the decision. The paper introduces an iterative algorithm to maximize this criteria to identify regions of heterogeneity, and provides theoretical results about the optimality of the proposed algorithm. The paper empirically demonstrates the algorithm on a semi-synthetic dataset and real clinical data about doctor recommendations for diabetic patients.\n\nContributions:\n\n+ A new criteria for defining regions where there is heterogeneity in decisions administered across agents.\n+ Results for estimating the criteria from observational data and an iterative algorithm for optimizing the criteria.\n+ Theoretical results about the algorithm and empirical validation of the algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThis paper presents an original estimation problem and algorithm to solve it. To the best of my knowledge, this paper provides a new formalization of the problem of wanting to learn about decision making heterogeneity and proposes a new criteria for measuring this heterogeneity based on regions of input space in which agents differ in their decision making.\n\nThe iterative algorithm is a reasonable optimization procedure and the theoretical results are obviously new for this algorithm, but probably borrow ideas from the optimization literature (I haven't checked the proof.)\n\nQuality:\n\nAlthough I haven't checked the proofs for the theorems, the results pass the sniff test. I looked at the algorithm closely to follow along, and there are again no surprises there, which is good.\n\nAt a high level, my one complaint is about the beginning about section 2 and Eqn 1. I feel that causality is almost incidental to the paper because of the way the estimand in Eqn 1 is defined. Indeed, as the paper says, the challenge of causal inference is if we needed to evaluate Y(a') for a person who was assigned A=a and got Y(a) = Y|A=a. But in Eqn 1, we don't consider the impossible counterfactual Y(a') because we contrast the observed outcome with a different quantity, the average decision over agents.\n\nEqn 1 could be rewritten as:\n$$\\mathbb{E}[Y|A=a, X \\subset S] - \\mathbb{E}[Y|X \\subset S]$$\nThe interpretation of the agent bias criteria still holds: we can think of it as the difference between a's decision and the expected decision across agents for that region of feature space. This formulation also connects the algorithm and criteria to the \"direct\" baseline.\n\nMainly, I wonder if invoking causality is an unnecessary distraction in this paper. Can the authors explain why we need to consider counterfactual notion from their view?\n\nAnother technical point where I have a doubt is: how does this algorithm fare when we have a high dimensional feature space? In the empirical studies, both settings consider just a few features. Is that because the algorithm fails with high dimensional data, or is it for ease? Theorem 2 does show a positive dependence between the \"size\" of S and the error term -- does this turn out to be a practical hindrance, restricting the algorithm to small feature sets?\n\nI think the empirical studies are thorough, with both a semi-synthetic study and a real world case study where we have validation based on domain knowledge. I think especially given that the authors had to come up with their own baselines, they did a good job of designing comparisons.\n\nHowever, I would've liked a better explanation about the motivation behind using TARNet baseline as its used. That is, what's the significance of predicting E[Y|X,A] and then calculating Var[E[Y|X,A]]?\n\nFinally, do the authors have a hypothesis for why Direct performs comparably with Iterative in the misdemeanor setting?\n\nClarity:\nThe paper is reasonably well written, with enough detail around the algorithm and experimental evaluation for a reader to implement the algorithm themselves or to understand the studies.\n\nThe authors also provided good plain English explanations about the agent bias criteria that they defined, which helped build intuition. There was also exposition to give intuition about the algorithm. \n\nHowever, I think that the notation of the form $\\mathbb{E}[Y | X \\subset S]$ is unconventional and needs a clear, mathematical explanation prior to equation 1. Initially, I interpreted it as E[Y|X], which is a random variable in X, and I got extremely confused with the equivalence in Eqn 2, which clearly evaluates to a scalar. \n\nI might even give Eqn 2 right away without Eqn 1 with more explanation about the notation. The intuitions that came after Eqn 2 were important for understanding the interpretation of the quantity.\n\nSignificance:\n\nThe paper formalizes the problem of decision making heterogeneity and offers a practical procedure to find regions where agents make different decisions. I think this offers a significant contribution to domain experts as well as researchers in causal inference who will want to build upon this work.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There isn't much a discussion about the limitations of this method, which I would've liked to see. This is a room for improvement.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "I-kyBKn9Mir", "reviewer": "Reviewer_DZe8", "summary": "In this paper, the authors develop an algorithm for identifying cases (e.g., defendants or patients) on which decision makers (e.g., judges and doctors) \"disagree\" on the decision. \n\n## Identification\nThe authors consider a setting in which cases are summarized by features $X$. Each case is associated with a decision maker $A$ and a binary decision $Y$. In the pretrial release setting, each case is a defendant, the features $X$ are observable characteristics of the defendant (such as demographics or charge information), the decision maker $A$ is the assigned judge and the binary decision $Y$ is to either release or detain the defendant. The authors define the potential decision Y(a), which describes the decision that would be made by decision maker $a$. \n\nThe authors propose a summary measure of agent disagreement using average contrasts of potential decisions across agents given sets of the observable features. In particular, the authors focus on the \"conditional relative agent bias\" as \n$$ \\mathbb{E}[Y(a) - Y(\\pi(x) | A = a, X \\in S], $$\nwhere $Y(\\pi(x))$ denotes the expected decision under the assignment mechanism of decision makers to cases. The authors key identification assumption is that decision makers are as-if randomly assigned to cases conditional on the features (i.e., $A \\perp Y(a) \\mid X$). This measures the extent to which decision maker a's average decisions over $X \\in S$ differs from the average decision that is made under the assignment mechanism. In other words, this measure of disagreement depends on the assignment mechanism of agents.\n\nThe conditional relative agent bias measures the extent to which a single agent a's decision disagrees relative to the average decision over $X \\in S$. The authors then propose aggregating this measure across agents. For some grouping of agents G, the authors define the estimand\n$$ Q(S, G) = \\sum_{a : G(a) = 1} P(A = a \\mid X \\in S) \\mathbb{E}[Y(a) - Y(\\pi(x) | A = a, X \\in S]. $$\nThe authors main identification result (Theorem 1) is to show that if agents are as-if randomly assigned to cases, then this estimand can be identified as the conditional covariance between the observed decision and the agent assignment\n$$ Q(S, G) = \\mathbb{E}[Cov(Y, G \\mid X ) \\mid X \\in S]. $$\n\n## Estimating Regions of Heterogeneity.\nUsing this identification result, the authors goal is to search over regions of the feature space $S$ and groupings over agents $G$ to find the combination that maximizes this proposed measure of disagreement. To do so, the authors propose to solve \n$$ \\max_{S, G} Q(S,G) \\mbox{ s.t. } P(X \\in S) \\geq \\beta, $$\nwhich requires that the region have probability at least $\\beta$ (a tuning parameter to be specified). The authors develop an iterative algorithm to optimize this objective function, and provide some conditions under which they are able to derive a generalization bound on the solution return after the first iteration (Assumptions 2-4 and Theorem 2).\n\n## Empirical Applications\nThe authors apply their proposed algorithm in two empirical applications. \n\nFirst, the authors analyze the behavior of their proposed algorithm in a semi-synthetic experiment that is calibrated to data from Lin et al. (2020) on recidivism risk predictions made by MTurk workers. The authors simulate data from two types of agents that follow different decision rules for detention vs. release. The authors compare their algorithm against direct models that fit logistic regressions for the decision on agent identifiers and observable features (direct models) and TARNet. The authors report that their algorithm does better at recovering the \"region AUC.\"\n\nSecond, the authors analyze an observational dataset on medical treatment decisions for diabetes. The data contain information on 3,956 patients and 458 group practices, and the authors search for the region of disagreement across these group practices.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found this paper to be interesting. The authors study a potentially important, policy relevant problem -- in many empirical settings such as the criminal justice and health care, it is useful to understand how much disagreement there is across decision makers and what types of cases have the most disagreement.\n\n## Significance\n\nMy main concern with the paper surrounds the authors' proposed measure of disagreement across agents. \n\n1) As mentioned above, the authors' proposed measure of disagreement (\"conditional relative agent bias\") measures the difference in the decisions of some fixed agent a against the average decision taken under the agent assignment mechanism.\n\nThis leads to strange properties when overlap fails. Consider a simple example where $X = \\{0, 1\\}$ and there are only two agents. Suppose the agents always make opposite decisions $Y(a_1) \\neq Y(a_2)$ and $P(A = a_1 | X = 0) = 1, P(A = a_2 | X = 1) = 1$. In this case, the conditional relative agent bias is zero. This occurs because there is no overlap across the two agents.\n\nOf course, as the authors point out an overlap assumption, this is not identified. But clearly this is a DGP in which we would naturally say there is a lot of disagreement across agents. However, their estimand would there is no disagreement across agents.  I think this deserves more discussion in the paper. \n\n2) The authors' proposed measure of disagreement is also not signed. It would appear to be more natural to consider an absolute conditional relative agent bias? In this way, disagreement in any direction contributes to overall disagreement. In contrast, with this current definition, it seems like you could have a setting where one group of agents is more likely to choose Y = 1 on average, and a second group of agents is less likely to choose Y = 0 on average but these positive vs. negative biases of the groups could cancel each other out. Is this not possible? If it is possible, why should we not be worried about understanding disagreement with this objective? \n\n3) The authors aggregate their proposed measure of disagreement into the estimand $Q(S, G)$. Again, the fact that the authors use the agent assignment mechanism leads to some strange behavior here. In particular, it implies that for any $S$, $Q(S, G) = 0$ if $G$ is equal to the set of all agents. This can be most clearly seen from their identification result in Theorem 1. If $G$ always equals 1, then \n$$E[(Y - E[Y | X]) G \\mid X \\in S] = E[(Y - E[Y | X]) \\mid X \\in S] = E[Y \\mid X \\in S] - E[Y \\mid X \\in S] = 0$$.\nThis is also strange since it implies that disagreement is zero whenever G is chosen to be all possible agents. That's not an intuitive property of a measure of disagreement and deserves some discussion. \n\nSince this estimand is the core object of the paper, it is essential for the authors to clearly motivate it, and discuss its limitations. The authors have not sufficiently done so in this paper. I have other comments about the significance of these results as well:\n\n4) The authors assume that agents are assigned to cases as-if randomly conditional on the features $X$. In many empirical applications of interest, such as the cited examples in the pretrial release system (e.g., Arnold, Dobbie & Yang 2018; Kleinberg et al. 2018 which the authors cite), decision makers are \\textit{quasi-randomly} assigned to cases conditional on some location-by-time cells (i.e., judges working in a given courthouse/county on a given day of the year are as-if randomly assigned to cases). Denote these location-by-time cells by $T$. The typical assumption in empirical applications is $A \\perp (Y(a), X) | T$, which of course implies that $A \\perp Y(a) | T, X$. \n\nUnder this assumption, the authors methods search for the region of heterogeneity over both (T, X). Can this be extended to only search for a region of heterogeneity that only depends on X? For example, policymakers may be interested in only measuring heterogeneity across defendant characteristics but not across the court-by-time cell itself.  Discussing this case would enhance the empirical relevance of these methods.\n\n5) The authors emphasize that avoiding the overlap/positivity assumption is a key contribution of the paper. \n\nIs this a common problem in practice? For example, if decision maker's are totally randomly assigned (for example, in my comment #3 imagine focusing on a single court-by-time cell), then A \\perp (Y(a), X). If P(X) > 0, then overlap is satisfied. It would be useful for the authors to provide some concrete examples (perhaps in their second empirical application?), where the failure of overlap is a serious problem.\n\n6) The authors require strong assumptions for their derivation of the generalization error in Section 3.4 -- e.g., only two groups of agents that follow distinct policies in Assumption 2 and a positivity type condition in Assumption 4. It would be useful for the authors to succintly summarize the roles of these assumptions in the paper. Could they be relaxed in any way? Moreover, it is strange that the authors introduce a positivity type condition in their error analysis but spend much of Section 2 emphasizing that they don't need to make this assumption for the identification result. If I need to make a positivity type condition for estimation, why wouldn't I be willing to make that same assumption when I determining what is identified from the DGP? \n\n7) Is the conditional random assignment assumption a reasonable assumption in the second empirical application to diabetes treatment? For example, if this is a primary care setting, there are many reasons we would suspect that patients are not randomly assigned to doctors -- e.g., patients may self-select to particular doctors based on information that we do not observe, doctors may have closed practices that limit the number of patients that they may see. The authors should discuss why this assumption is reasonable in this setting. \n\n## Originality \n\nThe authors study an important question about measuring heterogeneity in the choices of decision makers. I agree with the authors assessment that this question is often overlooked in statistics and computer science. However, there is an enormous and active literature studying heterogeneity across decision makers in economics (some of which they cite such as Arnold, Dobbie, Yang 2018 and Kleinberg et al 2018).\n\nFor example, recent work such as \n* Abaluck et al. (2016) (\"The Determinants of Productivity in Medical Testing: Intensity and Allocation of Care\")\n* Currie & Macleod (2017) (\"Diagnosing Expertise: Human Capital, Decision Making and Performance Among Physicians\")\n* Arnold, Dobbie and Hull (2020) (\"Measuring Racial Discrimination in Bail Decisions)\n* Chan, Gentzkow and Yu (2020) (\"Selection with Variation in Diagnostic Skill: Evidence from Radiologists\")\n* Norris (2020) (\"Examiner Inconsistency: Evidence from Refugee Decisions\")\n* Ulrich & Ribers (2020) (\"Machine predictions and human decisions with variation in payoffs and skill\") \nstudy this same question in the same empirical settings studied by the authors (i.e., criminal justice and health care). \n\nThese papers typically place behavioral/structural assumptions on what the authors call the \"potential decision\" and estimate heterogeneity across decision makers using discrete choice modelling. This additional structure is often useful for domain experts since it can be leveraged to ask and answer interesting policy counterfactuals (\"how would decisions change if all decision makers have the same threshold, etc.). In this sense, the authors approach is more \"nonparametric\" by not modelling the potential decision, but that has a cost since it is unclear how the resulting estimand they focus on is directly useful for domain experts.  Since this question is not novel relative to this large literature in economics, it would be useful for the authors to carefully discuss how their methods and approach differ from this existing research. \n\n## Clarity\n\nThere are two places where I found the paper to be unclear:\n\n1) In their empirical application to recidivism prediction, the authors report the \"Region AUC.\" I think this compares the AUC of region classifier S returned by their own model (which classifies whether X is in the region) vs. the ground truth region. Is that right? If so, it would be useful to explicitly define this in the main text.\n\n2) In Section 3.4, the authors introduce notation $S^*, G^*$ to describe the optimal $S^*, G^*$ that maximize the objective. Are these assumed to be unique?\n\n3) I was confused why there is some notion of a true parameter $\\beta$? I would have thought that $\\beta$ is a tuning parameter to be set by the user, not something that would be estimated/calibrated by the data. In other words, it is up to the user to specify what the appropriate region size is. \n\n## Update:\nI read the other reviews and the authors' responses. I thank the authors for clarifying some points that I raised on the interpretation of their proposed measure of disagreement. Provided the authors make these additional clarifications in the main text, I'd like to revise my score from 4 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see above for my comments on limitations and suggestions for improvement.\n\nI do not see any potential negative societal impact from this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "DY0JvQZpcx_", "reviewer": "Reviewer_9a1j", "summary": "This paper proposes a new method to identify \"regions of disagreement\" in a setting with human decision-makers assigned to individuals (eg. judges assigned to court cases). The \"regions\" are framed as subgroups of the data (eg. males over 30 years old) which can be interpretable (based on the chosen model). The paper formalizes this task as that of identifying the subgroup where the identity of the specific decision-maker has a causal effect on the decision, proposes an optimization algorithm to find this subgroup, develops generalization bounds (under some simplifying assumptions), and demonstrates compelling performance on semi-synthetic and real-world datasets.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## General Points\n\nFinding regions of \"disagreement\" or \"heterogeneity\" in the decisions made by human experts is important to diagnose decision-making issues. For example, if we find that judges are highly variable in the decisions they make for young males, it might be worth creating a uniform judgement policy for such males to improve judgement consistency, and reduce the potential impact of personal biases. If we find that teachers grading essays tend to disagree on the scores assigned to female students, it might be worth hiding student identities. In both cases, before enacting any policy changes, **we need a principled method to identify such regions of heterogeneity, which this paper provides.**\n\nThis specific task falls under the broader umbrella of *subgroup discovery*. One of my favorite recent examples of work in this space is [this paper](https://arxiv.org/pdf/1707.00046.pdf) on identifying regions/subgroups where 2 machine learning models (not humans) disagree (with a specific focus on disagreement  in fairness). This paper is also related to work on *heterogeneous treatment effect estimation* (the paper on [causal forests](https://www.tandfonline.com/doi/abs/10.1080/01621459.2017.1319839) is a canonical example of this space). **Unlike the aforementioned 2 papers, this paper does not use recursive partitioning and has a clever, novel formulation of the subgroups** (those where the causal effect of the assigned decision-maker is non-zero). I believe this formulation itself will be useful for the fairness literature, and that this paper can benefit from some of the ideas in the aforementioned 2 papers.\n\nI have summarized the key strengths and weaknesses of this paper below.\n\n## Key Strengths\n\n**Important problem and clever, novel formulation.** I have stressed the problem importance and novelty above. I want to note that the proposed formulation and optimization method is fairly flexible and can be made interpretable and tailored to specific domains. This helps expand the proposed method's applicability to a wide range of real world settings.\n\nI must also stress that the proposed causal estimand in eq. 2 is non-trivial to come up with: it is typically hard to come up with a causal estimand for a new task that makes intuitive sense, that is identifiable from observable data, and that is estimable from data in an efficient manner.\n\nIn addition, Theorem 1 transforms eq. 3 which contains the propensity scores P(a|s) into eq. 4 that does not require knowing these propensity scores. What this means is that, as long as the (conditional) randomization/no unobserved confounders is satisfied, we do not need to know the specific randomization protocol! This enables applying this method to settings like judges assigned to cases, and physicians assigned to patients, where we know that there is randomization but the specific assignment mechanism is proprietary/unknown. Without Theorem 1, this method would probably be restricted to randomized control experiments where the decision-maker assignment probabilities are known.\n\n**Clear and complete description of the causal task.** This paper states the causal estimand, the identification assumptions, and derives an identification proof to link unobservable counterfactual quantities to observable ones. As such, the description of the causal task is clear and complete. Importantly, the mathematical quantities are also described intuitively and linked to real-world quantities, which aids readers interested in applying this method to their setting.\n\n**Compelling synthetic and real world performance.** The paper empirically demonstrates performance in a well-described collection of experimental settings, one synthetic and two real world. The method is evaluated along several dimensions, and recipes are provided for essential real-world tasks such as interpreting and assessing the significance/stability of the identified region\n\n## Key Weaknesses\n\n**The proposed method finds the one region with maximum heterogeneity.** In practice, we can expect there to be multiple regions where the agents disagree; the proposed method would not help identifying all such regions. While it might be tempting to simply tweak the proposed method to find all such regions, it would be very difficult to address issues of multiple testing / false discovery (if we test enough regions of a large feature space, we will likely find at least a few that appear to have significant heterogeneity). The paper might want to clarify this in the problem definition, since the title is a bit misleading in this regard.\n\n**It is unclear how the method would perform and scale as the dimensionality of the feature space increases.** I expect that, with high dimensional $X$ and several irrelevant features, it would be harder to find valid regions of heterogeneity. In addition, I wonder how the running time / no. of iterations of the optimization algorithm scale with the size of the feature space, no. of decision makers, and no. of data samples.\n\n**While the no unobserved confounders and consistency assumptions are stated, its real world implication need to be clarified up front.** What would violate these assumptions in the real world (provide concrete examples)? It is very important to clarify, for future readers of this method, that you **need (conditional) randomized assignment of decision-makers for this method to work.** This is not a dealbreaker, but I can imagine future readers brushing over the no unobserved confounders assumption without realizing how restrictive it is.\n\n**While recipes to assess significance/stability are provided, it is unclear if these are statistically valid.** Ideally, one would have a way to construct \"confidence intervals\" around the identified region of heterogeneity. While this paper provides a recipe to test significance, I am not sure how statistically valid that is (can validity be demonstrated empirically?). The paper needs to caution readers about the potential for false discoveries without a statistically valid inference procedure.\n\n**Concerns with estimating the f(x) and h(x) models on the same data where the causal effect is estimated.** In the \"double machine learning literature\" where ML models are using along with causal inference, the ML models are estimated on a *separate sample* of the data. This is to disentangle the errors of the causal effect estimation and the errors arising from (i) overfitting the ML models, and (ii) underfitting the ML models. This paper does not consider what happens if f(x) and h(x) are poor models (does it lead to an increased or lowered risk of false discovery of regions of heterogeneity?).\n\n**Intuition behind the grouping objective in eq. 3.** The grouping mechanism was introduced without much intuition; could the equation be supplemented with a few descriptive lines? I believe because of the grouping, there is no need to place an absolute value around eq. 3, but I could be wrong.\n\n**Optimization guarantees.** Does the optimization method find a global optimum (I know most subgroup discovery methods are greedy and do not find the globally optimal region of heterogeneity, but it might be good to state this clearly to readers)?\n\n**Potential negative societal impact.** As a method of \"discovery\", there is is always the potential for false discoveries with bad consequences (especially when no statistical inference safeguards are provided). In addition, the models f(x) and h(x) could propagate biases in the data they are trained on, which might spill over in to the regions discovered in unexpected ways. The authors don't actually talk about the potential negative societal impacts in Section 1. It is essential that these concerns be mentioned early in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Potential negative societal impact.** As a method of \"discovery\", there is is always the potential for false discoveries with bad consequences (especially when no statistical inference safeguards are provided). In addition, the models f(x) and h(x) could propagate biases in the data they are trained on, which might spill over in to the regions discovered in unexpected ways. The authors don't actually talk about the potential negative societal impacts in Section 1 (they only state that Assumption 1 is a limitation). It is essential that these concerns be mentioned early in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work the authors propose a method for identifying regions (types of instances) where there is a high level of heterogeneity in in human decision making.  Their approach is grounded in a causal inference framework that models high-heterogeneity regions as ones where the choice of decision maker ("treatment") has a large effect on the resulting decision.  

This paper received a number of excellent high-quality reviews that raised important concerns and requests for clarification.  The authors responded well to the major concerns raised by the different reviewers and have clearly articulated how they will revise the manuscript to improve clarity and further contextualize the work within a broader set of relevant literature.  I agree with the reviewers that this paper tackles an important problem and that the work meets the bar both in terms of technical contribution and exposition.  In revising the manuscript, the authors should seek to respond to as many of the reviewer suggestions as possible, with an emphasis on:

- Adding and briefly discussing the related work suggested by reviewers
- Making assumptions more tangible by giving clear examples of when they may or may not hold in practice
- Clarifying the particular type of overlap assumption upon which the work relies, and contrasting with all-agent overlap
- Clarifying why agent-specific biases do not cancel out in the aggregate objective
Decision: Accept (Poster)