Finding Regions of Heterogeneity in Decision-Making via Expected Conditional Covariance

1 Introduction . Understanding heterogeneity in decision-making is an established problem in medicine ( Birkmeyer et al. , 2013 ; Corallo et al. , 2014 ; De Jong et al. , 2006 ) , consumer choice ( Ortega et al. , 2011 ; Scarpa et al. , 2005 ) , and law ( Kang et al. , 2012 ; Kleinberg et al. , 2018 ; Arnold et al. , 2018 ) . In the context of medicine , this is referred to as the study of practice variation ( Atsma et al. , 2020 ; Cabana et al. , 1999 ) , where it is often observed that doctors , facing the same clinical context , will make different decisions . Likewise , in a legal context , judges often differ in their leniency in their decisions regarding bail ( Kleinberg et al. , 2018 ) , juvenile incarceration ( Aizer and Doyle Jr , 2013 ) , the use of alternatives to incarceration ( Di Tella and Schargrodsky , 2013 ) , and incarceration length ( Kling , 2006 ) . In some scenarios this variation may be justified : The best medical treatment may not be obvious . In others , it may be grossly unfair , as in the case of racial bias in bail decisions ( Arnold et al. , 2018 ) . ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we tackle the question of how to find and characterize this variation in the first place . In particular , we present a learning algorithm for identifying a “ region of heterogeneity ” , defined as a subset of all contexts ( e.g. , patients , cases ) for which the identity of the decision-maker substantially affects the decision . In medicine , a better understanding of treatment variation can inform the development and dissemination of clinical guidelines . In the legal domain , characterizing the cases where judges vary most in their leniency may help with investigating potential issues of fairness . We formalize characterizing the region of heterogeneity as a causal inference problem : We want to characterize examples where changing the decision-maker would have resulted in a different decision . The challenge is two-fold : First , we only observe a single decision-maker per example , so we can not directly observe how ( for instance ) multiple judges would have decided the same case . Second , our data on individual decision-makers is often scarce . For instance , in Section 5 , we consider a medical dataset with more than 400 doctors , each of whom has fewer than 9 samples on average . We will refer to decision-makers as “ agents ” , and our contributions are as follows : In Section 2 , we propose an objective defined in terms of counterfactual decisions across different agents , and show that this objective can be identified from observational data . Moreover , this objective does not require the use of agent-specific statistical models , making it amenable to our sparse setting . In Section 3 , we give an iterative algorithm to identify regions of disagreement by maximizing this objective and provide intuition ( in the form of a generalization bound ) for the factors that drive its performance . In Section 4 , we use a semi-synthetic dataset , derived from crowd-sourced recidivism predictions , to demonstrate that our algorithm recovers the correct region of heterogeneity accurately , even when there are many agents . Finally , in Section 5 , we apply our algorithm to a real-world healthcare dataset and confirm that it recovers intuitive regions of variation in first-line diabetes treatment . We conclude with a discussion of related work and implications . Our code is available at https : //github.com/clinicalml/finding-decision-heterogeneity-regions . Our algorithm does not determine whether variation is inherently good or bad or how it should be addressed . Rather , more careful study with domain experts would be required to determine if variation can ( or should be ) reduced and how . In addition , false discovery of variation is possible and could have a negative impact . We expect that validation on independent datasets would be required in real-world applications , using the regions identified by our method as plausible hypotheses to test . 2 Characterizing Heterogeneity from a Causal Perspective . 2.1 Notation . Let the data be drawn from a distribution P ( X , A , Y ) , where X is a random variable representing context ( or features ) , A is a discrete agent , and Y ∈ { 0 , 1 } is the binary decision . The spaces of all X and A are denoted as X andA with realized values as lower case x and a , respectively . Indicator variables 1 [ · ] are one if the statement inside the brackets is true and zero otherwise . For a subset S ⊆ X , 1 [ x ∈ S ] is sometimes written as a function S ( x ) , where S : X → { 0 , 1 } . A subset S may have several disjoint regions . E [ Y |X ∈ S ] denotes the average Y across samples in S . For instance , if S = { X : X0 < 10 } , then E [ Y |X0 < 10 ] is a scalar average of Y among samples with X0 < 10 . 2.2 Heterogeneity as a Causal Contrast . Our conceptual goal is to identify a region S ⊆ X where different agents tend to make different decisions even when faced with the same context . We can formalize this in the language of potential outcomes from the causal inference literature ( Pearl , 2009 ; Hernán and Robins , 2020 ) , which for clarity we will refer to as potential decisions : In particular , we denote Y ( a ) to be the potential decision of agent a . The fundamental challenge of causal inference is that we do not observe all potential decisions { Y ( a ) : a ∈ A } for each sample , but only a single decision Y . With this in mind , we will make the following assumptions , standard in the literature on causal effect estimation . Assumption 1 ( Causal Identification Assumptions ) . ( i ) Consistency : Y = y , A = a =⇒ Y ( a ) = y , and ( ii ) No Unmeasured Confounding ( NUC ) : For all a ∈ A , Y ( a ) ⊥ A | X . Consistency links the potential Y ( a ) to the observed Y , and NUC says that there are no unobserved factors that influence both the assignment of agents and the decision itself . For instance , the quasirandom assignment of cases to judges conditioned on features X satisfies NUC ( Kleinberg et al. , 2018 ) . NUC may be violated if key aspects of the case ( e.g. , misdemeanor vs. felony ) are omitted as features . For instance , misdemeanor and felony cases may be seen by different judges and have different decision processes , but this variation is not due to agent preferences . Given these assumptions , we propose a causal measure of agent-specific bias , defined as a contrast between potential decisions . Definition 1 ( Conditional Relative Agent Bias ) . For an agent a ∈ A and a subset S ⊆ X , the conditional relative agent bias is defined as E [ Y ( a ) − Y ( π ( x ) ) | A = a , X ∈ S ] ( 1 ) where Y ( a ) is the potential decision of agent a , and Y ( π ( x ) ) B ∑ a′ E [ Y ( a′ ) | x ] π ( a′ | x ) denotes the expected potential decision under the agent assignment distribution π ( a′ | x ) B P ( A = a′ | X = x ) . Note that under Assumption 1 , Y ( π ( x ) ) = E [ Y | x ] ,2 but here we emphasize its causal interpretation as the expected decision of a random agent . Equation ( 1 ) represents the relative difference between the decision of an agent ( on their particular distribution of cases in the region ) and the potential decision of a random agent . In particular , Equation ( 1 ) can be written as follows under Assumption 1 E [ Y ( a ) − Y ( π ) | A = a , X ∈ S ] = ∫ x∈S E [ Y ( a ) − Y ( π ) | X = x ] p ( x | A = a , X ∈ S ) dx , ( 2 ) where we shorten Y ( π ( x ) ) to Y ( π ) . This is the average difference ( over p ( x | a ) , restricted to those x in the set S ) of the conditional expected difference between Y ( a ) and Y ( π ) . For example , suppose that the agent a is a judge who is particularly lenient on bail decisions for felony arrests ( the region S ) , and Y = 1 denotes granting bail . Then imagine the following counterfactual : Take the felony cases that are assigned to this judge and reassign each individual case , described by x , to a random judge a′ , proportionally to p ( a′ | x ) . We may then observe , on average , that the bail rate would decrease , because most judges are less lenient than judge a , corresponding to a positive value of Equation ( 1 ) . Equation ( 1 ) has the additional advantage of being easy to estimate : Under Assumption 1 , it can be rewritten3 as E [ Y−E [ Y | X ] | A = a , X ∈ S ] , the expected residual in predicting ( using the conditional expectation E [ Y | X ] ) the decision of an agent a across the context x typically seen by that agent . 2.3 Formalizing a Causal Objective . Our primary goal is to discover a region S where substantial heterogeneity exists across agents . To do so , we define an aggregate objective across a group G of agents , where G ( a ) ∈ { 0 , 1 } is an indicator function for membership . Q ( S , G ) B ∑ a : G ( a ) =1 P ( A = a | X ∈ S ) E [ Y ( a ) − Y ( π ) | A = a , X ∈ S ] , ( 3 ) We now show that this quantity can be identified and estimated from observational data without requiring agent-specific statistical models , before discussing the interpretation of this objective . Theorem 1 ( Causal Identification ) . Under Assumption 1 , Q ( S , G ) can be identified as Q ( S , G ) = ES [ Cov ( Y , G | X ) ] = ES [ ( Y − E [ Y | X ] ) G ] , ( 4 ) where ES [ · ] B E [ · | X ∈ S ] and Cov ( Y , G | X ) is the conditional covariance . Theorem 1 and other theoretical results are proven in Appendix A . The result follows from proving that the agent-specific bias ( Definition 1 ) is identifiable using the expected conditional covariance between Y and the binary indicator 1 [ A = a ] . With this in mind , we optimize the following objective , where the set S is constrained to be at least a certain size β and S is a hypothesis class of functions S . max S∈S , G Q ( S , G ) s.t. , P ( S ) ≥ β , ( 5 ) Interpretation : Intuitively , this objective measures the disagreement between the agents in the group G ( a ) = 1 and the overall average E [ Y | X ] on the region S . Hence , the choice of group is important for interpreting the objective : If G ( a ) = 1 for all agents , the objective will be zero for any set S , as can be seen from Equation ( 4 ) , applying the definition of the conditional expectation . 2See Proposition A1 for a short proof , and Proposition A2 for the derivation of Equation ( 2 ) . 3See Proposition A3 in Appendix A.1 . Accordingly , we seek a region S for which the partially maximized objective L ( S ) B maxG Q ( S , G ) is large : This partial maximization is obtained by taking G ( a ) = 1 whenever the conditional relative agent bias of agent a ( on the set S ) is non-negative . Thus , Equation ( 5 ) can be re-written as max G Q ( S , G ) = ∑ a∈A P ( A = a | X ∈ S ) |E [ Y ( a ) − Y ( π ) | A = a , X ∈ S ] |+ , ( 6 ) where |x|+ B max ( x , 0 ) , and this objective becomes an average over agents who have a positive bias . This population objective is also equivalent ( up to a constant factor ) to the ( weighted ) sum of the magnitude of each agent ’ s conditional relative agent bias . See Proposition A4 in Appendix A.1 . Lack of Overlap : We have not made the overlap or positivity assumption that P ( A = a | x ) > 0 for all x , a . While this assumption is required to identify conditional causal effects E [ Y ( a ) − Y ( a′ ) | X ] ( Nie and Wager , 2017 ; Wager and Athey , 2018 ; Shalit et al. , 2017 ) , it is not required for identifying our causal contrast . Our problem only requires each context has a positive probability of being seen by more than one decision maker . For instance , suppose that S contains both misdemeanors and felonies and there are four judges a0 , . . . , a3 . If judges a0 and a1 make bail decisions exclusively for felonies while judges a2 and a3 make bail decisions exclusively for misdemeanors , our measure captures disagreement between a0 and a1 and between a2 and a3 even though comparisons between a0 and a2 or other pairs are impossible to make . Thus , we have chosen to compare Y ( a ) to the decisions of viable alternative agents , weighted by their probability p ( a′ | x ) of being selected for such a case .

Summary: In this paper, the authors develop an algorithm for identifying cases (e.g., defendants or patients) on which decision makers (e.g., judges and doctors) "disagree" on the decision.   ## Identification The authors consider a setting in which cases are summarized by features $X$. Each case is associated with a decision maker $A$ and a binary decision $Y$. In the pretrial release setting, each case is a defendant, the features $X$ are observable characteristics of the defendant (such as demographics or charge information), the decision maker $A$ is the assigned judge and the binary decision $Y$ is to either release or detain the defendant. The authors define the potential decision Y(a), which describes the decision that would be made by decision maker $a$.   The authors propose a summary measure of agent disagreement using average contrasts of potential decisions across agents given sets of the observable features. In particular, the authors focus on the "conditional relative agent bias" as  $$ \mathbb{E}[Y(a) - Y(\pi(x) | A = a, X \in S], $$ where $Y(\pi(x))$ denotes the expected decision under the assignment mechanism of decision makers to cases. The authors key identification assumption is that decision makers are as-if randomly assigned to cases conditional on the features (i.e., $A \perp Y(a) \mid X$). This measures the extent to which decision maker a's average decisions over $X \in S$ differs from the average decision that is made under the assignment mechanism. In other words, this measure of disagreement depends on the assignment mechanism of agents.  The conditional relative agent bias measures the extent to which a single agent a's decision disagrees relative to the average decision over $X \in S$. The authors then propose aggregating this measure across agents. For some grouping of agents G, the authors define the estimand $$ Q(S, G) = \sum_{a : G(a) = 1} P(A = a \mid X \in S) \mathbb{E}[Y(a) - Y(\pi(x) | A = a, X \in S]. $$ The authors main identification result (Theorem 1) is to show that if agents are as-if randomly assigned to cases, then this estimand can be identified as the conditional covariance between the observed decision and the agent assignment $$ Q(S, G) = \mathbb{E}[Cov(Y, G \mid X ) \mid X \in S]. $$  ## Estimating Regions of Heterogeneity. Using this identification result, the authors goal is to search over regions of the feature space $S$ and groupings over agents $G$ to find the combination that maximizes this proposed measure of disagreement. To do so, the authors propose to solve  $$ \max_{S, G} Q(S,G) \mbox{ s.t. } P(X \in S) \geq \beta, $$ which requires that the region have probability at least $\beta$ (a tuning parameter to be specified). The authors develop an iterative algorithm to optimize this objective function, and provide some conditions under which they are able to derive a generalization bound on the solution return after the first iteration (Assumptions 2-4 and Theorem 2).  ## Empirical Applications The authors apply their proposed algorithm in two empirical applications.   First, the authors analyze the behavior of their proposed algorithm in a semi-synthetic experiment that is calibrated to data from Lin et al. (2020) on recidivism risk predictions made by MTurk workers. The authors simulate data from two types of agents that follow different decision rules for detention vs. release. The authors compare their algorithm against direct models that fit logistic regressions for the decision on agent identifiers and observable features (direct models) and TARNet. The authors report that their algorithm does better at recovering the "region AUC."  Second, the authors analyze an observational dataset on medical treatment decisions for diabetes. The data contain information on 3,956 patients and 458 group practices, and the authors search for the region of disagreement across these group practices.