Who’s Gaming the System? A Causally-Motivated Approach for Detecting Strategic Adaptation
Authors: Trenton Chang
Lindsay Warrenburg
Sae-Hwan Park
Ravi B Parikh
Maggie Makar
Jenna Wiens
Venue: NeurIPS 2024 poster (2024)
Keywords: causal inference
strategic classification
gaming
healthcare
Primary area: machine_learning_for_healthcare
TL;DR: We propose a causal effect estimation framework for detecting which agents are exploiting a payout model.

Abstract: In many settings, machine learning models may be used to inform decisions that impact individuals or entities who interact with the model. Such entities, or *agents,* may *game* model decisions by manipulating their inputs to the model to obtain better outcomes and maximize some utility. We consider a multi-agent setting where the goal is to identify the “worst offenders:” agents that are gaming most aggressively. However, identifying such agents is difficult without knowledge of their utility function. Thus, we introduce a framework in which each agent’s tendency to game is parameterized via a scalar. We show that this gaming parameter is only partially identifiable. By recasting the problem as a causal effect estimation problem where different agents represent different “treatments,” we prove that a ranking of all agents by their gaming parameters is identifiable. We present empirical results in a synthetic data study validating the usage of causal effect estimation for gaming detection and show in a case study of diagnosis coding behavior in the U.S. that our approach highlights features associated with gaming.

Who's Gaming the System? A Causally-Motivated Approach for Detecting Strategic Adaptation
Trenton Chang
Lindsay Warrenburg Sae-Hwan Park2
Ravi B. Parikh 2.3
Maggie Makar Jenna Wiens
University of Michigan University of Pennsylvania Emory University
{ctrenton, mmakar wi eis Musiiummice edu
(lindsay.wairl warrenburg sae- park eeeeeppeeiiiiinnneeeeeee upenn. edu ravi bharat parppaaiiiaaeepp ke@oory.y edu
Abstract
In many settings, machine learning models may be used to inform decisions that impact individuals or entities who interact with the model. Such entities, or agents, may game model decisions by manipulating their inputs to the model to obtain better outcomes and maximize some utility, We consider a multi-agent setting where the goal is identify the "worst offenders:" agents that are gaming most aggressively. However, identifying such agents is difficult without being able evaluate their utility function. Thus, we introduce framework featuring gaming deterrence parameter, a scalar that quantifies an agent's (un)willingnesss game. We show that this gaming parameter is only partially identifiable. By recasting the problem as a causal effect estimation problem where different agents represent different "treatments," " we prove that ranking of all agents by their gaming parameters is identifiable. We present empirical results in synthetic data study validating the usage of causal effect estimation for gaming detection and show in a case study of diagnosis coding behavior in the U.S. that our approach highlights features associated with gaming.
1 Introduction
Machine learning (ML. models often guide decisions that impact individuals entities. Attributes describing an individual or entity are often inputs to such models. In response, such entities may modify their attributes to obtain more desirable outcome. But changing one's attributes may be costly due to the difficulty of generating supporting evidence, or penalties for fraud. This behavior is called gaming strategic adaptation [1]. Strategic adaptation frames gaming as "utility maximization:" agents change their attributes to maximize payout, but incur a cost for modifying attributes.
As illustrative example, turn to the health insurance industry. In the United States (U.S.). contracted health insurance companies report their enrollees' diagnoses to the government, which calculates a payout based reported diagnoses via publicly available model [2]. The payout is intended to support care of the enrollee relation to the diagnosis. Companies may attempt to maximize payouts by reporting extraneous diagnoses, an illegal practice known "upcoding" [3]. Despite increasing awareness of upcoding [3, 4, 5, 6). upcoding costs U.S. taxpayers over $12B U.S. dollars annually [7], even with substantial investment audits ($100. 7M U.S. dollars, 2023 [8]) and payout changes to adjust for gaming [9, 10J. Since audits may not scale and often overlook fraud [11, 12), tools for flagging gaming-prone agents could help target audits. Beyond health insurance, gaming emerges in responses to credit-scoring algorithms [13, 14] and driver responses to rider allocation algorithms in ride-sharing apps [15].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Strategic adaptation with multiple agents
Agents behave if maximizing an unknown utility function Utility curves for Agent 1 and
Agents' observed decisions reflect
Agent decisions result in payouts
utility-maximizinn behavior
report diagnosis?
agent's & cost: Aprc-)
Input:
Output:
Paynut Cost. RC)-Ac()) -xc()))
$4M
Agent #
reward: Ri:
agent decisions
payout to agents
Agent
Enrollee 1
dj. 0.40
$2M
Enrollee 2
0 3 (dishonest)
S$
Eground mruth)
Enrollee 3
V
RV Rix
=
$1M
0.38
0,40
0.42 0.44
Agent 2
Enrollee 4
0 5 (dishonest)
Diagmosts rate
Enrollee 5
V
Agent (A, 30)
Payout model
Enrollee 6
Agent (A2 50)
(operated by decision-maker)
Figure 1: Left: Two agents with gaming deterrence parameters λ = 30 (purple) and X2 = 50 (blue) maximize utility (reward R cost e) with respect to diagnosis rate. Gaming costs increase in A and lower an agent's optimal diagnosis rate (stars). Center: Agents' observed decisions reflect uillit-maximizing behavior. Right: A decision-maker computes a payout based on agent decisions.
In this work, we study how one can identify agents with the highest propensity to strategically manipulate their inputs given a dataset of agents and their observed model inputs. A supervised approach is infeasible since fraud/gaming labels unavailable in our setting. A common paradigm for fraud/gaming detection is unsupervised anomaly detection. However, gamed attributes may not be outlier-like. The perspective of gaming as utilitymaximizing behavior in strategic classifica- tion provides an alternative to existing fraud/gaming detection methods. Many works in strategic classification (e.g., [1]) assume known utility functions and identical feature manipulation costs across agents, which assists in identifying an agent's "optimal" gaming behavior. However, such assumptions may not always apply. For example, in U.S. Medicare, due to the rarity of penalties for upcoding, it is unclear how to quantify the cost of fraud. Furthermore, due to the large number of companies contracted with U.S. Medicare, with distinct incentives (e.g., for-profit vs. non-profit groups) and disjoint populations of patients, there may be heterogeneity in feature manipulation costs.
To bridge this gap, propose novel framework for modeling agent utilities by introducing gaming deterrence parameter, which scales the perceived cost (to the agent) of gaming, First, we show that directly estimating the gaming parameter is not possible: the best we can do is lower bound on the gaming deterrence parameter. However, by re-casting gaming detection as a causal effect estimation problem, where each agent represents a "treatment," based on their gaming deterrence parameter is recoverable. Thus, we propose causally-motivated motivated ranking algorithm that produces a ranking of agents. Practically, agents most likely to game under our ranking could be flagged for further monitoring/auditin While our framework is inspired by health insurance fraud, applies more broadly to instances of gaming where multiple agents are gaming an ML-guided decision.
prove that a ranking of agents
We evaluate the performance of causal effect estimators for gaming detection in a synthetic dataset. Empirically, causal approaches rank the worst offenders higher than existing non-causal approaches that screen based on payouts/randomly, as well as anomaly detection methods. We then verify in real-world U.S. Medicare claims dataset that causal effect estimation yields rankings correlated with the prevalence of for-profit healthcare providers, a suspected driver of gaming [4, 16].
In summary: we 1) extend strategic classification to model differences in gaming behavior across agents (Section 3), 2) prove that point-identifying agents' gaming parameter is impossible without strong assumptions (Section 4), 3) show that, by recasting gaming detection as causal effect estimation, one recover ranking of agents based on their gaming deterrence parameters (Section 4), 4) demonstrate empirically that our framework identifies the worst offenders with fewer audits than baselines (Section 5), and 5) show in real-data case study that our approach yields rankings correlated with suspected drivers of gaming (Section 5). Code to replicate our experiments will be made publicly available at https: /:t::://ittttt //ggttttbbb.... com/MLD3// (ILL33/gaming. samingg_dtecctoo
2 Related works
Machine learning-based anomaly detection. Fraud detection is often framed as an unsupervised anomaly/outlier detection problem [17, 18, 19, 20, 21, 22, 23]. Such approaches assume gamed model inputs are outliers in some distribution. However, this assumption may be incorrect if gaming

---

is common in the data, or if gaming results in only small changes to observed agent attributes. In borrowing from strategic adaptation, we frame gaming as utility maximization, rather than making distributional assumptions about gamed agent attributes.
Strategic classification & gaming in machine learning. A large body of work in strategic classi- fication aims to design incentives to mitigate gaming/strategic behavior by agents. However, such works assume that feature manipulations costs are known/can be estimated across agents, or are identical [1, 24, 25, 26, 27, 28, 29, 30, 31]. In contrast, in our setting, feature manipulation costs are unknown and may differ across agents, but exhibit some shared structure that facilitates comparisons across agents. Shao et al. [32] assumes that agent gaming capacities may differ by placing bounds on manipulation, which may be unrealistic. Closest to our work is that of Dong et al. [33], which also assumes unknown and differing agent costs, but does not leverage similarities gaming across agents. Our work supplements the strategic classification literature by studying a related yet fundamentally distinct problem: rather than designing incentives mitigate strategic behavior, which is infeasible under unknown manipulation costs, we leverage differences in costs across agents to identify agents more likely to game.
3 Background & Problem Setup
We review strategic adaptation and extend it to model differences in gaming across agents.
What is strategic adaptation? Our work builds upon strategic classification [1]. Consider a pre- existing model f: D + IR that maps agent attributes E D to a payout. An agent may leverage its knowledge of f to change their attribute(s) d according some function A:
(d) arg max R(d: f) =gd,
dED
where R: D > IR is the payout of changing to d, and g: D - R4 is the cost of manipulating d. When D C TR. g is often assumed to be "separable;" Le., for some function c, g(d, d) = c(d d). (.)) describes how an agent manipulates d to obtain a higher payout from f. This behavior is called strategic adaptation or gaming. For simplicity, we assume = f: i.e., the model f directly determines the payout.
Modeling agent variation in gaming. To extend strategic adaptation to multiple agents, we add a non-negative gaming deterrence parameter Ap E E+ to Eq. 1. Consider an observational dataset D2 & {xx, "i of an agent p's decisions d, E {0, 1} given some information x; X.. For simplicity, we assume d, is binary, though the proposed framework generalizes to non- binary decisions and arbitrary numbers of independent decisions. For example, health insurance plan p chooses whether to report that an enrollee has a diagnosis d; given enrollee characteristics x;. Agent assignment is mutually exclusive (e.g., individuals are enrolled in one health insurance plan).
If the agent knows that d, will be used as input to a payout model, they may have an incentive to increase de (without loss of generality) to obtain higher payout. Let d = 1 My d,, and suppose each agent p chooses d according to the following utility-maximizatee problem:
M,
P(d, = p) = Ap A arg max R(d) - Apeld
(2)
de(o.1]
where R: [0.
c: RR R__ and dr is the ground truth value of d given the x; seen by agent Thus, Ap(d,) is the gamed/observed decision rate of agent p in a population where the ground truth decision rate is dr.. Although d [0, 1] since it is proportion, arbitrary intervals. This formulation assumes that each x; is equally likely to be gamed, that x; are truthfully observed, and that any difference between Ap(dt)) and d; is due to gaming.
framework applies to d on
We focus on the gaming deterrence parameter Xp. which scales the cost of manipulation c(-). Ap is non-negative and represents an agent's "aversion" to gaming. Lower values of Xp mean that agent p is more willing to game. Thus, identifying agents most willing to game means finding agents with the lowest Xp. We summarize multi-agent strategic adaptation in Figure! Next, we introduce assumptions on the reward function R,, cost function e, and ground truth d'y
3

---

Assumption 1 (Shared rewards & costs). Reward (R) and cost (c) functions are shared across agents.
Sharing encodes the assumption that all agents are reacting to the same payout model, while sharing across agents encodes the belief agents must take similarly-costly actions to manipulate features (e.g., if insurance plans must followfrraddlentll report specific procedures to justify a diagnosis). Many works in strategic classification implicitly make similar assumption (e.g., [1, 34J). Assumption 2 (Increasing rewards). The reward function R is strictly increasing in d.
Increasing rewards formalizes the agent's incentive to perturb its decisions (i.e., inputs to the payout model) from d; 0 to 1 in Eq. 2.
Assumption (Cost convexity). The cost function c is strictly corvex and minimized at 0 such that c(0) = 0 and c' (0) 0, and increases for all agents p for any d2 d; One possible is c(z) = x². Strict convexity ensures a unique cost-minimizing action, and c(0) = ensures that dy is ground truth, such that increasing d incurs greater cost (e.g., Fig. 1, left).
Assumption 4 (Diminishing or linear returns). The reward function R is concave in d₃.
For example, R may be a log or affine function. Assumption (strictly convex c) and 4 ensure that R cannot grow fast enough to offset manipulation costs. Furthermore, due to Assumptions 2- 4:
Remark (Gaming is utility-maximizze maximizing). Given any agent and di we have that Ap(d;) ž dr
Equivalently, optimal gaming entails increasing from the ground truth. Note that Assumptions 2- are more general versions of assumptions placed on rewards/costs in the strategic classification literature (e.g.. [1, 33, 25]).
Assumption 5 (Non-strategic behavior is feasible). dz E [0, 1] is constant depending solely on Due to Assumption 5, ground truth d may vary by agent due to differences in X; (e.g., health insurance plans serve populations with varying levels of health).
4 Theoretical analysis: finding agents most likely to game
We aim to identify agents most likely to game a decision-making model, i.e., agents with the lowest gaming parameters Ap- Here, we prove that A cannot be point-identified without further assumptions (Section 4.1), but ranking Ap is possible via causal effect estimation (Section 4.2). Detailed proof's in Appendix B.
4.1 Partial identification of the gaming parameter
Here, we show that given our assumptions, Ap is only partially identifiable (cannot be uniquely determined):
Proposition 1. Define (-(() as dR and c-) as de For any agent p. given Assumptions 1- 4 and an observed (d()))
dez
day.
R'(Ap(d)))
Ap E
(c(A,dd))) ,00
and the bound is sharp.
Intuitively, different values of the unknown d; yield different estimates of X consistent with the observed Ap(d)). Thus, uncertainty in de results in uncertainty in Ap. Equivalently, point-identifying Ap requires perfect knowledge of d; Thus, without further assumptions, Ao is only partially identifiable. The lower bound is attained for d; = 0 (all d, = 1 are manipulated), while Ap - co as Ap(d;) + de (no manipulation). Intuitively. increases in Ap further disincentivize increases to Ap (d)). such that (d())) gets closer to di.
A naïve approach to gaming detection would be to rank individuals using the above bound. To see why this problematic, consider an Agent (A1 10) and Agent 2 (A2 30), and let R(x) and c(x) = 2 Suppose Agent 1 is a health insurance plan serving relatively healthy population (di = 0.05), while Agent 2 serves population with a higher burden of illness (d) 0.12). Via Eq. 2, we have (dj)) = 0010, while A2 2 0.14. Substitution into Eq. 3 yields A 2 5 and
Aa 2 3.66, flipping the true ranking of >p. Thus, acting on this bound may incorrectly penalize agents when a high (d,) is appropriate; e.g., insurance plans serving sicker populations.

---

d
Pi
(p)))
d((p))
Patient features
True diagnosis
0.2
p
?
d"
-0.3
0
0
?
-0.1
p
?
Plan IDD
Reported diagnosis (binary)
0.4
0
p
?
0
(categorical)
Figure 2: Left: Toy dataset with observed factual outcomes (p) and d, (p'). "?" denotes missing counterfactual outcomes. Right: Causal graph for gaming detection with confounders x, agent indicator p. ground truth diagnosis d", and agent decision
Causally-motivated gaming detection
Q: Which agent gaming more aggresively:
Impute counterfactual decisions
Estimate pairwise treatment effects
Recover a gaming parameter ranking
Given 's what would do? report report diagnovis?
decisions
decisions
observations...
(abuerved))
(lupated)
Effect of
-5%
Enrollee
V
(mmre: rigenta
diagnosis?
Enrollee
X
Enrollee Enrollee 2
Enrollee
v
x
X
greater gaming tendency
lesser gaming tendency
Enrollee 3
v
x
Total
v 15% vate 10%
(repeat for
& other agent pairs)
Effect of
-5%
gaming more than
Figure 3: motiivaed gaming detection. Left: First, impute counterfactual decisions for each agent. Middle: The imputed counterfactuals yield average treatment effects (ATEs) across pairs of agents. Right:
Using ATE estimates to rank agents yields ranking of the gaming parameter Xp- We show one direction of comparison across agents for simplicity. In practice, we impute decisions for both directions of comparison and average (given blue agent's observations, impute purple agent's decisions).
4.2 Identifying ranking of the gaming parameter
Since we showed that point-identifying Ap is impossible without further assumptions, we relax gaming detection to a ranking problem. Intuitively, differences in agent behavior under similar conditions may indicate different gaming capacities, from which the proposed approach follows.
Ranking D by estimating counterfactuals. Recall that we aim to find agents with the lowest Ap- Thus, it suffices to rank agents by Ap, which can be done as follows:
Theorem 1. Under Assumptions 1- and A (dz) defined as
Ap((d) arg max R(d) Ap dny), dE(o.1)
(4)
we have that Ap(d)) < Ap if and only if Xp > Ax.
Theorem 1 tells us that estimation of (p()7) and Apr can be used to rank Xp vs. Ay Eq. differs from Eq. 2: while R, and d; are the same, Ap changes to Ap While subtle, the distinction is key to gaming detection: Ap (dz) denotes what agent p' would have done in a population with ground truth di as opposed to what agent p' actually did in their population (ground truth dry).
To build ranking strategy, first consider ranking agents p and p'. Define Dr.p'' A {(x;, de P:) i = 1...., n. and p; E {p.r')}, where p; is an indicator for the agent that observed example i. Following the Neyman-Rubin potential outcomes framework [35], let d,(p) be the value of d, was set to (i.e. had agent been compelled to make a decision). Such variables are called counterfactuals. Figure 2 (left) shows toy dataset with counterfactuals di(p), de(p'). where "q" unobserved decisions d,. Dropping unobserved data, the average d, (p) is (p(d)) by definition. Thus, if (p) and de (p') are fully observed, one could estimate Ap(d;) and Apr(d as follows:
(d,) = Tpp
Σ
dp(p)
Σ
di(p')
(5)
Thp
(x..d..;)ED p.y. p:=p
(x).).D))ED pi=p
pay!
5

---

a) Causally-motivated gaming detection.
b) Python-style pseudocode.
Input: D ((x, d, pt) and list of
train_ data, test_data aplit(dataset) model fit_causal__eee eettusaaestttmtrr///eeeo (ttiimatorrtraaindaaaaa. comparisons = O
agents
Output: sorted list of P agents + fitted causal effect estimator using
for p.i, P-j in agents where p.i p-J:
# counterfactual eat., plugging in agent IDs agent_i._f model podel.predict prepddicccttst_daae (atst p.i) agent_j.cf model-predict (test_data, P-j) comparisons((p.1, P-j)] agent_i_cc. agent.j.cf
T empty P-by-P matrix
for all Pi where i < j do
T Pz)
end for
ranking sort comparisons) use comparisons dictionary as lookup table return ranking
A < sort p, using T to make comparisons return A
Figure 4: Pseudocode for causally- gaming detection. Causal effect estimators take pairs of agentss (p,p') and their observations xf+) as inputs, and output the mean difference in predicted decision rate.
where Rp is the number of observations by agent p. and use these estimates to rank Ap as per Theo- rem 1. However, only one of (p) d((p') is ever observed. The need to estimate counterfactual (namely, Ap (dr))) suggests a causal inference approach, which proceeds assuming the following [36]:
Assumption 6 (Conditional exchangeability). For all i, (p(ps) 1L ps | potential outcome of d, under agent ps-
xx where d, (p:p) is the
Assumption 7 (Consistency). For all i, d;(p:) d,.
Assumption (Positivity/overlap)) For any agent and x E E 0 < Pip x] <
Via Assumptions 6- 8, have that E(d,(p) x Eld, x, pl and Eld,(p') x,J Eld, x;.p') [37]. Hence, the causal effect is identifiable. This estimand corresponds to a varielvariable causal graph, where x; confounders, the agent indicator p; "treatment," and the agent's decision d, is the outcome (Fig. 2, right).' We proceed by estimating the effect of "swapping" agents:
Corollary 1. Define T (p. p¹) & Ex, [B(d.() xx]] - Ex, [[d((')) xx]]. Then, given Assumptions 1- (p,p')) if and only if Ap < Ap.
Since E[d<(p) x, = Eld | p). it is an unbiased estimator of (d())) by definition, and substitution into Theorem yields the result. Thus, one rank Xp by estimating the effect of switching from agent p to p' on the observed decision rate for all pairs of agents p. p'. Each effect estimate a pairwise comparison of agents, from which a ranking of Ac.) follows. We summarize maosally-motvvated gaming detection in Fig. 3, with pseudocode in Fig. 4.
Obtaining well-ordered ranking. Since we aim to rank Ap, our chosen estimator should yield well-ordered ranking. Via Corollary the oracle treatment effect suffices, but T must generally be estimated. We show that "sufficiently" accurate estimate & also yields the desired result:
Proposition 2. Let T(.) be the oracle treatment effect function as defined in Corollary and & be its sample estimate. Given Assumptions 1- 8, for any E >> O, if sup If(p, p') (p, p')l I s, then, for all p.p' such that min (T(p > E, f(p.p') > 0 if and only if Xp < Xxp-
P.P
The result is immediate: sufficiently low estimation error in 7 (i.e., S 5) cannot "flip" any pairwise rankings where 7 > E. Thus, any consistent estimate of yields (asymptotically) a well-ordering of Ap- We defer to past asymptotic analyses of causal effect estimation for further discussion [38, 39].
5 Empirical results & discussion
We aim to demonstrate that causal inference can be used for gaming detection. First, discuss our setup (Section 5.1). Then, we show in synthetic data (Section 5.2) that causal methods require fewer audits than existing non-causal methods to catch the worst offenders. Finally, real-world case study (Section 5.3), we find that causal methods yield rankings correlated with suspected drivers of gaming.
Note that, under the causal graph of Figure (right), the non- non-ideniifiabiiity of Xp is immediate from the d- d-separation properties of the graph; Appendix B for further comment.

---

5.1 Setup
We describe the datasets, evaluation method, and gaming detection methods under consideration.
Datasets. In real datasets, ground truth gaming rankings are often unavailable. Thus, we validate our framework in a synthetic dataset. We hand-select 20 Xp values (one per agent; see Appendix C.I for raw Xp values) and simulate confounding by generating covariates from agent-specific Gaussians with means Htp E IR?2. To control confounding strength, we set Hy = g(log(>p) where is an affine transformation such that the range of μ4 across agents equals chosen range parameter Rp E (0, (0.0.1.... 1.0). Smaller Ry implies less confounding, since (l()) varies less across agents.
We generate 500 observations x()) E RR² and ground-truth (=(()
per agent via
(Hpto , o² 22x2)
(1) Ber(a (1) oo Tx"xxx x(() + b),
where (0, 1)2, such that increasing increases (+) (and thus P(d 1)), b is chosen such that (1) for the mean x is v 5%, and o² 1. We simulate gamed agent decisions d()) as follows:
a ~ (
(1) p = arg max log(dz) - Ap - d²()))
2_E(0.1]
Recall that the decisions ( are also agent inputs to a payout model. We generate 10 datasets (cach N 10, 000; 20 agents 500 observations) for all 11 levels of confounding (as measured by the range of means Ry). Causal inference assumptions hold in the synthetic data: all confounders x() are observed (Assump. 6), consistency holds by construction (Assump. 7), and overlap holds since all (2) p are supported on IR2 (Assump. 8). Full synthetic data generation details are in Appendix C.I.
To benchmark causal effect estimation in a more realistic setting, we apply causal methods to gaming detection in U.S. Medicare claims. Medicare is the public health insurance system in the U.S. for residents aged 65 and over. Since private insurance claims data is not widely available, we conduct gaming case study in healthcare providers. In Medicare, the U.S. government pays healthcare providers on a per-service basis [40]. Thus, providers may be incentivized to label enrollees with as many diagnoses as possible to secure extra payment from the government. We select 0.2% sample of all Medicare enrollees with a claim in 2018; i.e., those who utilized a service covered directly by Medicare (N 37, 893). We use demographic information and diagnoses in 2018 as covariates and select the rate of uncomplicated diabetes diagnosis in 2019 as the outcome. Given differences in healthcare policy and access across U.S. states, we pool data the U.S. state level and treat each state as an "agent." Additional cohort details are in Appendix C.2.
Evaluating rankings. Given an observational dataset of the form {(xi, di, p:) with covariates x,, observed decisions di, and agent indicators gaming detection algorithms output an ordinal agent ranking in terms of the gaming parameter Ap- We aim to measure the efficiency of a predicted ranking given some level of resources committed by a decision docision-maker maker (i.e., # of agents audited).
Ground truth rankings are available in synthetic data. Thus, measure the top-5 sensitivity at & (Sy). the % of top-5 worst offenders in the predicted top-k, and the discounted cumulative gain (DCG) at k, weighted sum of ground-truth "relevance scores" for the top-k predicted agents, across audit intensities & E (I..... 20}. For K-agent dataset, we define the relevance score as + minus the ground-truth rank (e.g., true rank = relevance K, true rank 2 relevance K etc.). Concretely, for r; defined as the ith ranked agent in a predicted ranking, and rank(-) as the function returning the ground-truth ordinal rank with respect to Ap. our ranking evaluation metrics are computed follows:
K - rrnk(rr):) 1=1 logo +
S₁ A
(rank(rs) < 5]
DCG
Note that sensitivity is an all-or-nothing measure of audit quality given fixed audit intensity k. However, DCG rewards higher predicted rankings for top-k worst offenders, regardless of the absolute ranking position. Furthermore, DCG weights decrease with predicted rank (Eq. 6), which prioritizes
For case of implementation, (4) is generated conditioned on p(), This remains consistent with the causal DAG (Fig. 2, right), since the DAG factorizes as P(D P.P) | X))(X), or equivalently, P(D) P(X P)P(). as in our data-generation process.
7

---

Ranking performance, causal vs. non-causal approaches
Top-5 sensitivity
Disc. cumulative gain (DCG, t)
Top-5 sens. ( ()), 7 audits
1.0
sans.
Top-5 sens.
0.5
DOG
0.5
0.0
10
15
10
15
Random
Payout-oaly
EcoD
DIF
DragonNet R R-learnor S+IPW
KNN
# of agents audited
of agents audited
Random
ECOD
R-learner -X- S+IPW
Payout-only
DIF
NeagonNet
Figure 5: Mean top-5 sensitivity (left) and DCG (center) across # of agents audited, and top-5 sensitivity with 7 audits (right) at mean range 0.9, with to error. Causal methods improve over non-causal baselines. naive baselines. or anomaly detectors. x: causal effect estimators.
correctly ranking the worst offenders over correctly ranking agents unlikely to game. To summarize audit efficiency across k, we also report area under the top-5 sensitivity curve (AUSC) across k.
In contrast, ground truth gaming rankings are unavailable in the Medicare cohort. As an exploratory analysis, we compare an estimated gaming ranking to 104 state-level healthcare statistics from the 2003-2017 National Neighborhood Data Archive [41] and 2018 Medicare Provider of Service files [42] relating healthcare access and hospital information (e.g., ownership and size). We report the top five statistics most positively and negatively correlated with our predicted rankings in terms of Spearman rank-correlation. State-level summary statistics used are enumerated in Appendix C.2.
Models. To demonstrate the utility of causal effect estimation for gaming detection, we compare non- approaches include payout-only ranking
causal baselines causal effect estimators. (based on P(d; 1) per agent) and random ranking. We also compare to existing approaches in anomaly detection, which do not make causal assumptions, but assume that gamed decisions are lute We test k-nearest neighbor outlier detection (KNN) [17], empirical-cumulann distribution-basee outlier detection (ECOD) [22], and deep isolation forests (DIF) [23]. These methods use (x, di) as inputs to an "anomaly score" model. We use average within-agent anomaly scores as ranking. We discuss other works in algorithmic anomaly/fraud detection in Appendix A.
We implement the following causal effect estimators. PSM fits a propensity score model to match points in one agent's 's population to its nearest neighbor in the other agent's population with respect to propensity score estimates [43]. The S-learner trains one outcome prediction model for all agents, while the T-learner trains model per agent [44]. DragonNet jointly models the outcome (one prediction "head" per agent) and propensity score to control for confounding [45]. S+IPW fits the S-learner with estimated sample weights that reweight the observed distribution to resemble an unconfounded distribution (randomized treatment assignment) [43]. The R-learner fits an outcome and propensity model, then regresses the residuals on one another obtain final unconfounded estimator [39]. Hyperparameters for all baselines and causal effect estimators are in Appendix E.
Implementation details. We use neural networks for all modeling (causal effect estimators + DIF). We use one-hot encoding for treatments (agent indicators). Matching across pairs of agents occurred without replacement (one-to-one), dropping unmatched individuals. We use generalizations of IPW and R-learners to multiple treatments, namely permutation weighting [46] and the structured intervention network [47], respectively. We perform 7:3 dataset train-test split, training all models on the larger split. All rankings are computed on the test split. Early stopping is performed on 20% validation split randomly sampled from the training set. Full modeling and training details, including the architecture and hyperparameters used, are in Appendix E.
5.2 Gaming detection in synthetic data
Causal effect estimators identify gaming more efficiently than non-causal baselines. Figure 5 shows the top-5 sensitivity and DCG of rankings produced by causal non-causal gaming detection approaches at high confounding (mean range: 0.9). Since the S-, T-learner, and DragonNet perform
"AUSC is similar to the area under the ROC curve (AUROC) with the top-k worst offenders defined as the "positive class." except that the r-axis is the # of audits rather than false positive rate. Unlike AUROC, random performance is less than 0.5, but tends to 0.5 as the number of agents K - oo (Appendix B.6).
8

---

Area under the sensitivity curve (AUSC, t vs. confounding strength
0.9
0.3
0.0
0.2
0.4
0.6
D.B
1.0
Confounding level (range of means across agent-spectic covariate distributions)
Random
Payaut-only
ECOD
i DIP i KNN B DragenNet
& Relearner
S+IPW
Figure 6: Area under the sensitivity curve (AUSC) for causal vs. non-causal methods across levels of confounding, with to error. As confounding increases, a payout-only ranking degrades. Anomaly detection performance does not vary across confounding strength, maintaining slightly better than random rankings. Causal methods generally maintain higher mean AUSC than baselines across confounding levels. V: naïve baselines. o: anomaly detectors. x: causal effect estimators.
similarly (Appendix D.2), we show only DragonNet here. Since PSM underperforms due to challenges with
confounding control, we also defer results for PSM to Appendix D.2. Across audit intensities, causal approaches outperform non-causal methods in terms of sensitivity (e.g.. at 7 audits, Fig. 5, right; S+IPW: 0.860++..15 135 KNN: 0.520=002115) and DCG (S+IPW: 56.1=5.11 vs. KNN: 42.3+ 10.8).* Trends are similar at other levels of confounding (Appendix D.I), though the gain in ranking performance of causal approaches over non-causal methods diminish at lower levels of confounding (Figure 7; S+IPW AUSC: 0.61410.096 vs. KNN: 0.648 129, mean range 0.0).
A payout-only approach yields worse than random ranking with sufficient confounding between covariates and agent decisions (payout-only AUSC: 0.299+10.067 vs. random: 0.473:10. 101, Figure 6, mean range 1.0; e.g., if healthier patients are enrolled in more gaming-prome plans). Anomaly detection methods (KNN, ECOD, DIF) are ill-suited for de- tecting gaming in dense regions of covariate space by design, while causal approaches would excel due to improved over- lap. If outliers are more likely be manipulated (e.g., if populations with lower ground-truth d are more likely to be gamed), an anomaly detection method would identify gam- ing in such points. Indeed, anomaly detectors empirically outperform random ranking but lag causal methods. We further discuss why anomaly detection methods underperform causal approaches in Appendix D.I.
S+IPW KNN, AUSC (1)
AUSC
0.8
0.6
0.2
0.0
Confounding strongth (mean range)
S+1PW
KNN
Figure 7: AUSC of S+IPW (causal) vs. KNN (non-causal) across confounding strength with to error. The advantage of S+IPW over KNN decreases as con- founding diminishes.
Trends in ranking performance vary across causal effect estimators. DragonNet and the R-leamer degrade slightly as confounding increases (DragonNet AUSC: 0.755+10.99 009 - 0.696+0.083 R- learner: 0.728=0.114 + 0.635=0.111. mean range 0.0 + 1.0), likely due to slightly worse overlap. However, S+IPW improves as confounding increases (AUSC: 0.694 - 0.837:10.067 mean range 0.0 1.0). This is likely since the oracle IPW weights deviate from a uniform weighting as confounding increases. In such settings, estimation error in IPW weights may be less likely to incorrectly up-weight points that an oracle propensity score would down-weight, and vice versa. While such eтгоr would bias the pointwise treatment effect estimate, for the purposes of ranking, causal effect estimators only need to correctly estimate the sign of the treatment effect. Thus, we hypothesize that our ranking may be less affected by such errors in the propensity score estimate. We leave formal analyses of the properties of IPW with respect the sign of causal effect estimates to future work. A sensitivity analysis of all causal approaches is in Appendix D.2.
Takeaways. In synthetic dataset, causal effect estimation approaches identified gaming more efficiently than non-causal baselines across levels of confounding. The empirical results provide proof-of-concept for causal effect estimation as a gaming detection method.
5.3
Case study: Detecting upcoding in U.S. Medicare
As exploratory analysis, we use the best-performing approach in the synthetic data (S+IPW) analyze upcoding by U.S. state in U.S. Medicare. Table shows the five state-level healthcare statistics most positively and negatively correlated with gaming rankings predicted by S+IPW.
We report to error bars in the figures.
9

---

Table 1: Top 5 features most positively and negatively correlated with state gaming rankings predicted by S+IPW, as measured by Spearman correlation. We report p-values given a null hypothesis of zero correlation between predicted rankings and the statistic of interest. OP PT/SP: Outpatient physical therapy and speech pathology. SNF: skilled nursing facility.
State-level healtheare system statistic
Spearman corr.
p-value
% of OP PT/SP providers that are for-profit ratio of for- for-profit to non con-profit hospitals % of hospice providers that are for-profit % providers that are ambulatory surgery centers % of hospitals that are for-profit
0.294
0.036
0.272
0.053
0.232
0.101
0.222
0.118
0.222
0.118
Upcoding is correlated with for-profit provider prevalence. Four of the top five features most positively associated with our predicted ranking reflect a greater state-level prevalence of for-profit healthcare providers. This matches the intuition that for-profit providers may game more aggressively due to stronger profit motives. Notably, the feature 2nd-most positively correlated with our rankings (ratio of for-profit non-profit hospitals) is a suspected driver of upcoding in Medicare [4, 16], with the idea that competition from for-profi profit providers drives non-profit providers towards gaming. Note that many of the correlations are not statistically significant, and unmeasured factors such as healthcare quality could explain differences in diagnosis coding, rather than gaming. Despite the limitations, causal effect estimation shows promise as a practical approach to gaming detection.
Takeaways. In an exploratory case study of gaming in U.S. Medicare, causal effect estimation yields a ranking of U.S. states that positively correlates with the prevalence of for-profit healthcare providers, matching domain expertise suspected drivers of gaming.
6 Conclusion
We propose a mousally-motivated framework for ranking agents by gaming propensity in the context of strategic adaptation. We show that the gaming parameter is only partially identifiable, but ranking of set of agents based on the gaming deterrence parameter is identifiable via causal inference. We demonstrate the utility of causal effect estimation for gaming detection on synthetic data and a case study of upcoding in Medicare.
Limitations& & broader impact. We assume agents always increase d; with respect to ground truth, and that gaming explains all differences in agent behaviors, ignoring factors such as agent "quality" (e.g., quality of care). assumptions, but statistically unverifiable. Many works in game theory and causal inference share these limitations. We caution that policies informed by our framework could reinforce imbalanced power dynamics (i.e., are individual citizens [26] or more powerful entities gaming model) via extraneous or weaponized accusations of gaming, since not all entities have equal capacity to respond to such claims. In particular, gaming by individuals may potentially reflect structural inequities rather than inherently pathological behavior. To mitigate such risks, we suggest "shadowing" studies (decisions visible, but not acted upon) alongside existing audit mechanisms before adoption.
behavior and conditional exchangeability are strong
Acknowledgements
We thank (in alphabetical order) Amanda Kowalski. Daniel Shenfield. Donna Tjandra, Divya Shan- mugan, Dylan Zapzalka, Ezekiel Emanuel, Jung Min Lee, Meera Krishnamoorthy, Sarah Jabbour, and Serafina Kamp for helpful conversations and feedback. Special thanks to Dexiong Chen, Michael Ito, Shengpu Tang, Stephanie Shepard, and Winston Chen for their comments on drafts of this work, to Kathryn Ashbaugh, Michael Shafir, and the Advanced Research Computing team at the University of Michigan for assistance with data access and usage, and Matt Guido for coordinating our meetings. The authors are supported by a grant from Schmidt Futures (Award No. 70960). The funders had no role in the study design, analysis of results, decision to publish, or preparation of the manuscript. This study was deemed exempt and not regulated by the University of Michigan institutional review board (IRBMED; HUM00230364).
10

---

References
[1] Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters. Strategic classi- fication. In Proceedings of the 2016 ACM conference innovations in theoretical computer science, pages 111-122, 2016.
[2] Gregory C Pope, John Kautter, Randall P Ellis, Arlene Ash, John Z. Ayanian, Lisa lezzoni, Melvin J Ingher, Jesse M Levy, and John Robst. Risk adjustment of medicare capitation payments using the ems-hee model. Health care financing review, 25(4):119, 2004.
[3] Michael Geruso and Timothy Layton. Upcoding: evidence from medicare on squishy risk adjustment. Journal of Political Economy, 128(3):984-0026 2020.
[4] Elaine Silverman and Jonathan Skinner. Medicare upcoding and hospital ownership. Journal of health economics, 23(2):369-33) 389, 2004.
[5] Faiz Gani, Joseph Canner, and Timothy M Pawlik. Assessing coding practices for gastroin- testinal surgery over time in the united states. Surgery, 164(3):530-55 2018.
[6] Kacie L Dragan, Sunita M Desai, John Billings, and Sherry A Glied. Association of insur- ance mix and diagnostic coding practices in new york state hospitals. JAMA Health Forum, 39))x222919 e22299992222.....199 2022.
[7] Michael E. Chernew. Report to the Congress: Medicare Payment Policy, 2022.
[8] Centers for Medicare and Medicaid Services. Fy2024 justifitttno of estimates for appropriations committees, 2024.
[9] Richard Kronick and W Pete Welch. Measuring coding intensity in the medicare advantage program. Medicare & Medicaid Research Review, 4(2), 2014.
[10] Center for Medicare Medicaid Center. Note to: Medicare advantage organizations, prescrip- tion drug plan sponsors, and other interested parties, 2023.
[11] Casie C Rodenberger. Pre-submission risk adjustment audits: Preventing medicare advantage plans from draining medicare funds dry. lowa I.. Rev., 103:841, 2017.
[12] Maggie Shi. Monitoring for waste: Evidence from medicare audits. The Quarterly Journal of Economics, (39(2):993-1044 2024.
[13] Danielle Keats Citron and Frank Pasquale. The scored society: Due process for automated predictions. Wash. L. Rev. 89:1, 2014.
[14] Jane Bambauer and Tal Zarsky. The algorithm game. Notre Dame L. Rev., 94:1, 2018.
[15] Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. Working with machines: The impact algorithmic and data-driven management on human workers. In Proceedings of the 33rd annual ACM conference on human factors in computing systems, pages 1603-1612, 2015.
[16] Elaine Silverman and Jonathan S Skinner. Are for-profit hospitals really different? medicare upcoding and market structure, 2001.
[17] Sridhar Ramaswamy, Rajeev Rastogi, and Kyuseok Shim. Efficient algorithms for mining outliers from large data sets. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 427-433 2000.
[18] Daniel De Roux, Boris Perez, Andrés Moreno, Maria del Pilar Villamil, and César Figueroa. Tax fraud detection for under-reporting declarations using unsupervised machine learning approach. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 215 222, 2018.
[19] Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008.
[20] Sahand Hariri, Matias Carrasco Kind, and Robert J Brunner. Extended isolation forest. IFEE transactions knowledge and data engineering, 33(4):1479-148) 2019.

---

[21] Chamal Gomes, Zhuo Jin, and Hailiang Yang. Insurance fraud detection with unsupervised deep learning. Journal of Risk and Insurance, 88(3):59-6624 2021.
[22] Zheng Li, Yue Zhao, Xiyang Hu, Nicola Botta, Cezar lonescu, and George H Chen. Ecod: Unsu- pervised outlier detection using empirical cumulative distribution functions. IEEE Transactions on Knowledge and Data Engineering. 35(12):12111-1219) 2022.
[23] Hongzuo Xu, Guansong Pang, Yijie Wang, and Yongjun Wang. Deep isolation forest for anomaly detection. IEEE Transactions on Knowledge and Data Engineering, 2023.
[24] Yiling Chen, Yang Liu, and Chara Podimata. Learning strategy-aware linear classifiers. Ad- vances in Neural Information Processing Systems, 33:15265- 15276, 2020.
[25] Sagi Levanon and Nir Rosenfeld. Strategic classification made practical. International Conference on Machine Learning, pages 6243 6253. PMLR, 2021.
[26] Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. The social cost of strategic classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 230-239,, 2019.
[27] Hanrui Zhang and Vincent Conitzer. Incentive-aware pac learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(6):5797-5844, 2021.
[28] Tosca Lechner and Ruth Urner. Learning losses for strategic classification. Proceedings the AAAI Conference on Artificial Intelligence, 36(7):7337-7344, 2022.
[29] Tosca Lechner, Ruth Urner, and Shai Ben-David. Strategic classification with unknown user manipulations. In International Conference on Machine Learning, pages 18714-18732. PMLR, 2023.
[30] Daniel Björkegren, Joshua E Blumenstock, and Samsun Knight. Manipulation-proof machine learning. arXiv preprint arXiv:2004.03865 2020.
[31] Ravi Sundaram, Anil Vullikanti, Haifeng Xu, and Fan Yao. Pac-learning for strategic classifica- tion. Journal of Machine Learning Research, 24(192):1-38, 2023.
[32] Han Shao, Avrim Blum, and Omar Montasser. Strategic classification under unknown personal- ized manipulation. Advances in Neural Information Processing Systems, 36, 2024.
[33] Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 55-70, 2018.
[34] Yahav Bechavod, Katrina Ligett, Steven Wu, and Juba Ziani. Gaming helps! learning from strategic interactions in natural dynamics. In International Conference on Artificial Intelligence and Statistics, pages 1234-1242. PMLR, 2021.
[35] Paul W Holland. Statistics and causal inference. Journal of the American statistical Association. 81(396):945-966 1986.
[36] Judea Pearl. Causality. Cambridge university press, 2009.
[37] Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015.
[38] Alicia Curth and Mihaela Van der Schaar. Nonparametric estimation of heterogeneous treat- ment effects: From theory to learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1810-1818. PMLR, 2021.
[39] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):999-31, 2021.
[40] Your Medicare coverage choices - medicare.gov. https://www/ medi care. ware.gov/ gov/ [Accessed 25-04-2024).
12

---

[41] Anam Khan. Mao Li, Jessica Finlay, Michael Esposito, Iris Gomez-Lopez, Philippa Clarke, and Megan Chenoweth. National neighborhood data archive (nanda): Health care services by census tractstates, 2003-2017 [data set]. Inter-University Consortium for Political and Social Research (ICPSR), 10:E120907V2, 2020.
[42] Provider of services files, Sep 2022.
[43] Paul Rosenbaum and Donald B Rubin. The central role of the propensity score observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[44] Sören R Künzel, Jasjeet S Sckhon, Peter Bickel, and Bin Yu. Metalcarners for estimating heterogeneous treatment effeets using machine learning. Proceedings of the national academy of sciences, 1 166100)4456-4165,, 2019.
[45] Claudia Shi, David Blei, and Victor Veitch. Adapting neural networks for the estimation of treatment effects. Advances in neural information processing systems, 32, 2019.
[46] David Arbour, Drew Dimmery, and Arjun Sondhi. Permutation weighting. In International Conference on Machine Learning, pages 331-341. PMLR, 2021.
[47] Jean Kaddour, Yuchen Zhu, Qi Liu, Matt Kusner, and Ricardo Silva. Causal effect inference for structured treatments. Advances in Neural Information Processing Systems, 34:24841-24854, 2021.
[48] Richard A Bauder and Taghi M Khoshgoftaar. Medicare fraud detection using machine learning methods. In 2017 16th IEEE international conference on machine learning and applications (ICMLA), pages 858 865. IEEE, 2017.
[49] John T Hancock, Huanjing Wang, Taghi M Khoshgoftaar, and Qianxin Liang. Data reduction techniques for highly imbalanced medicare big data. Journal of Big Data, 11(1):8, 2024.
[50] Stijn Viaene, Richard Derrig, Bart Baesens, and Guido Dedene. comparison of state-of- the-art classification techniques for expert automobile insurance claim fraud detection. Journal of Risk and Insurance, 693333373-42, 2002.
[51] Tahir Ekin, R Muzaffer Musal, and Lawrence V Fulton. Overpayment models for medical audits: multiple scenarios. Journal of Applied Statistics, 42(1 42(1 12(11):331-2405 2015.
[52] Sharon Tennyson and Pau Salsas-Forn. Claims auditing in automobile insurance: fraud detection and deterrence objectives. Journal of Risk and Insurance, 69(3):289- 308, 2002.
[53] Georges Dionne, Florence Giuliano, and Pierre Picard. Optimal auditing with scoring: Theory and application to insurance fraud. Management Science, 55(1):58-70 -0888----000 2009.
[54] Christopher Weaver, Tom McGinty, Anna Wilde Matthews, and Mark Maremont. Insurers pocketed $50 billion from medicare for diseases no doctor treated. Wall Street Journal, 2024.
[55] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
[56] Yue Zhao, Zain Nasrullah, and Zheng Li. Pyod: A python toolbox for scalable outlier detection. Journal of Machine Learning Research, 20(96):1-7, 2019.
[57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach De Vito,, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
[58] Marian Tietz, Thomas Fan, Daniel Nouri, Benjamin Bossan, and skorch Developers. skorch:
A scikit-learn compatible neural network library that wraps PyTorch, July 2017.
13

---

[59] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duchesnay. Scikit-learn: Machine learning in Python, Journal of Machine Learning Research, 12:2825-2830, 2011.
[60]
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, lan Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-271 2020.
[61] Steven Diamond and Stephen Boyd. CVXPY: A convex optimization. Journal of Machine Learning Research, 17(83)11-5, 2016. modeling language for
[62] Suresh Bolusani, Mathieu Besançon, Ksenia Bestuzheva, Antonia Chmiela, João Dionísio, Tim Donkiewicz, Jasper Doommalen, Leon Eifler, Mohammed Ghannam, Ambros Gleixner, Christoph Graczyk, Katrin Halbig, Ivo Hedtke, Alexander Hoen, Christopher Hojny, Rolf van der Hulst, Dominik Kamp, Thorsten Koch, Kevin Kofler, Jurgen Lentz, Julian Manns, Gioni Mexi, Erik Mühmer, Marc E. Pfetsch, Franziska Schlösser, Felipe Serrano, Yuji Shinano, Mark Turner, Stefan Vigerske, Dieter Weninger, and Lixing Xu. The SCIP Optimization Suite 9.0. Technical report, Optimization Online, February 2024.
[63] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fer- nández del Río, Mark Wiebe, Pearu Peterson, Pierre Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, September 2020.
Kevin Sheppard, Tyler
[64] The pandas development team. pandas-dev/pandas: Pandas, 2020.
A Additional related works
Algorithmic anomaly/fraud detection. Our framework can be understood as an algorithmic audit- ing method for fraud/anomaly detection. Many approaches assume that ground-truth fraud/gaming labels are available, reducing gaming detection to supervised learning [48, 49, 50]). We do not assume access to such labels. Unsupervised approaches for anomaly/fraud detection [18, 19, 20, 21] generally assume that anomalies are outliers with respect to some distribution. Mixture-modeline modeling approaches similarly assume that learnable under restrictive parametric assumptions [51]] Instead of distributional assumptions, we make behavioral assumptions about agents following strategic classification. Existing models of agent behavior in the context of fraud detection make domain-specific assumptions about features predictive of fraud [52], agent utility (e.g., constant penalties [53]), access to audit labels [52]. We generalize past work by making looser assumptions on agent utility and does not assume access to auxiliary information (e.g. audit labels), and circumvents the need for ground-trutt labels by making assumptions about gaming.
decisions correspond to distributions
B
Omitted Proofs
Here, provide detailed proof's for all theoretical results.
14

---

B.1 Proposition
Proposition. Let R' - d ddz
and d - d_
For any agent Ps given Assumptions J- 4 and fixed adz"
observation of pp(pp)
(d)))))))) d((pp(;))) )00
Ap E
Proof. Fix some R, satisfying assumptions Assumptions 1- and an arbitrary Ap(d)) Note that, for all Ap(d)) + di we can write
(d)))())) A d(pp(d)
(8)
Note that this quantity is monotonic in d', since the strict convexity of e implies that c' is strictly increasing. Hence, we can substitute an upper and lower bound d' to obtain desired result. First, since d' E [0, (d()))),, we can substitute d' 0 to reach
(())) Ap (d))
Before considering the case where d' = Ap(d)),, we note that direct substitution yields d' c'(Apdd)) - Ap(d))) = c(()) 0, since e is strictly convex and minimized at 0 via Assump- tion However, we can use limiting argument since e differentiable and therefore continuous:
(d())(d)))
lim Ap d' -Ap(()))
lim d'd)) d(Ap(d))) d')
(10)
To evaluate the limit, we treat R' as positive constant (via Assumption 2) and note that the limit
lim d' (d)))))) (()p(()))))))) dd)
(11)
from which conclude d' Ap1())
lim
Combining with the lower bound Xp yields the
desired bounds.
causal DaG-based perspective on non-identifiability of Ap- In the context the causal graph of strategic adaptation (Figure 2. right), the -on-iceniifiabiiity of Xp is immediate. Since it is proxy for the treatment effect plan (p) on reported diagnosis rate (d), without accounting for and d², conditional exchangeability (Assumption 6) does hold. The utiltt-maximizzatio formulation of gaming borrowed from strategic classification provides additional information not encoded in the causal graph. Namely, have that R' (d,A,,)))) (A, d;,) is monotonie in dz and d; E [0, Ap (dd)), provided that R and satisfy Assumptions These observations allow us to further bound the range of Ap-
B.2 Theorem 1
Theorem. Define Ag (d7))
(p(( A arg max R(d) Ap dz) de(o,1]
(12)
Then, given Assumptions 4, Ap(d)) < By (d)) if and only if Xo > Apr
Proof. For all portions of the proof, let R' A dd= and c ddc
15

---

d²) d')= FI - 
We have the simultaneous first-order conditions
(AA( - Xpd ( -
-
- Ap (dt) - -d)
(13)
- 14' (dx)) - -Ape' (A,((t))))))))))))))))))))))) d3) = ->yd - d¹)
(14)
C020
Ap(()) dd) - C' = Aprd(Ap (dh) - dd)
(15)
d' A (d7) - +d'
Ap >pp dl (d)))))) - dd²)
Xp'
(16)
C151
and hence Xp > A as desired. Note that Co 2 0 as per Assumption 2, and C > 1 as per our
assumption that Ap Ap(d)) < Ap from which we can conclude c' c'(A (dt) d²) c (Ap(d)))
d') ), because the striet convexity of e implies that e' strictly increases (Assumption 3).
( (=) ) Pick any Apr < Xp and fix some d'. By contradiction; suppose Ap 2 Ap (dz) as well.
Then:
d (()))(d))) -d) R' - Ap (4, (d d')
(17)
- R' (d))) R'A
= (x - d²) - Apr (A,/ (dr)) d²). (18)
dd)
Now, note that A' (dp)) - R' (dz)) S O. since Ap(dh) 2 Ay (dht) by assumption and R! is non-increasing in its argument. Furthermore, we can write
- d=) (Ap (d)) d=) > (d
d') -
> 0, (19)
where the first inequality is due to Ap Xp and factoring, and the second inequality is since c'(A (A)((d)) d²) d( (Ap (db)) d²) >> 0 (via Assumption 3) and X()) > 0 (by definition). Returning
to Eq. 18, write
0 R' (d))))))) (d))) = (A, d') 0
(20)
which is contradiction (0 7 0). Thus, A < Xp implies 4d(()) Ap (dr)) desired.
Potential extensions to upcoding detection. With stronger assumptions, a weaker form of upcoding detection, which is stronger than ranking, is possible:
Assumption 9 (Known cost and reward derivatives). R' and d' can be evaluated at arbitrary points.
Assumption 10 (Uncertainty in di is bounded). Lower and upper bounds on d;, are possible to obtain.
Then, define the following:
Definition (s-gaming). For e > 0, an agent p is s-gaming if Ap(r)) - do >
In other words, we can define a threshold based on some & tolerance in deviations from di for detecting gaming. Now, recall that
R2(Ap(d))) A d(A)(d))- - dt)
(21)
which is non-increasing in Ap(d;) due to the concavity of R and strict convexity of e. Thus, substituting, we could define an agent-specific threshold
(p) R'(E + dp)
(22)
d'{))
and if some procedure for estimating Ap yields an estimate Ax < X (p), we conclude that the agent is e-gaming, while Xp > (p) rules out gaming.
(p) = R' + dz;)
(23)
c'(E)
16

---

However, d is still unknown. Suppose that, we can bound d; E [d,. dpl with probability - 6: (e.g.. via parametric binomial proportion confidence interval about some estimate of dr)::
(E + + c'(c)
+(
(²))
(24)
c(ec)
and abbreviate these bounds to (p) E (p:e), X (m:e)). We can use the same bounds on d to produce range of estimates for Ap:
(d2))
(C))
Xp
(25)
c(Ap(d)) - -dy) ((A,(d)))
and abbreviate these bounds to 2₂ E Ap Apl. Then, we can compare the intervals for X* (p; and Sp as follows:
If Xp < X Ep; then agent is e-gaming with probability --8.
If E > (p:E), then rule out that agent p is e-gaming with probability 8.
Otherwise, Ap E L3P (p) (p; e]] is non-empty, and cannot definitively rule out or prove the existence of e-gaming.
However, this algorithm may be of purely technical interest: is doubtful that the requisite assump- tions satisfied in our motivating setting (upcoding detection), especially Assumption 9, and similarly unclear whether such assumptions apply in other settings. addition, if agents game similarly (i.e, have similar values of Ap), may be difficult to rule out/prove e-gaming for the vast majority of agents.
B.3 Corollary
Corollary. Define T(p,y') above. Then, given Assumptions 1- 8, r(p,p') > 0 if and only if Ap < Apr.
Proof. It is sufficient to show that 7(p p') ) > O if and only if Ap(d)) < Aprrd))) from which Theorem 1 yields the desired result. We can do so by showing (without loss of generality) that E| p, .r]]] is an unbiased estimate of (p(()) (and the case for Ap (dr)) proceeds symmetrically).
Since Ap (dz) is equivalent to P(d,- 1 p] (Eq. 2), the result is immediate:
E| p, x:]] E(d, pl = P(d, = pl = (d))
(26)
L
B.4 Proposition
Proposition. Let 7(-) be the oracle treatment effect and + be some sample estimate of Given Assumptions 1- 8, for any e > 0, if sup [f(p.'' T (p, I S e, then, for all p' such that inf (((p,/)) 8, f(p.p) 0 if and only if Xp < >pr. p.p'
Proof. Choose т, f, and some arbitrary E as specified in the thcorem statement. It suffices to show that for all p, p.p' such that inf (r(p (o) > holds that f(p, ((p,'') > 0 if and only if (pp,p') > 0.
p.p"
(
By contradiction; suppose that f(p,p') > but T (p. p/) < 0. Then, by assumption,
T(p.p" < - But sup [+(v.p') -(pp,v') S
e, so +(p. p') 0. yielding contradiction. Thus, proceeds identically. Thus, the
+(r.p') > 0 = T(p.p') p') The reverse direction =
proposition is true.
17

---

Connections to robustness to non-rational actors. We assume throughout that agents behave rationally; i.e., always perfectly maximize the utility function given by Eq. However, the rational actor assumption may not always hold; egg., if agents do not have the resources to carry out the resource maximizing action, or incorrectly estimate their costs. The former is particularly salient for our motivating problem of Medicare upcoding, in which representatives of certain plans may shedule a home visit with a healthcare professional to generate diagnosis codes [54]-a potentially resource-intensive process. Our ranking formulation can afford some robustness violations of the rational actor assumption:
Remark 2 (Robustness to bounded rationality violations). Let Ap(d)) be the utility maximiznn action, and let Ap(d%) be an agent's 's observed action, such that (dp))) - Ap(dt) E1/22for some
E1 O. Let F(p p) be some sample estimate of T. fitted via (-)) Then:
|+(p,') F(p. =
Ap(d%)) (Ap (dp) A (d,)) S
(27)
Then, sup ((p.p'') T(p. E2 such that E1 + E2, we can apply Proposition 2 directly to conclude that f(p.p') if and only if Xp < >xy-
Intuitively, violations of the rational actor assumption are another source of noise in the estimation of If the noise due to rationality violations plus noise due standard estimation error are low, then no rankings are flipped, as desired.
B.5 Identifiability result
For completeness, we show the derivation of the standard causal effect identifiability result for our problem setting (i.e., as in [37])- We note that this is direct application of known result in the literature.
Proposition. Given Assumptions 6- 8, (p)((p) [n P.Pp'...
R[d4 24 p) and (Ed,(p)) [n R[d,
Proof. We show that E[d((p) wi] E[d: [a,p]; p): the case for [pd)
xy proceeds identically.
We can write
[[4 (p) æ = Eld:(p) xi,p] = Elde s;.
(28)
where the first equality is an application of conditional exchangeability (Assumption 6), and the second is application of consistency (Assumption 7). Positivity (Assumption 8) ensures that the conditional expectations are well-defined.
B.6
What is the expected AUSC?
For completeness, we also analyze the AUSC metric and provide closed-form expression for its expected value under a random ranking. First, consider a population K N agents, where we are interested in top-k sensitivity for some k K1.....K). Suppose that conduct random audits, for some m [1....,K], K}.
The number of ground truth top-k agents that are in the set of m audited agents can be modeled as hypergeometric random variable R'm ~ Hyp(k, K, m). which a variable describing the number of "successes" " observed by random draw from population of K objects with 1 "success states" in m draws without replacement. By standard properties of the hypergeometric distribution, we know that [[ns] = km/K. Thus, by linearity of expectation, the average (across audit intensities m) number of agents identified by random auditing is given by
k K²
K(K 1)
k(K -
E/nml =
(29)
K
2
2K
We divide the result by & to obtain proportion of the top-k identified (as used in the definition of top-k sensitivity), which yields
1 K-1
(30)
2
K
For a finite number of agents, this bounded above by 0.5, but approaches 0.5 as K - oc (a
population of infinite agents).
18

---

C Data Processing
C.1 Fully synthetic data
Overview, Our general fully synthetic data-generation pipeline as follows. First, for each agent, we draw some value of Mp E R². We draw individual observations x(1) E IR² for each agent from Gaussian with mean Mp and some fixed variance o². Given the x(", we simulate a "ground-truth" decision d((3), Note that d=( represents the ground truth decision rate. Then, for each agent we simulate a "gamed" (i.e., strategically perturbed) version of d via a version of the agent utility function in Eq. 2. We now formally describe the data- sitta-geeration process.
Generating a "population" for each agent. To generate a "population" upon which each agent makes decisions, we randomly draw mean vector specifying Gaussian distribution for each agent. Given P agents, the mean depends on Ap as follows:
Hp = R,, max, Ap miny' 2, 3,)=b. X 1og10)())) X
Xp ming Xp
P Σ
Xp
(31)
where a >> 0 is a parameter controlling the range of agent-specific means, and b E IR is some constnat offset. In other words, we fog-transform the Ap values, then apply min-max scaling and a constant shift. This design allows us to control the level of confounding in the synthetic data by changing Ry- We use uy to generate populations for each agent as described below.
Concretely, we consider λc-) E [0.001, 0.003, 0.005, 0.007, 0.009, 0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04, 0.045, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.3, (20 agents), and set b -1. In practice, this means that M4 E [-1, Ry 1]. We manually chose Ac.) values set the difficulty of ranking agents by such that a payout-only approach would succeed under low confounding, but fail under high levels of confounding.
Generating mound-truth and gamed decisions. (i) are drawn as per
The covariates x and ground-truth decisions
x() Nuppt 22
(1) ~ Ber(a a =o(w x + ba),
where ((-) is generated for each agent p(0) as described previously, w is randomly generated positive vector, and ba R. is some offset.
The p(ds are assigned deterministically: i.e., we sequentially draw fixed number x( from each agent-specific distribution and concatenate the results. For realism, inspired by the health insurance setting, and the fact that diagnosis rates for most conditions are relatively low, we set ba to logit(0.05) E[w x] such that (1) is relatively low, where [[[] denotes the sample mean.
Lastly, we simulate gamed agent decisions (d)) by solving a per-agent utility maximization problem:
d(t) Ber(a a arg max log(d,) >p (0))²
d,
Implementation details. process.
All randomness is seeded once the start of the entire data- gett-geenration
C.2 Medicare cohort
We select a 0.2% pseudo-random sample of all U.S. Medicare beneficiaries using the last characters of encrypted beneficiary IDs for inclusion in the cohort. As covariates, we choose age, racial category, biological sex, and diagnosis code categories (defined using the 2018 version of the "Hierarchical Condition Category" [HCC] schema released by the Center for Medicare Services). This yields a total of 97 features.
We exclude enrollees who did not generate any claims (no healthcare utilization; i.e., zero cost as recorded by Medicare), those not located in the 50 U.S. states and the District of Columbia, well
19

---

Ranking performance across auditing thresholds
Top-5 sensilivity (1)
Discounted cumulative gain (DCG,
1.0
75
DCG
50
Top-5
8
10
12
of agents audited
of agents audited
Random
Payout-only
BCOD
DIP
KNN
DragenNet
Relearner
S+PW
Figure 8: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited mean range 0.0, with to error. V: naïve baseline. o: anomaly detection method. x: causal effect estimator.
as dual-eligible beneficiaries. Dual-eligibility refers to individuals simultaneously eligible for U.S. Medicare and U.S. Medicaid. While eligibility for Medicare is primarily based on age, eligibility for Medicaid is primarily based on disability status. Dual-eligible beneficiaries are excluded since the U.S. government uses a different payout model for dual-eligible vs. non-dual-eligible enrollees, potentially violating Assumption 1 (shared rewards), since agents may not be reacting to the same payout model for all enrollees.
Licensing. Our cohort drawn from a 20% sample of all U.S. Medicare beneficiaries provided to the authors under data usage agreement with the Center for Medicare & Medicaid Services.
State-level healthcare statistics. We use a mix of raw and engineered features from the CMS (Center for Medicare Medicaid Services) Provider of Service file (license: Public Use File) [42]* and the National Neighborhood Data Archive (NANDA; CC-BY 4.0) [41]. In NANDA, data is already aggregated at the state level (including the District of Columbia). We keep statistics pertaining to per-capita or per-sq. mi. healthcare provider density (50 features), filtering to providers with non-zero sales. From the CMS Provider of Service file, data reported at the provider level. First, we filter out providers ineligible for Medicare participation, and providers that are longer active. We then manually code ownership information following the provided data dictionaries (for-profit vs. non-profit vs. publicly owned). Next, we compute at the state level the prevalence of for-profit, non-po non-profit, and publicly-owned providers of each type (as defined by CMS) by state, as well as the ratio of for-profit to non-profit providers. Additionally, physician count, medical school affiliation rate, and compliance rate as certified by CMS, for a total of 54 features. All aggregations for the Provider of Service file unweighted (i.e., all hospitals/other healthcare facilities contribute equally). This yields a total of 104 state-level health indicators.
extract the average per-hospital bed count,
We report the statistics derived from the provider of service file (Table 2) and NANDA (Table 3), with summary statistics across states. Summary statistics are computed excluding infinite and NaN values (i.e., ratios of 0/0 or 1/0). When computing correlations, exclude states with NaN values for that feature (i.e., ratio of 0/0). Proportions of publicly-owned, non-profit, and for-profit providers do not sum to because providers reporting "unknown" "other" ownership excluded. For further information on the feature definitions, consult the data dictionaries for the provider of service" and NANDA files.
D Supplementary results
D.1 Causal vs. non-causal approaches, all audit thresholds & all levels of confounding
We show plots with the top-5 sensitivity and DCG at all levels of confounding evaluated, as measured by the mean range R2 (i.e., R2 max Me min 14p across agents). We choose Rp E {0, 0.., 1.03. For convenience, we provide an index of results:
Full information on Public Use File licensing as defined by CMS is here: https://www.com cms www.gov/... . research- .ber. http:///w....eeaaa/////////aarra hhttps://www www.wwwwwpccppr....mmm.m wwwpeesrrrrropppeeieeeeeee htwpa///ge.oooaooooooo/o///////a
20

---

Feature name
Meants.d.
bed count
95.125=27.826
% affiliated with medical school
3.385 0.350
# of personnel
87.182=47.477
% non- non-compliant
0.115=0.074
% of providers that are non-profit hospitals
0.055=0.030
% of providers that are for- for-profit hospitals of providers that are publicly-owne hospitals % of providers that are non-profit skilled nursing facilities
0.024
0.028-0.022
0.069=0.046
% providers that are for-profit skilled nursing facilities
0.171=0.068
% of providers that are publicly % providers that are non-profit home health agencies % of providers that are for-profit home health agencies
skilled nursing facilities
0.020=0.021
0.033=0.018
0.104=0.075
of providers that are publicly-ownn % of providers that non-pron non-profit OP PT/SP providers of providers that are for-profit OP PT/SP providers
home health agencies
0.008=0.010
0.005=0.010
0.025=0.018
% of providers that publicly-ownn OP PT/SP providers
0.001=0.001
% of providers that are non-profit end-stage renal disease facilities
0.014-0.012
% of providers that are for-profit end-stage renal disease facilities
0.096=0.042
of providers that are publicly-owned end-stage renal disease facilities
0.001=0.002
% of providers that are non-profit ICF/IDs
0.041=0.052
% of providers that are for-profit ICF/IDs
0.025=0.050
% of providers that publicly-owned ICF/IDs
0.005=0.007
% of providers that are non-pronn non-profit rural health clinics
0.044=0.044
of providers that are for-profi rural health clinies
0.024=0.028
% of providers that are publicly-owned rural health clinics
0.014=0.020
of providers that are non-profit ambulatory surgery centers
0.004=0.005
% of providers that are for-profit ambulatory surgery centers
0.084=0.061
% of providers that are publicly-on lownie ambulatory surgery centers % of providers that are non non-profit hospice care facilities
0.002-0.004
0.020-0.012
of providers that are for-profit hospice care facilities
0.039=0.033
% of providers that are publicly-owne hospice care facilities of providers that are non-profit % of providers that are for-profit % of providers that are publicly-owned % of urbanized population
0.002=0.004
0.285
0.592=0.154
0.081=0.066
0.679-0.228
of providers that are ambulatory surgery centers
0.091=0.064
% of providers that are end-stage renal disease facilities
0.111=0.040
% providers that are home health agencies
0.145=0.069
% of providers that hospitals % of providers that are hospice care facilities
0.118=0.038
0.070-0.035
of providers that ICF/IDs % of providers that are OP PT/SP providers
0.007=0.009
% of providers that are rural health clinics % of providers that are skilled nursing facilities ratio of for-profit non-profi providers ratio of for profit to non non-profit hospitals
0.082=0.067
0.261=0.073
2.838-2.024
0.750-0.955
ratio of for- propt to non-profit skilled nursing facilities
3.633-2.387
ratio of for for-ptofit to non-profit home health agencies
5.135=6.020
ratio of for profit non-pron non-profit OP PT/SP providers
+10.10
ratio of for-profit to non-profit end-stage renal disease facilities
15.905 +18.5411166
ratio for-profit to non-profit ICF/IDs
1.074*+1.96
ratio of for-profit to non-profit rural health clinics
0.936 1.380
ratio of for-profi to non-proe non-profit ambulatory surgery centers
32.826 430.949
ratio of for-profe profit to con-profit hospice care facilities
3.989
Table 2: Features chosen for analysis derived from the provider service file, with mean and standard deviation across states. S.d.: standard deviation. OP PT/SP: outpatient physical therapy and speech pathology. ICF/ID: intermediate care facility for individuals with intellectual disability. "; invalid (1/0 or 0/0) ratios dropped.
21

---

Feature name
Mean-Es.d.
ambulatory health care specialists per 1000 people
4.429=2.745
ambulatory health care specialists per sq. mi.
15.480=116.925
physicians per 1000 people
1.196
physicians per sq. mi.
6.477+7.593
physicians (excluding mental health) per 1000 people physicians (excluding mental health) per sq. mi. mental health physicians per 1000
1.656=11.00
6.137=7.195
0.096:10.12
mental health physicians per sq. mi.
0.340=0.412
dentists per 1000 people
0.639=0.242
dentists per sq. mi.
2.553=2.945
all other health practitioners (besides the above) per 1000 people all other health practitioners (besides the above) per sq. mi.
1,483=11039
5.949455.264
chiropractors per 1000 people chiropractors per sq. mi.
0.252
0.840+00.99
optometrists per 1000 people optometrists per sq. mi.
0.100=0.026
0.307
non-physician mental health practitioners per 1000 non-phi con-physician mental health practitioners per square P/O/ST and audiologists per 1000 people P/O/ST and audiologists per sq. mi.
0.176=10.390
0.687
0.133=0.054
0.474=0.657
other health practitioners (besides the above) per 1000 people other health practitioners (besides the above) per sq. mi.
0.792++0.610
2.640-22.722
outpatient centers per 1000 people outpatient care centers per sq. mi.
0.181+0.133
0.556+00.516
diagnostic labs per 1000 people diagnostic labs per sq. mi. home health services per 1000 people home health services per sq. mi.
0.075=0.114
0.229-1.2330
0.173+:..12
0.517=0.441
other ambulatory care services (besides the above) per 1000 people other ambulatory care services (besides the above) per sq. mi.
0.125=0.327
0.198=0.176
nursing and residential care facilities per 1000 people all nursing and residential care facilities per sq. mi.
0.334+0.324
0.656
nursing care facilities per 1000 people
0.224++0.330
nursing care facilities per sq. mi.
0.473++0.08
residential IDD care facilities per 1000 people residential IDD care facilities per sq. mi.
0.026+10014
0.102
inpatient facilities for care of individuals with IDDs per 1000 people
0.012+10010
inpatient facilities for care of individuals with IDDs per sq. mi.
0.039=00068
inpatient facilities providing MHSA care per 1000 people inpatient facilities providing MHSA care per sq. mi.
0.01410.007
0.063=0.109
continuing care and assisted living facilities per 1000 people continuing care and assisted living per sq. mi.
0.023++0011
0.070++0.062
other residential care facilities (besides the above) per 1000 people other residential care facilities (besides the above) per sq. mi.
0.061+0.031
0.191
pharmacies and drug stores per 1000 people pharmacies and drug stores per sq. mi. optical goods stores per 1000 people optical goods stores per sq. mi. misc. other health and personal care stores per 1000 people misc. other health and personal care stores per sq. mi.
0.31410.669
0.853+1.336
0.076:10.025
0.282+00360
0,054+00014
0.149=00000
Table 3: Features chosen for analysis derived from NANDA, with mean and standard deviation states. All chosen summary statistics for providers with SO U.S. dollars sales. S.d.:
standard deviation. Sq. mi.: square mile. P/O/ST: physical, occupational, and speech therapists. IDD: intellectual and developmental disabilities. MHSA: mental health and substance abuse.
22

---

Ranking performance across auditing thresholds
Top-5 sensilivity (1)
Discounted cumulative gain (DCG,
1.0
75
DCG
50
Top-5
10
of agents audited
of agents autited.
Racdom
Payout-only
BCOD
DIP
KNN
DragenNet
Relearner
S+IW
Figure 9: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited mean range 0.1, with to error. V: naïve baseline. o: anomaly detection method. x: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity (1)
Discounted cumulative gain (DCG,
1.0
75
DCG
50
Top-5
8
10
12
14
18
of egents audited
agents aucited
Random
Poyout-only
BCOD
DIF
KNN
-X-
DragonNet
-X-
R-learner
S+IPW
Figure 10: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited at mean range 0.2, with =o error. V: naïve baseline. or anomaly detection method. x: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity (7)
Discounted cumulative gain (DCG,
1.0
75
50
Top-5
0.5
8
10
12
14
16
18
2
8
10
12
14
16
18
of egents audited
of agents audited
.
Random
Payout-only
BCOD
DIF
KNN
-X
DragonNet
-
R-Jearner
S+IPW
Figure 11: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited at mean range 0.3, with to error. V: naïve baseline. or anomaly detection method. x: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity 11)
Discounted cumulative gain (DCC, t
1.0
75
DCG
50
0.5
8
10
12
14
16
18
2
8
10
12
14
16
18
agents audited
agents audited
Random
Payout only
--- BOOD
DIF
KNN
DragenNet
-
R-learner
-
S+IPW
Figure 12: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited at mean range 0.4, with to error. v: naïve baseline, or anomaly detection method. x: causal effect estimator.
R 0.0: Figure
* R, 0.1: Figure
R2 0.2: Figure 10
R2 0.3: Figure 11
R2 0.4: Figure 12
R 0.5: Figure 13
23

---

Ranking performance across auditing thresholds
Top-5 sensilivity (1)
Discounted cumulative gain (DCG,
1.0
75
DCG
50
Top-5
8
10
of agents audited
of agents aucited
Random
Payyut-only
BCOD
DIP
KNN
DragenNet
Relearner
S+PW
Figure 13: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited mean range 0.5, with too error. V: naïve baseline. o: anomaly detection method. x: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity (
Discounted cumulative gain (DCG, 1
75
DCG
0.5
25
0.0
12 agents andited
10
18
agents aucited
Random
-
Payout-only
ECOD
DIF
KNN
-
DragenNet
*
R-learner
S+IPW
Figure 14: Mean top-5 sensitivity (left) and DCG (right) across of agents audited at mean range 0.6, with to error. naïve baseline. anomaly detection method. causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG,
1.0
75
50
Top-5
25
0.0
of agents audited
of agents autited
Random
Payout only
RCOD
KNN
DragenNet
R-learner
S+IPW
Figure 15: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited mean range 0.7, with to error. V: naïve baseline. o: anomaly detection method. x: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG, t
1.0
75
DCG
0.5
25
8
10
12
14
18
10
12
16
18
agents audited
agents aucited
Random
.
Payout-oni only
ECOD
DIF
KNN
DragenNet
-
R-learner
S+IPW
Figure 16: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited at mean range 0.8, with to error. 7: naïve baseline. anomaly detection method. x: causal effect estimator.
R 0.6: Figure
* R 0.7: Figure 15
Ry 0.8: Figure 16
R2 0.9: Figure
R2 1.0: Figure 18
24

---

Ranking performance across auditing thresholds
Top-5 sensilivity
Discounted cumulative gain (DCG,
1.0
75
DCG
50
Top-5
8
10
of agents audited
of agents aucited
Random
Payout-only
BCOD
DIF
KNN
DragenNet
Relearner
S+PW
Figure 17: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited mean range 0.9, with to error. V: naïve baseline. o: anomaly detection method. x: causal effect estimator.
Ranking performance auditing thresholds
Top-5 sensilivity (1)
Discounted cumulative gain (DCG, 1
75
DCG
0.5
25
0.0
10
of agents audited
of agents audited
Random
Payout-only
ECOD
DIF
KNN
DragenNet
*
R-learner
S+IPW
Figure 18: Mean top-5 sensitivity (left) and DCG (right) across # of agents audited at mean range 1.0, with too naïve baseline, anomaly detection method. causal effect estimator.
Arna under the sensitivity curve (AUSC, 1) vs. confounding strength
0.8
0.6
0.4
0.2
0,0
0.1 Confounding (range across agent specific covariate distributioas)
0.2
0.3
0.4
0.5
Random
ECOD
KNN
S-learner
*
DragonNet
$ Blearmer
Fleamnr
S+IPW
Payout-only
Figure 19: Area under the sensitivity curve (AUSC) for all methods tested across levels of confounding (mean range R S 0.5). v: naïve baseline. or anomaly detection method. x: causal effect estimator.
Ama under the sensitivity (AUSC, 1 confounding strength
0.8
AUSC
0.6
0.4
b
0.2
0.7
0.8
0.9
Confourdingg level (range of means across agent-speciffe covariate distribsttons))
Random
KNN
S-larnee
DragonNet
Rleaareer
Fleamer
S+IPW
Popout-oniy
DIF
PSM
Figure 20: Area under the sensitivity curve (AUSC) for all methods tested across levels confounding (mean range R, > 0.5). v: naïve baseline. anomaly detection method. x: causal effect estimator.
We summarize the main trends for non-causal approaches and defer discussion of causal methods to the sensitivity analysis of all causal effect estimators. For convenience, we also plot the AUSC for all approaches across levels of confounding Figures 19 (R,, S 0.5) and 20 (R,, > 0.5). For readability, in contrast to our other figures, causal effect estimators are marked with "6" instead of "x
The payout-only approach can degrade to worse-than-random ranking due confounding. The random auditing method performs similarly across all levels of confounding, as expected.
25

---

However, as confounding increases, the payout-only ranking degrades toward random, then worse than random. The latter can occur if confounding is so strong that the relationship between gaming and observed diagnosis rates flips; e.g., if (in the health insurance setting) very dishonest plans tend to serve relatively healthy populations compared to more gaming-averse plans.
In the synthetic dataset, anomaly detection methods use the variance of the observed decision (V[d)D as gaming signature. Most anomaly detection methods perform near-random, or slightly better than random. While this is due to the properties of the synthetic dataset, the results highlight potentially interesting characteristics of anomaly detection methods for gaming detection. Recall that the anomaly detection methods take covariates and agent decisions (2i, di) as input. V(s; is identical across agents by design, and all agents see the same number of observations. However, V[d.] may vary across agents. Thus, KNN uses V[d:] as a signal of gaming, which has utility under low confounding, but is less useful as confounding increases.
To see this, recall that de is binary decision, and let pa - P(d; 1) for some agent. V[d, is proportional to Pa (1 (1-Pa) Pa), and is concave in Pa with maximizer pa 0.5. Since P(d,) is generally low in simulation (i.e., < 0.5), agents with higher observed P(d,) rates will generally have higher V[d,] as well. yielding higher anomaly score. Under no confounding, these are precisely the agents that are gaming more. Indeed, KNN performs slightly better than random, but its advantage over random performance diminishes slightly with confounding as the utility of V(d.) as signature for gaming. However, even at low confounding, KNN and related anomaly detection methods are inherently unable to detect gaming in non-outlier points, which occur in denser regions of covariate space. In these regions, causal methods enjoy advantage over anomaly detection approaches due to improved overlap (Assumption 8). Thus, KNN does not exceed the ranking performance of the best causal methods. Note that this argument assumes that V [z:]] is similar across agents, which holds in the synthetic dataset.
More advanced anomaly detection methods can also achieve slightly better than random performance (e.g.. DIF), but since these approaches transform the covariate space in highly non-linear ways (e.g.. via random projections as in DIF, or via feature- CDFs as in ECOD), they may destroy outlier information useful for gaming detection. Ultimately, anomaly detection methods have inherently limited utility for gaming detection, since some gamed decisions may appear distributionally close to other gamed decisions.
D.2 Sensitivity analysis of eausal effect estimators
Ranking performance across auditing thresholds
Top-5 sensitivity )
Discounted cumulative gain (DCG, t)
20
6
10
12
14
16
18
10
12
14
16
18
aagenss audited
of aggnts audited
Causel offect estimator B-loarner
S-learner
DragoaNet
Tleamme
- S+IPW
PSM
Bandom
Payout-only
Figure 21: Sensitivity analysis of all causal methods tested, mean range 0.0. V: naïve baseline. o: causal effect estimator.
We show plots with the top-5 sensitivity and DCG at all levels of confounding evaluated (as measured by the mean range R2 E (0,0.1.... 1.0)) for causal effect estimators only, plus random and payout-only methods for comparison. An index of figures follows:
Figure 21: R2 0.0
Figure 22: R,, 0.1
Figure 23: R. = 0.2
Figure 24: R2 -.3
Figure 25: R. 0.4
26

---

Ranking performance across auditing thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG, 1)
Top-5
20
10
12
14
16
18
10
12
14
16
18
agents anditted
of aggnnt auditted
Causal
S-leurne
DrageaNet
Tleamer
Relearner
S+IPW
PSM
Random
Payout-unly
Figure 22: Sensitivity analysis of causal methods tested, mean range 0.1. V: naïve baseline. o: causal effect estimator.
Ranking performance across audiling thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG,
1.0
0.5
40
20
0.0
10
14
16
10
12
16
16
agants andited
agents and
Causal effect estimatzor
S-leurner
DrugoaNet
Payout-unly
Figure 23: Sensitivity analysis of all causal methods tested, mean range 0.2. V: naive baseline. causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity )
Discounted cumulative gain (DCG, t)
60
20
0.0
10
10
4
12
10
agents audited
od agents audited
Causal effect Relearner
S-learner
DrageaNet
Tleamme
SHIPW
PSM
Random
Payout-only
Figure 24: Sensitivity analysis of causal methods tested, mean range 0.3. V: naïve baseline. causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity (t)
Discounted cumulative (DCG, t
1.0
80
0.5
40
20
2
10
12
14
16
18
8
10
12
14
16
18
agents
agents
Causal effect estimator R-leanner
S-leurner
DrageaNet
Tlearer
S+IPW
PSM
Random
Payout-only
Figure 25: Sensitivity analysis of all causal methods tested, mean range 0.4. V: naïve baseline. causal effect estimator.
Figure 26: R, 0.5
Figure 27: R, 0.6
Figure 28: R, 0.7
27

---

Ranking performance across auditing thresholds
Top-5 sensitivity t
Discounted cumulative gain (DCG, T)
60
Top-5
20
10
12
14
16
18
10
12
14
16
18
agents andited
of aggnns auditedd
Causal
S-leurne
DrageaNet
Tleamer
Relearner
S+IPW
PSM
Random
Payout-unly
Figure 26: Sensitivity analysis of causal methods tested, mean range 0.5. V: naïve baseline. o: causal effect estimator.
Ranking performance across audiling thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG,
1.0
DO3
0.5
25
0.0
10
14
16
10
12
16
10
agants andited
agents and
Causal effect estimaterr
S-leurner
DrugeaNet
Payout-unly
Figure 27: Sensitivity analysis of all causal methods tested, mean range 0.6. V: naive baseline. causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity )
Discounted cumulative gain (DCG, t)
50
0.0
10
10
4
12
10
agents audited
od agents audited
Causal effect Relearner
S-learner
DragoaNet
Tleamer
-
SHIPW
PSM
Random
Payout-only
Figure 28: Sensitivity analysis of causal methods tested, mean range 0.7. V: naïve baseline. causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity (
Discounted cumulative (DCG, 1
1.0
75
DCG
0.5
25
2
6
10
12
14
16
18
8
10
12
14
16
18
agents
agents
Causal effect estimator
S-leurner
DrageaNet
Tlearee
R-leanner
S+IPW
PSM
Random
Payout-only
Figure 29: Sensitivity analysis of all causal methods tested, mean range 0.8. V: naïve baseline. causal effect estimator.
Figure 29: R, 0.8
Figure 30: R, 0.9
Figure 31: R, 1.0
28

---

Ranking performance across auditing thresholds
Top-5 sensitivity t
Discounted cumulative gain (DCG, T)
75
50
Top-5
10
12
14
16
18
10
12
14
16
18
agents andited
of aggnnt audd
Causal effect
S-leurne
DrageaNet
Tleamme
Releazner
-
S+IPW
PSM
Random
Payout-unly
Figure 30: Sensitivity analysis of all causal methods tested, mean range 0.9. V: naïve baseline. o: causal effect estimator.
Ranking performance across auditing thresholds
Top-5 sensitivity
Discounted cumulative gain (DCG,
1.0
0.5
DCG
25
2
10
14
16
4
12
16
18
agonts audited
agents sudited
Causal efficct estimator
S-learnor
DrageaNet
Telarner
Bllarner
SH1PW
PSM
Random
Payout-only
Figure 31: Sensitivity analysis of all causal methods tested, mean range 1.0. V: naïve baseline. o: causal effect estimator.
Note that, even absent confounding, causal approaches outperform the payout-only model. This is because causal approaches explicitly incorporate the covariates into modeling, while the payout-only model directly uses the marginal outcome distribution. Incorporating covariates into regression models for treatment effect estimation can decrease estimator variance [37] (but is not guaranteed to do so), consistent with the empirical results.
Propensity score matching (PSM) also performs poorly across all levels of confounding. Since the matching approaches conduct matching pairwise across observations seen by agent pairs, the causal effect estimates are computed in a subset of similar observations across one pair of agents, but not the subset of similar observations across all distributions agent observations. This suggests that controlling for confounding simultaneously across all levels of treatment is potentially important for applying causal effect estimators to gaming detection. Ultimately, matching approaches may not scale to large numbers of treatments, such as those expected multi-agent strategic adaptation. Non- optimal matching approaches (e.g., greedy matching without replacement) potential workaround, but we leave the adaptation of matching methods to large numbers of treatments to future work.
We note that, at low levels of confounding, the difference between the S-learner and T-learner may be detaset-d empirically, S-learners often regularize causal effect estimates towards zero, while T-leamers thrive when causal effects are non-zero and heterogeneous [44, 55], in synthetic dataset. Thus, per-agent modeling, as done by the T-learner and DragonNet, can better capture the complex treatment effects in our dataset.
Furthermore, the R-learner and S+IPW both perform poorly low levels of confounding, but improve at high levels of confounding. Since both the R-learner and S+IPW fit a nuisance propensity score estimator, this suggests that difficulties in propensity score estimation low levels of confounding could potentially explain the observed trends.
The underperformance of the R-learer may be surprising given its doubly-robust properties, but the high-dimensional generalization [47] requires restrictive conditions for convergence. Formally, the R-learner fits four models m, g, h, e of the form
d 2~m m(x) 9(xx) (h(p)) e(x)),
(32)
29

---

where m is fit independently, and 9, h, are fitted using alternating optimization. The final treatment effect estimate of swapping from agent p to p' is given by g(z) h(p) g(x) h(p',, but the oracle representation of l(.)) is unknown and must be fitted. Convergence of the nuisance parameter estimate of e(x) the oracle value of h(p)) necessary for convergence of the overall treatment effect estimate.
E
Training
E.1
Model architectures
All approaches that fit model are built based on a fully-connected neural network with two hidden layers and 300 neurons per layer plus ReLU activations. The output of the neural network either has size two with softmax non-linearity (for classification; i.e., predicting agent decisions), or no activation and a pre-specified output size (i.e., for generating feature maps in the high-dimensional R-learner).
We describe approach-specinn modifications to the architectures as follows:
S+IPW.
The estimated weights are used a sample weight when training the S-learner. At inference time, weights are also computed for test examples based on the propensity model fitted on the training set take an inverse propensity-weihtted average of the S-learner estimates.
DragonNet. We changed the propensity prediction head from binary classification (as in the original paper [45]) to a multi-class classification head, since there are multiple treatments in our setting. Furthermore, introduce outcome modeling head per agent. Targeted regularization omitted due to the multi-treatment setup.
R-Learner. Feature maps for all nuisance parameters have dimensionality 10. The generalized R- learner alternating optimization to fit some nuisance parameters, where two models representing product decomposition the response function are updated K times for every update of "propensity feature" model. We K 10; we defer [47], page for more details about the training procedure for the generalized R-learner, from which we designed our implementation.
E.2 Anomaly detection hyperparameters
For KNN, we keep the neighborhood size at 5, the default value. DIF uses neural network-based random projections to compute anomaly score. Thus, used for causal approaches, with an ensemble of 50 representations. Each representation is used as input into an isolation forest of size [19].5 ECOD does not take hyperparameters [22].
the same architecture for DIF as
E.3 Dataset splits
We use a seeded random development-test split (7:3) for all datasets. The development split is reserved for all model fitting, while all causal effect estimates are reported on the test split. The development split further randomly split into training and validation set. All model selection techniques (e.g., early stopping) are performed with respect to evaluation metrics on the validation
E.4
Training hyperparameters
Fully synthetic data We use the following hyperparameters for training all models:
Optimizer: SGD with learning rate 10-2 and weight decay 10-3
$ Ann isolation forest a separate anomaly detection method, which anomaly score is related to number random "splits" with respect to a single randomly-select covariate (i.e., a threshold of the form æ; 2 r) needed to isolate point. This method assumes that outliers are "easier" to isolate, and require fewer splits. Multiple randongspliting routines (isolation trees) ensembled to form an isolation forest.
30

---

Learning rate schedule: We reduce the learning rate by a factor of 0.1 after 5 epochs of con-improvement with respect the validation loss.
Training length: A maximum of 1000 epochs, with early stopping (patience: 10 epochs) based on validation loss.
Medicare FES
Optimizer: SGD with learning rate 10-2 and weight decay 10 10-3,
Learning rate schedule: We reduce the learning rate by a factor of 0.1 after epochs of non-improvement with respect the validation loss.
Training length: A maximum of 1000 epochs, with early stopping (patience: 10 epochs) based on validation loss.
Software and Hardware
F.1
Software
All code was written in Python 3. 10.4 (license: PSF). All non non-causal anomaly detection approaches
were implemented using PyOD (license: BSD 2-clause) [56]. All neural networks were implemented
in PyTorch 2.2.0 (licens:: Custom "BSD-style"") [57], using Skorch 0.15,0 (license: BSD computed using both Scikit-Learn 1.3.2 (license: BSD
clause) [58] wrapper. Metries
3-clause) [59] and Scipy 1.11.4 (license: BSD 3-clause) [60]. For the fully synthetic data generation process, CVXPY 1.4.2 (license: Apache 2.0) [61] was used to solve each agent's utility maximization problem, and used tandem with SCIP 9.0 (ppscipopt 5.0.0; license: Apache 2.0) for the matching approaches (formulated mixed-integer programs) [62]. Numpy 1.22.3 (license: BSD-style) [63] and Pandas 2.0.3 (license: BSD 3-clause) [64] were used for data manipulation. Matplotlib 3.8.2 (empirical results; license: PSF-style)" and Adobe Illustrator 2023 (overview figures; license: commercial, "Named User Licensing" were used for figure generation. For the Medicare cohorts, we generated HCC (Hierarchical Condition Categories; used by the Center for Medicare Services) codes from raw diagnosis codes reported in claims data via HCCPy 0.1.9 (license: Apache 2.0)²²
Other dependencies include tqdm 4.66.2 for rendering progress bars (license: MPL 2.0 and MIT), gitpython 3.1.43 for bookkeeping (license: BSD 3-clause), pandarallel 1.6.5 for parallel data processing (license: BSD 3-clause), and ruamel 18.6 (license: MIT) for configuration file manage- ment. All software excepting Adobe Illustrator is open-source and free for use.
F.2 Hardware
All experiments were on either one Titan V V100 GPU using 12.9GB of RAM managed via a Slurm job submission system. Computing nodes had two 2. 10 Intel Broadwell (Xeon E5-2620V4) processors each (16 cores total). Execution time was limited six hours per run, but all training runs (one model type on 10 datasets) lasted under one hour due to the relatively small size of the architectures and datasets under consideration.
G Code
The fully-synthetic datasets, experimental code, and implementations of all approaches under eval- uation will be made publicly available at https://gil //github. com/MLD3/ happ///aagngg.///e detection. authors do not have permission to release any of the Medicare data, but will release the relevant data processing code.
gaming_detection. The
SSc httpt/////ggggtooooooooo///////a
'SSee httpp///tthttloooooom/mm//m///aa txt. USee 1eSee http:: htt.::/s...ppp....o..c....ooooea html.
31

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] is generally preferable to "[No]" ", it perfectly acceptable to answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" " or "we were unable to find the license for the dataset we used"). ). In general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction claim a causally-motivated framework for gaming propensities (Section 3), theoretical results (Section 4) proving the feasibility of ranking using causal effect estimation (Theorem and Corollary 1), and a study of causal effect estimators (as enumerated in Section 5.1) in a synthetic data study (Section 5.2) and case study in health insurance (Section 5.3), as required.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes; see the "Limitations and Broader Impact" section in the Conclusion.
Guidelines:
32

---

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions numbered and formally stated. Each theorem statement enumerates the necessary set of assumptions. Detailed proofs are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have aimed to provide sufficient details about the method and datasets to
replace our work on the fully synthetic data. Hyperparameter details are included in the Appendix. Due to restrictions on the usage of the Medicare data, we are unable to share the underlying dataset, but provide detailed information about cohort selection and data processing instructions.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: To the extent legally permitted, we provide code and open access the fully synthetic data used, and all methdological contributions. Due to restrictions on the usage of the Medicare data, we are unable to share the underlying dataset and the authors do not have the power to grant access to the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.. ec/ public/ htte//guudd///eeooee SlodeSubmissionPP wwwooooooogy (oooolyyy)) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: // //nips cc/ htt.cccppppcclceeeeeeeeee httplllcgggiese/eeee www.li//gg.ddds/odddiiiiiiiisiis https//oooooaammisisooooooe.ee.e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
34

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Data splits and hyperparameter settings (including the optimizer) are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report I-sigma error ranges (as explicitly stated figure captions) of performance metrics on the test set across training runs on different instances of fully synthetic data. We do not make empirical claims about statistical significance, but report p-values for correlations with respect to state-level healthcare statistics in Section 5.3.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
35

---

Answer: [Yes]
Justification: We describe the software and hardware used in the Appendix. An upper bound on computation time was provided (time limit on Slurm job submission system).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neut http:///wwrrppscccccpp.ttttttta/
ines?
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and certify that this work meets the code in their best judgment.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Yes; see the "Limitations and Broader Impact" section of the Conclusion for further discus- sion. We advise careful consideration regarding the implications of models developed using our theoretical results on structural inequities. We fully acknowledge that models developed using our theoretical work can potentially have both positive and negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
36

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our contribution is theoretical, minimizing the direct risks of misuse. However, indirect/downstrramm risks are discussed in the "Limitations and Broader Impact" section the Conclusion for further discussion. We advise careful consideration regarding implications of models developed using theoretical results structural inequities. Risk downstream misuse is contingent on the institution adopting methods designed using our framework. Risks stemming from data release are minimal, since the authors not permitted to release the real-data cohort.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Citations and/or links to relevant code, and data and software are provided in the Appendix. License information is provided for datasets where possible.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
37

---

Answer: [Yes]
Justification: The only new asset is the code repository used for our experiments. We provide instructions in the README for regenerating the datasets used for the synthetic data experiments, data-processing code for the Medicare cohort, and experiments. Figures were created in Jupyter notehook included with the code supplement.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The extent of interaction with human subjects in this work is secondary data analysis of retrospective cohort. The data was purchased and used following data usage agreement by the Center for Medicare Services.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [Yes]
The Medicare datasets were requisitioned from the U.S. Center for Medicare & Medicaid Services under data usage agreement authorizing scope of analyses including gaming in U.S. Medicare. IRB review was not required.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
38


Reviews:
[{"review_id": "nTdHOLsdzd", "reviewer": "Reviewer_dPj8", "summary": "The paper considers the problem of identifying agents with the highest values of scaling parameters in a stylized strategic adaptation optimization model under a wide range of assumptions. The paper casts this problem as a ranking problem via causal effect estimation and provides an algorithm to rank the parameters of agents. \n\nThe paper then provides an experimental evaluation using synthetic and real-world medicare data to show the observations from the proposed models and algorithm.", "questions": "N/A", "limitations": "See the above and below comments. \n\n\nAbstract \n\n\"machine learning models\" -> provide examples of how the interaction would look like \n\n\"their inputs to the model\" -> such as? what are the outcomes here? from the ML models? what utility? the agents using the models for something? it is very unclear\n\n\"We consider a multi-agent setting\" -> what is the motivation for this goal? what is the implication if it is achieved? why do we care if it games? What do you mean by aggressively? in fact, how do you even define aggressive? \n\n\"identifying such agents is difficult\" -> why? it is not clear to me how would you do this even if you know the utility; why make it more difficult now? \n\n\" is parameterized via a scalar\" -> the meaning is not clear; what is parametrized? agent utility function? what is a scalar here? why is this realistic at all? \n\n\"is only partially identifiable\" -> what does it mean by this? \n\n\"By recasting the problem\" -> why recasting the problem? why this approach is justifiable? \n\"causal effect estimation problem\" -> what is this problem? what is the connection to the worst offenders? very confusing here; the next phrase makes little to no sense at all; why/what is the identifiable property? \n\n\"in a synthetic data study\" -> only one data set? \nwhat is this coding behavior? what does it have to do with gaming? \nwhat are the features? what do you mean associated with gaming? \n\n1 Introduction\n\n\"guide decisions that impact individuals or entities\" -> provide examples \n\nplease review the literature on mechanism design and game theory as well as their ML related applications \n\n\"obtain a more desirable outcome\" -> outcome is what here? what is the connection between ML models and outcomes? \n\n\"to the difficulty of generating supporting evidence\" -> provide examples; explain more \n\n\"\"utility maximization:\" -> fix\n\n\"to maximize a payout\" -> what is the payout? utility function? \n\n\"which calculates a\" -> the government or the companies? \n\" via a publicly available model\" -> how do you know they use this model? \n\n\"Companies may attempt to\" -> what happens to the companies that do this? \n\n\"Beyond health insurance, gaming emerges\" -> how does health insurance impact individuals? who is doing the gaming in the said applications? what are you trying to change in the features? \n\nAlso, are the models already trained? or are you talking during learning? \n\n\"agents with the highest propensity \" -> how do you measure this? \n\n\"given a dataset of agents\" -> dataset of agents doesn't sound right? what does it mean here? do you mean a set of agents? \n\"their observed model inputs\" -> is the model fixed here? how do they interact with the models? why can't do this together? \n\n\"fraud/gaming labels\" -> what does this mean? \n\nwhat is the utility here? what are they affecting? what is the connection to distributional assumptions? these are used with little to no context\n\n\"But past works in strategic adaptation\" -> what does it mean by costs in this context? is that the only difference from this vs the other models? \n\n\"scalar gaming parameter that scales costs \" -> the meaning of this is not clear; what is this parameter? it is not explained in texts; the figure is not meaningful without a clear description \n\n\"partially identifiable\" -> what is the meaning of this? \n\n\"However, by recasting gaming detection as causal\" -> again, why is this justifiable? what is this causal effect estimation going to do? \n\n\"that ranking agents\" -> what ranking is solving? I am very confused \n\n\"a cost-scaling factor\" -> why is this more realistic? \n\n\"Furthermore, much work in strategic a\" -> this paragraph seems to be more like related work section; it is out of place and disrupts the flow of reading; you should probably consider an independent related work section; they are also hard to understand with little to no background context presented \n\nWhat is the contribution here? there should be a contribution subheadings and/or related work also \n\n\"a synthetic dataset\" -> how do you have a ground truth on this? what about other datasets? \n\n\"causal approaches rank the \" -> hard to understand this sentence \n\n\"healthcare providers, a suspected driver of gaming\" -> you can say that unless you have concrete evidence; otherwise, you will sued with claims like this \n\n\"In summary: we \" -> this paragraph provides new information that is not discussed or connected to earlier messages\n\n\n2 Background & Problem Setup\n\n\"to a payout\" -> what is a payout here? why f is mapping only a single agent attribute? should it maps from more agents? or even datasets? \n\n\"according to some function\" -> what is this function? How is R dependent on d and f? why d' has to be in D? \n\n\"For simplicity, we assume R = f ; \" -> if R = f, the function in (1) makes no sense what is the meaning of f has itself as input? the math is not correct \n\nHow is strategic classification used in strategic adaptation? \n\n\n\"To extend strategic adaptation to multiple agents\" -> why is this justifiable at all? \n\nWhat is M_p? What is d_i modeling? what decisions are they making? \n\n\"Agent assignment is\" -> how is this model? how do you indicate assumption; i am still not clear about D_p? is that for an agency? within the agnecy they have M_p agents who can manipulate? \n\n\" to obtain a higher payout.\" -> why do they want to increase d_i? why can't they decrease? \n\nWhat do they perturb the average instead of individual d_i? how does this connect to f defined earlier? \n\n\"is the ground truth value\" -> how do you actually get the ground truth? before you define c(,) as two parameters and now you have a single parameter; it doesn't look consistent \n\n\"we introduce assumptions on the\" -> you need to justify these assumptions; are they common in the literature? how do you model multiple agents here? it seems (2) only for one agent\n\n3 Theoretical analysis: finding agents most likely to game\n\n\"We aim to identify agents most likely\" -> wait; is lambda_p unknown? why is this the right way to identify agents? \n\n\"be point-identified\" -> unclear meaning \nwhat is the meaning of partially identifiable? You need to provide implications of Prop 1 \n\nI still don't get why  estimating counterfactuals is used in this context; how come figure 2 has no in-text explanations? \n\nThere are algorithms and figure on page 5 without explanations and connections to the paper \n\n\n4 Empirical results & discussion\n\n\n\"We hand-select \" -> what about other connection", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper considers an interesting manipulation problem", "weaknesses": "- The motivation and justification for the considered problems seem to be lacking at the beginning (mostly presented with very little context and connections) \n\n- The model is very marginal compared to existing studies (i.e., the new component seems to assume non-similarities in gaming via a cost-scaling factor); the notations/paper contents are poorly written, the results have few explanations in terms of implications, and there are assumptions in the models that are hard to justify \n\n- The proposed approaches (i.e., casts as causal effect estimation) have little to no justifications\n\n- The synthetic experimental evaluation is based on a single dataset with manual tuning that has little to no explanations, which limits the generalizability of the observations", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer dPj8 for their detailed comments, which improved our work. First, we address the motivation for our approach and clarify conceptual questions. We then briefly discuss connections to past work. Then, we address concerns about generalizability and our assumptions. We will add these clarifications to our revision.\n\n**Motivation for causally-motivated/counterfactual-based approach.** Our main contribution is providing theoretical and empirical evidence that causal effect estimation is a well-motivated, practical avenue for ranking agents by their gaming parameter (i.e., propensity to game). To see why causal inference is needed, consider the following: we observe two insurance companies/healthcare providers, one of whom receives a much higher payout due to higher reported rates of a certain condition. That provider could be seeing sicker patients or they could be gaming and inflating reported diagnosis rates to secure the higher payout. Causal inference methods help us adjust for underlying patient health and compare these reported rates. If the two patient populations are similarly healthy, yet one provider receives a higher payout, it indicates potential gaming/fraud.\n\nConcretely, we use pairwise comparisons between agents to construct a ranking of agents by their gaming parameter. Theorem 1 shows that this pairwise comparison problem is equivalent to a causal effect estimation problem (i.e., estimating counterfactual decisions under counterfactual agents) under our assumptions. This yields the algorithm in Figure 4.\n\n**Why ranking?** Given a ranking, one could prioritize agents to screen/audit for gaming. One of our main results is demonstrating that directly predicting gaming is impossible without access to ground truth labels stating which agents are gaming. Proposition 1 shows that, absent stronger assumptions, we can only estimate a loose lower bound on the gaming parameter. To reinforce this result, we provide a two-agent example in Section 3.1 (L126-L132) showing how this lower bound may flip rankings with respect to the ground truth gaming parameter. While directly estimating tendency to game is impossible, we show that recovering a ranking is possible.\n\n**Other conceptual questions.** We provide clarifications about our approach not covered above. \n* **Q:** We group these questions about the gaming parameter $\\lambda_p$:\n    - \"We aim to identify agents most likely\" -> wait; is lambda_p unknown? why is this the right way to identify agents? \n    - \"agents with the highest propensity \" -> how do you measure this? \n    - \"scalar gaming parameter that scales costs \" -> the meaning of this is not clear; what is this parameter? it is not explained in texts\n* **A:** Yes, $\\lambda_p$ is an unknown scalar, and is how we define the propensity to game. One can think of it as a “penalty factor” for gaming. Lower values of $\\lambda_p$ indicate higher gaming propensity, since such $\\lambda_p$ down-weights the cost of gaming. We aim to identify agents who are most likely to game (lowest $\\lambda_p$). This is of interest in settings where one has limited resources to audit gaming. A ranking approach allows auditors to prioritize investigating a subset of agents in which gaming is more likely (e.g., the top-$K$ agents with the lowest gaming parameters).\n* **Q:** \"fraud/gaming labels\" -> what does this mean?\n* **A:** Fraud/gaming labels refer to ground-truth indicators that denote whether an agent is definitively gaming. With such labels, one could simply use supervised learning to detect gaming, but we assume no such labels in our setting.\n* **Q:** \"identifying such agents is difficult\" -> why? it is not clear to me how would you do this even if you know the utility\n* **A:** If we know the *value* of the utility function and the reward function (i.e., in cases when the reward model $R$ is exactly the model to be manipulated $f$, as in Section 2), then we can infer that cost = utility - reward. Then, all agents that incurred non-zero cost (due to manipulating their own features) gamed by definition. We will more clearly delineate this in the revision.\n* **Q:** \"obtain a more desirable outcome\" -> outcome is what here? what is the connection between ML models and outcomes?\n* **A:** The “desirable outcome” is obtaining higher payout (e.g., higher compensation for treating patients with more complicated conditions). Such an ML model ($f$ in Section 2.1) might take as input the conditions that a provider claimed to treat, and output a dollar payout amount (e.g., via the U.S. government’s CMS-HCC model [A1]).\n* **Q:** “Also, are the models already trained? or are you talking [about] during learning?”\n* **A:** We assume a fixed model. Agents are responding to that model by perturbing their own features according to a utility-maximization problem. We make no further assumptions on the model training/fitting procedure.\n* **Q:** \"to obtain a higher payout.\" -> why do they want to increase d_i? why can't they decrease?\n* **A:** Good question: since agents are utility-maximizing by assumption, decreasing $d_i$ can provably never maximize utility. We comment on this in Remark 1 (L106-107): note that costs are zero for $\\Delta_p(d^*_p) < d^*_p$ (Assumption 3), but the reward function is increasing; hence, decreasing $\\Delta_p(d^*_p)$ below ground truth reduces utility.\n* **Q:** \"partially identifiable\" -> what is the meaning of this?\n* **A:** Partial identifiability (e.g., Definition 18.1, [A2]) denotes that multiple parameter estimates are compatible with the data; e.g., a range of values of $\\lambda_p$ as in our Proposition 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMCu3tNmPh", "reviewer": "Reviewer_P2tW", "summary": "This work studies the problem of identifying agents who would likely game a given system. When the gaming parameters are unknown, the authors show that identifying these parameters requires strong assumptions. In contrast, they show an ordering of agents based on their ranking order is learnable from a dataset. They use this ranking to detect gaming in a Medicare application.", "questions": "Is there any general approach to using rankings as a subroutine to detect gaming? If so, under what conditions would this approach provably work?", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The authors generalize the study of strategic adaptation to a much more realistic setting and provide provable results for computing rankings, which can then be used to detect gaming. The application of Medicare is also interesting.", "weaknesses": "Although the framework is interesting, it is unclear in general how to use rankings to detect gaming, which is the motivator of the study. A general provable approach to using rankings for gaming, even under some conditions, would improve the applicability of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer P2tW for their comments. We appreciate that the reviewer found the application area of U.S. Medicare to be interesting.\n\n**Can we actually detect gaming/use it as a subroutine?** Good question — to that end, our Proposition 1 demonstrates that definitive gaming detection is not possible without knowing ground-truth decisions. Intuitively, if the ground truth “correct” decisions were known/easy to collect, gaming detection would be trivial: we could simply check all observed decisions against ground truth in such cases. In summary:\n* **No framework under our assumptions can answer (via Proposition 1):** “Is Agent 1 gaming their decisions?”\n* **Our framework can answer (via Theorem 1 & Corollary 1):** “Is Agent 1 gaming their decisions more/less than Agent 2?”\nUnder our assumptions, to enable definitive gaming detection, one would need to assume that ground truth is fully observed, which is likely too strong of an assumption (i.e., everyone is subjected to an accurate audit, or is honest about whether they gamed). As-is, we envision that our ranking-oriented approach would serve as an initial flag for auditors to prioritize which agents to investigate. So, one could think of our ranking-based gaming detection approach could be a “subroutine” in a policy/procedural sense (albeit not in a purely algorithmic sense). \n\nHowever, the reviewer’s comment motivated us to revisit Proposition 1 and explore whether tighter versions of this bound could assist in gaming detection. We found a simple extension of our existing results that could allow for partial gaming detection. Let’s assume, for sake of exposition, that we can evaluate $R’$ and $c’$, similarly to the two-agent example of Section 3.1. This means we can evaluate the lower-bound of Proposition 1. Then, a partial gaming detection approach might take on two steps: \n1. Define a threshold $\\lambda^*$ such that plans with $\\lambda_p$ lower-bound greater than $\\lambda^*$ are definitively not gaming\n2. Estimate upper and lower bounds on $d^*_p$ to yield tighter lower/upper bounds on $\\lambda_p$; plans with $\\lambda_p$ upper-bound less than $\\lambda^*$ are definitively gaming.\n\nSuch a threshold might be defined via some tolerance for deviation from ground truth; *i.e.*, we can define gaming as $\\Delta_p(d^*_p) - d^*_p \\geq \\varepsilon$ for some $\\varepsilon > 0$, which allows agents to have an error rate of $\\varepsilon$ without being considered to be gaming. Based on $\\varepsilon$, we can then compute a “threshold” value of $\\lambda^*$ that demarcates gaming from non-gaming agents. For the two-agent example in Section 3.1 where $R(x) = x$ and $c(x) = x^2$, this would yield $\\lambda^* = \\frac{1}{2\\varepsilon}$. \n\nThen, we can combine the $\\lambda^*$ threshold with a tighter version of the bounds of Proposition 1. Suppose that we can claim $d^p \\in [\\underline{d}^*_p, \\bar{d}^*_p]$. Then, revisiting Eq. 8 (Appendix B.1, pg. 14) of our proof for Proposition 1, we’d reach the following:\n\n$$\\lambda_p \\in \\left( \\frac{R’(\\Delta(d^*_p))}{c’(\\Delta(d^*_p), \\bar{d}^*_p)}, \\frac{R’(\\Delta(d^*_p))}{c’(\\Delta(d^*_p) - \\underline{d}^*_p)} \\right).$$\n\nFor clarity, let’s abbreviate the terms in these bounds as $\\underline{b}$ and $\\bar{b}$ for the lower and upper bounds, respectively, such that $\\lambda_p \\in [\\underline{b}, \\bar{b}]$. Since we assumed $R’$ and $c’$ can be computed, and we assume estimates of $\\bar{d}^*_p$ and $\\underline{d}^*_p$ are available, we can calculate $\\underline{b}$ and $\\bar{b}$ directly. Thus, we would conclude:\n* If $\\bar{b} \\leq \\lambda^*$, then the agent is gaming (where $\\lambda^*$ depends on the error tolerance $\\varepsilon$), and\n* If $\\underline{b} \\geq \\lambda^*$, the agent cannot possibly be gaming.\n\nAlthough such a method would be unable to detect or rule out gaming in agents for which $\\lambda^* \\in [\\underline{b}, \\bar{b}]$, it could provide more definitive gaming detection results for a subset of agents. \n\nNote that this idea is preliminary — we have not had the opportunity to empirically validate this or identify similar approaches in the literature. This method also introduces two previously unused assumptions:\n1. One can compute $R’$ and $c’$\n2. One can estimate lower and upper bounds on $d^*_p$.\n\nWhile prevalence estimation via a random sample could yield confidence interval-based bounds on $d^*_p$, and $R’$ can be calculated if we know the model to be manipulated (e.g., the publicly-available CMS-HCC model used by U.S. Medicare), evaluating $c’$ may be unrealistic since it can be any strictly-convex function. Although testing and proposing such a new approach is likely out of scope for our revision, we will refine this argument and comment on this direction in Appendix B.1 as a potential avenue of future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sbm3QTxZqK", "reviewer": "Reviewer_Drhf", "summary": "The paper studied how to identify and rank agents who strategically manipulate their inputs to game machine learning models in multi-agent settings. A causally-motivated approach was proposed to address this challenge.", "questions": "Please find my first question in the above part. Here's another question. \n\nAfter the proposed approach is adopted and the player gaming the most is identified, how can we find our if the identification is correct? That is to say, how to validate your approach.", "limitations": "The authors have appropriately discussed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is an interesting follow-up to [1]. Provided that the proposed mythology is solid, I can foresee many scenarios where it can be applied. Especially, I feel that it may be applied to analyze college admission mechanisms.\n\n[1] Hardt, Moritz, et al. \"Strategic classification.\" Proceedings of the 2016 ACM conference on innovations in theoretical computer science. 2016.", "weaknesses": "The paper assumes players to be almost perfectly rational, i.e., they have a strong motivation to maximize utility. In reality, however, people are often only bounded rational. If so, that is, if some players do not particularly care about their utility, how would the accuracy of your approach be affected?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Drhf for their comments. We appreciate that the reviewer found our approach to be practical, and applicable to other real-world problems such as college admissions mechanisms.\n\n**Real-world validation.** Great point — real-world validation is an inherent limitation of the synthetic data setting indeed. To mitigate this issue, we did aim to verify that our rankings line up with known drivers of upcoding, such as the prevalence of private healthcare providers in a state as studied in [C1, C2] (Section 5.2; also see Table 1, pg. 9). In practice, real-world validation of causally-motivated approaches, including ours, is difficult, but this limitation is shared by many works in causal inference [C3, C4] that also opt for synthetic/semi-synthetic evaluations [C5] to verify the theoretical claims. \n\nFurthermore, we envision our ranking-oriented approach as an initial flag for auditors to subsequently investigate and confirm/rule out gaming.  A purely algorithmic validation grounded in the observed data is likely infeasible under our problem assumptions as per our Proposition 1, since any uncertainty in the correct ground truth decisions results in uncertainty in the gaming parameter.\n\n**Robustness to the rational actor assumption.** This is an insightful point. Indeed, while rationality is a standard assumption, and much game-theoretic literature shares this limitation, we will more explicitly highlight the assumption that agents are rational as a limitation. However, we wanted to highlight that our focus on rankings of gaming propensity rather than point estimates of the gaming parameter could afford some robustness via Proposition 2 (Section 3.2, pg. 5).\n\nIntuitively, violations of the rational actor assumption could simply be another source of noise; then, if such violations plus estimation error are too small to flip any pair of rankings with respect to the ground truth, then our estimated gaming rankings will still match the ground-truth rankings. Informally stated:\n\nLet $\\Delta_p(d^*_p)$ be the optimal utility-maximizing response, and let $\\tilde{\\Delta}_p(d^*_p)$ be the observed response. If $|\\Delta_p(d^*_p) - \\tilde{\\Delta}_p(d^*_p)| \\leq \\varepsilon_1$ and $\\sup |\\tau - \\hat{\\tau}| \\leq \\varepsilon_2$ for some $\\varepsilon_1, \\varepsilon_2 > 0$, then no rankings where the ground-truth $|\\hat{\\tau}(p, p’)| > \\varepsilon_1 + \\varepsilon_2$ will be flipped. \n\nNote that this corollary requires stronger conditions than Proposition 2, which only accounted for estimation error in the causal effect estimator $\\hat{\\tau}$. \n\nWe conclude by grounding our discussion in the motivating example of gaming the CMS-HCC model. Here, the notion of rationality is well-defined: rational agents game (i.e., perturb their reported diagnoses) such that profit is maximized; our utility function directly corresponds to a dollar amount. We speculate that deviations from the rationality assumption could occur if such agents are unable to carry out the utility-maximizing action (i.e., insufficient resources to, for example, schedule a home visit with a healthcare professional to generate diagnosis codes [C6]), or incorrectly estimate the utility-maximizing action (e.g., agents may incorrectly estimate costs). We leave the formalization of these rationality violations to future work. We will add this discussion of potential violations of the rational actor assumption and its implications on our ranking-based framework to Appendix B.4.\n\n**References**\\\n[C1] Silverman, Elaine, and Jonathan Skinner. \"Medicare upcoding and hospital ownership.\" Journal of health economics 23.2 (2004): 369-389.\\\n[C2] Silverman, Elaine, and Jonathan S. Skinner. \"Are for-profit hospitals really different? Medicare upcoding and market structure.\" (2001).\\\n[C3] Shi, Claudia, David Blei, and Victor Veitch. \"Adapting neural networks for the estimation of treatment effects.\" Advances in neural information processing systems 32 (2019).\\\n[C4] Louizos, Christos, et al. \"Causal effect inference with deep latent-variable models.\" Advances in neural information processing systems 30 (2017).\\\n[C5] Hill, Jennifer L. \"Bayesian nonparametric modeling for causal inference.\" Journal of Computational and Graphical Statistics 20.1 (2011): 217-240.\\\n[C6] Weaver, C., et al. “Insurers Pocketed $50 Billion From Medicare for Diseases No Doctor Treated.” The Wall Street Journal, 8 July 2024. https://www.wsj.com/health/healthcare/medicare-health-insurance-diagnosis-payments-b4d99a5d", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful comments, which helped improve our work. Reviewers positively commented on the interestingness of our problem setting [R1, R3], as well as the practicality and realism of our proposed methodology [R2/R3]. We respond to requests for clarification from reviewers individually.

Meta-review: Two of the reviewers were relatively positive, acknowledging the interesting and novel motivation of the paper, with potential practical applications. 

The main concerns have to do with the assumptions and the fact that the dataset is synthetic. After looking at the paper myself, I believe said assumptions/dataset are in line with a large amount of the strategic classification literature, and that these issues are not paper-breaking. 

As such, I recommend the paper to be accepted, however with low confidence and with the need for a SAC discussion. The meta-review also accounts for the fact this paper only received 3 out of 4 requested reviews.
Decision: Accept (poster)