Counterfactual
Inference
SUSAN ATHEYSTANFORD UNIVERSITY

Stability of Black‐Box ML

Artificial Intelligence/Machine Learning
Desired Properties for Applications
DESIRED PROPERTIESCAUSAL INFERENCE FRAMEWORKInterpretability

Goal: learn model of how the world works

Stability/Robustness
Transferability
Fairness/Non‐discrimination“Human‐like” AI◦ Reasonable decisions in never‐experienced situations◦ Impact of interventions can be context‐specific◦ Model maps contexts and interventions to outcomes◦ Formal language to separate out correlates and causes

Ideal causal model is by definition stable, interpretable
Transferability: straightforward for new context dist’n
Fairness: Many aspects of discrimination relate tocorrelation v. causation◦ Performance may depend on physical and mental ability,
psychological factors (e.g. risk taking)◦ Gender and race may be correlated with factors that shiftthese distributions, relatively limited direct causal effects

Artificial Intelligence/Machine Learning
Desired Properties for Applications
DESIRED PROPERTIESCAUSAL INFERENCE FRAMEWORKInterpretability

Goal: learn model of how the world works

Stability/Robustness
Transferability◦ Impact of interventions can be context‐specific◦ Model maps contexts and interventions to outcomes◦ Formal language to separate out correlates and causes

In practice, challenges remain,
to: stable, interpretable
Ideal causal e.g.
modeldueis by definition

Fairness/Non‐discrimination

Transferability: straightforward for new context dist’n

Lack of quasi‐experimental
dataMany
for aspects
estimation;
Fairness:of discrimination relate tocorrelation
v. insufficientcausation
Unobserved contexts/confoundersordata◦ Performance may depend on physical and mental ability,
factors (e.g. risk taking)to control for observedpsychologicalconfounders;◦ Gender and race may be correlated with factors that shift
Analyst’s lack of knowledge
model
these about
distributions,relatively limited direct causal effects

Artificial Intelligence andCounterfactual Estimation
Artificial intelligence◦ Select among alternative choices◦ Explicit or implicit model of payoffs fromalternatives◦ Learn from past data
◦ Initial stages of learning have limited data
◦ Inside the AI is a statistician performing
counterfactual reasoning◦ Statistician should use best performing
techniques (efficiency, bias)

Simple example: contextual bandit

Estimation is challenging: Contextual Bandit example●

Inherent bias in estimation due to adaptive assignment of contexts to arms.
○ context assigned to arm with highest reward sample or confidence bound○ creates systematically unbalanced data

What was the impact of the policy?

Counterfactual
Inference
Approaches◦ Minimum wage, training program, classsize change, etc.

Did the advertising campaign work?
What was the ROI?
Do get‐out‐the vote campaigns work?

“Programevaluation”,“treatment effectestimation”

What is an optimal policy assigningworkers to training programs?

Goal: estimate the impact of interventions ortreatment assignment policies◦ Low dimensional intervention

Counterfactual
Inference
Approaches

Estimands◦ Average effect◦ Heterogeneous effects◦ Optimal policy

Confidence intervals

Designs that enable identification andestimation of these effects“Programevaluation”,“treatment effectestimation”◦ (Alternative treatments observed in historical data inrelevant contexts)◦ Randomized experiments◦ “Natural” experiments (Unconf., IV)◦ Regression discontinuity◦ Difference‐in‐difference◦ Longitudinal data
◦ Randomized and natural experiments in social
network/settings w/ interferenceTreatment Effect
Estimation:
Designs
Regression Discontinuity Design
Mbiti & Lucas (2013) estimate
impact of secondary school
quality on student achievementin Kenya.
Discontinuity: cut‐off on the
primary exit exam required to
get into better secondary
schoolsTreatment Effect
Estimation:
Designs
Difference‐in‐Difference Designs
Athey and Stern (2002) look at
the impact of Enhanced 911(automated address lookup) on
health outcomes for cardiac
patients
Counties adopt at different
times; estimate time trendusing other counties todetermine counterfactual
outcomes in the absence ofadoption

Counterfactual
Inference
Approaches“Structural
estimation”,“Generative
Models” &Counterfactuals

What would happen to firm demand if
price increases?
What would happen to prices,
consumption, consumer welfare, andfirm profits if two firms merge?
What would happen to platformrevenue, advertiser profits andconsumer welfare if Google switchedfrom a generalized second priceauction to a Vickrey auction?

Goal: estimate impact on welfare/profits of participants inalternative counterfactual regimes

Counterfactual
Inference
Approaches◦ Counterfactual regimes may not have ever been observed in
relevant contexts◦ Need behavioral model of participants

Still need designs that enable identification and estimation,
now of preference parameters◦ E.g. need to see changes in prices to understand price sensitivity“Structural
estimation”,“Generative
Models” &Counterfactuals

Use “revealed preference” to uncoverpreference parameters
Rely on behavioral model to estimate behavior
in different circumstances◦ Also may need to specify equilibrium selection

Dynamic structural models◦ Learn about value function from agent choices indifferent states◦ See Igami (2018) who relates to AI

Advertiser Profit Maximization Example

Counterfactual
Inference
Approaches◦ Bidder in search advertising auctions has value‐per‐click v◦ Q(b) is the share of available ad clicks per search from bidding b per
click; upward sloping◦ Bidder profit per search:𝑄 𝑏 ⋅ 𝑣
◦ Bidder first order condition:

𝑣

𝑏

𝑏

𝑄 𝑏
𝑄′ 𝑏

Inferring preferences (value per click) from data

“Structural
estimation”,“Generative
Models” &Counterfactuals◦ Analyst estimates Q() from historical log data
◦ For each advertiser, can infer the value v that rationalizes bid (satisfies
FOC)

Counterfactuals◦ With knowledge of advertiser values and behavioral model, can solve for
new equilibria◦ Changing auction format◦ Changing quality scores

See: Athey and Nekipelov (2012)

Single Agent Decision Problem◦ Rust (1987) studies problem of a decision‐maker replacing bus engines◦ Analogous to a grand master playing chessCounterfactual
Inference
Approaches◦ Agent maximizes discounted sum of profits◦ Using principles of dynamic programming, Bellman equation is:𝑉 𝑠max 𝜋 𝑠 , 𝑠; 𝜃, 𝜖
∈

𝛿𝑉 𝑠′

◦ Policy function :𝜎 𝑠; 𝜃

arg max 𝜋 𝑠 , 𝑠; 𝜃, 𝜖
∈

𝛿𝑉 𝑠′

◦ Assume stochastic shock 𝜖 to flow profits

Dynamic Structural
Estimation
InverseReinforcement
Learning

Solution: Nested fixed point◦ Outer loop: Optimize likelihood function for 𝜃, where data are(state,action) pairs and model predicts optimal actions as function of 𝜃
◦ Inner loop:◦ Given 𝜃 , solve for value function by iterating over Bellman’s equation◦ Evaluate policy function given value function, and evaluate likelihood

See: Igami (2018) who develops relationship between this andBonanza algorithm; also analysis of AlphaGo algorithm relative to
Hotz and Miller (1993)

What can we learn from decades of methodological and empirical workin economics, that is relevant for AI?

Counterfactual
Inference
Approaches◦ Applications to human or firm behavior are challenging◦ Conceptual framework has been clear from 80s and 90s◦ Big problem: not enough training data, and not enough knowledge about
game payoffs to create artificial training data
◦ Economics has some insights to help in data‐poor environments…
◦◦◦◦

Use as much structure as is known, carefully examine functional forms for how they extrapolate
Think about independence assumptions and biases that might arise from diff’t training data
Take behavioral models seriously to draw better inference from agent behavior
See Igami (2018) for some more discussion

How can recent advances in AI help solve economic problems?

Dynamic Structural
Estimation
InverseReinforcement
Learning◦ New algorithms of past 10‐15 years in ML/AI focus on computational
performance and problems with large state spaces◦ Coupled with games that can be played by computers with large number of repetitions,generating very large datasets◦ The rules are clear, so possible to test different strategies against one another◦ The analyst knows the mapping from final state to payoffs, just doesn’t know the value function
at intermediate states◦ In economic problems◦ Computational advances definitely help in problems with large state spaces…
◦ But the analyst doesn’t know the per‐period payoff function, and thus doesn’t know enoughabout the game to simulate play and know what the final payoffs are.
◦ Can only do that given parameter values.

Counterfactual
Inference
Approaches“Causal discovery”,“Learning the causal
graph”

Goal: uncover the causal structure of asystem◦ Many observed variables◦ Analyst believes that there is an underlying
structure where some variables are causes ofothers, e.g. a physical stimulus leads to biologicalresponses

Focus on ways to test for causal
relationships
Applications◦ Understanding software systems◦ Biological systems

Counterfactual
Inference
Approaches

Recently, literatures
have started coming
togetherMultiple literatures on causality within
economics, statistics, and computer science
Different ways to represent equivalentconcepts
Common themes: very important to haveformal language to represent concepts
Recent literatures: Bring causal reasoning,statistical theory and modern machinelearning algorithms together to solve
important problems

Preview of Themes
Causal inference v. supervised learning

Insights from statistics/econometrics◦ Supervised learning: can evaluate in test set inmodel‐free way◦ Causal inference◦ Consider identification, then estimation◦ Parameter estimation‐parameter not observed in test set◦ Change objective function, e.g. consistent parameter
estimation◦ Can estimate objective (MSE of parameter), but often
requires maintained assumptions◦ Often sampling variation matters even in large data sets
◦ Requires theoretical assumptions and domain knowledge◦ Tune for counterfactuals: distinct from tuning for fit, also
different counterfactuals select different models◦ Could you solve problem with infinite data?
◦ Design‐based approach◦ Estimation: scaled up with many experiments◦ Regularization induces omitted variable bias
◦ Omitted variables challenge causal inference,
interpretability, fairness◦ Semi‐parametric efficiency theory can be
helpful, brings insights not commonly exploited
in ML◦ Cross‐fitting/out of bag estimation of nuisance parameters◦ Orthogonal moments/double robustness◦ Use best possible statistician inside bandits/AI agents◦ Exploit structure of problem carefully for bettercounterfactual predictions◦ Black‐box algorithms reserved for nuisance parametersEstimating ATE under
Unconfoundedness
SOLVING CORRELATION V. CAUSALITY BY CONTROLLING FORCONFOUNDERSSetting
Only observational data is available
Analyst has access to data that is sufficient for the part of the information
used to assign units to treatments that is related to potential outcomes
Analyst doesn’t know exact assignment rule and there was somerandomness in assignment
Conditional on observables, we have random assignment
Lots of small randomized experiments
Application: logged tech company data, contextual bandit data

Example: Effect of an Online Ad
Ads are targeted using cookies
User sees car ads because advertiser knows that user visited car
review websites
Cannot simply relate purchases for users who saw an ad and those
who did not:◦ Interest in cars is unobserved confounder

Analyst can see the history of websites visited by user
◦ This is the main source of information for advertiser about user interests

SetupAssume unconfoundedness/ignorability:◦

Assume overlap of the propensity score:
Then Rubin shows:◦ Sufficient to control for propensity score:◦◦ If control for X well, can estimate ATE◦.

Intuition for Most Popular Methods
Control group and treatment group are different in terms of observables
Need to predict cf outcomes for treatment group if they had not been treated
Weighting/Matching: Since assignment is random conditional on X, solve problem byreweighting control group to look like treatment group in terms of distribution of X
◦ P.S. weighting/matching: need to estimate p.s., cannot perfectly balance in highdimensions
Outcome models: Build a model of Y|X=x for the control group, and use the model topredict outcomes for x’s in treatment group◦ If your model is wrong, you will predict incorrectly
Doubly robust: methods that work if either p.s. model OR model Y|X=x is correct

Y
Treatedobservations havehigher X’s onaverageX

Y

Reweighting control
observations withhigh X’s
adjusts fordifferenceXY
Outcome modelingadjusts fordifferences in X

XReweighting control
observations with highX’s
AND using outcome
modeling is doubly
robustY

With correct
reweighting, don’t
need to adjust
outcomes
With outcome
adjustments, don’t
need to reweightXUsing Supervised ML
to Estimate ATE Under
Unconfoundedness
Method I:
Propensity scoreweighting or KNN on
propensity score◦ LASSO to estimate propensity score; e.g.
McCaffrey et al. (2004); Hill, Weiss, Zhai (2011)

Using Supervised ML
to Estimate ATE
Under
Unconfoundedness

Method II:
Regression
adjustment◦ Belloni, Chernozukov, Hansen (2014):◦ LASSO of W~X; Y~X◦ Regress Y~W, union selected X◦ Sacrifice predictive power (for Y) for
causal effect of W on Y
◦ Contrast w/ off‐the‐shelf supervised
learning◦ Off‐the‐shelf LASSO Y~X,W does not select all X’s that are
confounders◦ Omitting confounders leads to biased estimates◦ Prioritize getting the answer right about treatment
effects

Using Supervised ML
to Estimate ATE
Under
Unconfoundedness

Method III:
Estimate CATE
and take averages◦ Hill (2011) uses BART (Chipman, 2008) or otherflexible method to estimate
𝜇 𝑥; 𝑤

𝐸 𝑌 |𝑋

𝑥, 𝑊𝑤◦ Estimate ATE as E 𝜇̂ 𝑋 ; 1
𝜇̂ 𝑋 ; 0◦ See further papers by Hill and coauthors◦ Performs well in contests, can use propensity
adjustments in estimating conditional meanfunction◦ Performance relies on doing a good job
estimating this outcome model—depends on
DGP, signal‐to‐noise

Using Supervised ML
to Estimate ATE
Under
Unconfoundedness

Method IV:
Doublerobust/double
machine learning◦ Cross‐fitted augmented inverse propensity scores◦ These are the efficient scores (see literature on semi‐parametricefficiency)◦ Orthogonal moments◦ Cross‐fitted nuisance parameters: 𝜏̂ 𝑋 , 𝑒̂ 𝑋 , 𝜇̂ 𝑋 ; 𝑊 , e.g.
OOB random forest◦ Score given by
◦Γ

𝜏̂

𝑋
̂

̂
̂

𝑌 𝜇

𝑋 ;𝑊

◦ ATE is average of Γ
◦ DR: consistent estimates if either propensity score OR
outcome correct◦ Can get 𝑛 convergence even if nuisance parametersconverge more slowly, at rate 𝑛 / , which helps in highdimensions

Using Supervised ML
to Estimate ATE
Under
Unconfoundedness

Method V:
Residual
Balancing◦ Athey, Imbens and Wager (JRSS‐B, 2018)◦ Avoids assuming a sparse model of W~X,
thus allowing applications with complex
assignment◦ Not just slow convergence of assignment model—
assignment model does not need to be estimated at all!

◦ LASSO Y~X◦ Solve a programming problem to findweights that minimize difference in X
between groups◦ Maintains the orthogonal moment form

Residual
BalancingResidual
BalancingResidual
Balancing

Instrumental Variables

What if unconfoundedness fails?
Alternate assumption: there exists an instrumental variable Zi that iscorrelated with Wi (“relevance”) and where:
𝑌 0 , 𝑌 1 ⊥ 𝑍 |𝑋
Treatment Wi

Instrument Zi

Outcome Yi

Military service

Draft Lottery Number

Earnings

Price

Fuel cost

Sales

Having 3 or more kids

First 2 kids same sex

Mom’s wages

Education

Quarter of birth

Wage

Taking a drug

Assigned to treatment group

HealthSeeing an ad

Assigned to group of users
advertiser bids on in experiment

Purchases at advertiser’s
web siteInstrumental Variables:
Binary Experiment Case

Compliers
Always‐Takers
Never‐Takers
Defiers

Assigned toTreatment
TreatedTreatedNot treated
Not treated

Not Assigned toTreatment
Not treatedTreatedNot treatedTreatedWhy not look at who was actually treated?
◦ Those who complied or defied were probably not
randomDifferent
Estimands

Intention‐to‐treat (ITT)◦ Compare average outcomes of those assigned totreatment with those assigned to control◦ This may be interesting object if compliance will besimilar when you actually implement the treatment,
e.g. recommend patients for a drug
Local Average Treatment Effect (effect of treatment oncompliers)◦ Calculated as ITT/Pr(treat|assigned
treatment)=ITT/Pr(Wi=1|Zi=1)◦ This clearly works if you can’t get the treatment without
being assigned to treatment group (no always‐takers, nodefiers)◦ This also works as long as there are no defiers◦ LATE is always larger than ITT

Special case: Wi,Zi both binary
Relevance: Zi is correlated with Wi
Exclusion:

Local Average
Treatment
Effects

Monotonicity: No defiers
Then the LATE is:Special case: Wi,Zi both binary

Local Average
Treatment
Effects:
Including
Covariates

Relevance: Zi is correlated with Wi
Exclusion: 𝑌 0 , 𝑌 1⊥ 𝑍 |𝑋

Monotonicity: No defiers
Then the LATE conditional on Xi =x is:
𝔼𝑌 𝑋
𝔼𝑊 𝑋

𝑥, 𝑍
𝑥, 𝑍

1
1

𝔼𝑌 𝑋
𝔼𝑊 𝑋

𝑥, 𝑍
𝑥, 𝑍

1
1

Two‐stage least squares approach𝛽𝑊 𝛽 𝑋𝑌 𝛽
𝑊 𝛾
𝛾 𝑍 𝛾 𝑋IV
Approaches:
Including
Covariates𝜀𝜀Chernozhukov et al:◦ Use LASSO to select which X’s to include and partial
them out◦ If there are many instruments, use LASSO to construct
the optimal instrument, which is the predicted value ofWi◦ Formally, estimate first stage using Post‐LASSO◦ In second stage, run 2SLS using predicted value oftreatment as instrument◦ Theorem: if model is sparse and instruments arestrong, estimator is semi‐parametrically efficient
Note: doesn’t consider observable or unobservable
heterogeneity of treatment effects
See also Peysakhovich & Eckles (2018)

Two‐stage least squares approach𝛽𝑊 𝛽𝑋 𝜀𝑌 𝛽
𝑊 𝛾
𝛾 𝑍 𝛾 𝑋 𝜀IV
Approaches:
Including
Covariates

Chernozhukov et al example:◦ Angrist and Krueger quarter of birth paper◦ Instruments: quarter of birth, and interactions with
controls◦ Using few instruments gives large standard errors

Clicks as a Fraction of Top Position 1
ClicksUser Model of Clicks:
Results from Historical
Experiments(Athey, 2010)

Search phrase:

OLS Regression:

Model:

iphoneOLS

IV

viagraOLS

IV◦ Features: advertiser effects andposition effects
IV Regression

Top Position 2
Top Position 3

0.66
0.40

0.67
0.55

0.28
0.14

0.66
0.15

◦ Project position indicators on A/B
testid’s.
◦ Regress clicks on predicted positionindicators.
Estimates show smaller position impact
than OLS, as expected.

Side Position 1

0.04

0.39

0.04

0.13

Position discounts important for
disentangling advertiser quality scores

IV:
Heterogeneous
Treatment
Effects

What if we want to learn about conditional
average treatment effects (conditional onfeatures?)
For simplicity, assume treatment effects areconstant conditional on X.
Illustrate with two approaches:◦ Generalized random forests (Athey, Tibshirani,
and Wager, Annals of Statistics, 2018)◦ Asymptotic normality and confidence intervales◦ Deep Instrumental Variables (Taddy, Lewis,
Hartford, Leyton‐Brown (UBC))Then apply to optimal policy estimation◦ Athey and Wager (2016), Zhou, Athey andWager (2018)

Instrumental Variables (IV)

z

y

x
p

e

The exclusion structure implies

You can observe and estimate
to solve for structural gand
we have an inverse problem.
cf Newey+Powell 2003

∈

2SLS:

and

So you first regressso thaton then regresson

to recover .

∈Or nonparametric sieves where

and(Newey+Powell)or(BCK, Chen+Pouzo)
Also Darolles et al (2011) and Hall+Horowitz (2005) for kernel methods.
But this requires careful crafting and will not scale with

∈Instead, Deep IV targets the integral loss function directly
For discrete (or discretized) treatment• Fit distributions• Trainwith probability masses

to minimize

And you’ve turned IV into two generic machine learning tasks

Search Ads Application of Deep IV: Relative Click Rate

Heterogeneity across advertiser and searchGeneralized
Random Forests:
Tailored Forests as
Weighting
FunctionsGeneralized Random Forests• Athey, Tibshirani & Wager establish asymptotic normality ofparameter estimates, confidence intervals• Recommend orthogonalization• Software: GRF (on CRAN)

Local Linear Forests
Friedberg, Athey, Tibshirani, and Wager (2018)

Comparing Regression Forests to Local Linear
Forest: Adjusting for Large Leaves/Step Functions

Randomized Survey Experiment: Are you favor of “assistance to the poor” versus“welfare”
How does treatment effect (CATE) change with political leanings, income?
LLF has better MSE of treatment effectOptimal Policy
Estimation

Estimating Treatment Assignment
Policies
Scenario: Analyst has Observational Data
◦ Historical Logged Data
◦ Tech firm using contextual bandit or black box
algorithms◦ Logged data from electronic medical records◦ Historical data on worker training programs
and outcomes◦ Randomized Experiment with Noncompliance

Goal: Estimate Treatment Assignment
Policy◦ Minimize regret (v. oracle assignment)

Large Literature Spanning Multiple
Disciplines◦ Offline policy evaluation (e.g. Dudik et al,
2011, others…) versus efficient estimation ofbest policy from a set◦ Two actions vs. multiple actions vs. shifting
continuous treatment◦ Designs◦ Randomized experiments◦ Unconfoundedness with known (logged)
propensity scores◦ Unknown propensity scores◦ Instrumental Variables

max

2𝜋 𝑋∈

Alternative
Approaches to Policy
Evaluation/Estimation

Different authors have proposed using different scores in the
optimization problem

𝜏̂ 𝑋

CATE: ΓDesign:Unconfoundedness
Literature focuses on
this case

1 Γ

IPW: Γ
Cross‐fit AIPW: Γ

𝜏̂

̂

𝑋

;

̂

⋅𝑌

;

⋅ 𝑌

𝜇̂

𝑋 ,𝑊

Multi‐Arm Generalization(Zhou, Athey and Wager, 2018)

Instrumental Variables
Application
Build on Chernozhukov et al(2018) – “CEINR”
Framework for estimating
treatment effects withorthogonal moments

Example: Voter mobilization
Treatment: Calling voter
Randomized Experiment:
Voter list (not all have #s)
Outcome: Did citizen vote
Question: Policy for whichpeople should be called

General Approach: Choose Policy to Assign
Treatment to Units with High Scores
max
∈

2𝜋 𝑋

1 Γ

Key insights:• Scores should be orthogonalized/doubly robust• Use cross‐fitting/out‐of‐bag for nuisanceparameters• Can solve as weighted classification problem (e.g.
Beygelzimer et al; Zhou, Athey & Wager propose
tree search algorithm)

Contextual Bandits

See John Langford, Alekh Agarwal, and coauthors for surveys, tutorials,
etc…
Online learning of treatment assignment policies

Contextual
Bandits

Issues with contexts:◦ No context, small finite set of contexts: bandit for each context◦ With many contexts, we need to solve a hard estimation problem (aswe’ve been discussing)◦ Best performance: state of the art causal inference methods
Most contextual bandit theory◦ Assumes outcome model correct (no need for double robust, double
robust can add variance)
Proposal in Dimakopoulou, Zhou, Athey and Imbens, AAAI 2019
◦ Use double robust estimation, shows regret bounds match existing
literature
Many open questions from causal inference perspective◦ Establish improvement from double robust methods with
misspecification

Contextual bandits●

Arm space A with |A| = K arms.

●

Context space X with dimensionality d.

●Environment generates context and rewards (xt, rt) ~ D, rt = (rt(1), …, rt(K))○ Αgent selects action at and observes reward only for the chosen arm, rt(at)●Goal: assign each context x to the arm with the maximum expected reward○ μa(x) = E[rt(a) | xt = x] = f (x; θa) is a function of x, parameters θa are unknown…

●

Balance exploration (information gained for arms we are uncertain about) with
exploitation (improvement in regret from assigning context to the arm viewed best).

Examples●

Content recommendation in web services○ arms: recommendations○ context: user profile and history of interactions○ reward: user engagement and user lifetime value●

Online education platforms○ arm: teaching method○ context: characteristics of a student○ reward: student’s scores●

Survey experiments○ arm: what information or persuasion to use○ context: respondent’s demographics, beliefs, characteristics○ reward: responseLinear contextual bandits●

Build parametric model for expected reward of each arm given covariates○ linear bandit: E[rt(a) | xt = x] = θaT x for all a●

LinUCB and LinTS have near-optimal regret bounds (requires correct specification).

●

LinUCB○ use ridge regression to get an estimate of θa and a confidence bound of θaT x○ assign context x to arm with highest confidence bound

●

LinTS
○ start with a Gaussian prior on parameter θa○ use Bayesian ridge regression to obtain the posterior of θa○ sample parameters for each arm and assign x to arm with highest sampled rewardEstimation is challenging●

Inherent bias to the estimation due to the adaptive assignment of contexts to arms.
○ context assigned to arm with highest reward sample or confidence bound○ creates systematically unbalanced data
○ complete randomization gives unbiased estimates, but this defeats the purpose

Estimation is challenging●

Inherent bias to the estimation due to the adaptive assignment of contexts to arms.
○ context assigned to arm with highest reward sample or confidence bound○ creates systematically unbalanced data
○ complete randomization gives unbiased estimates, but this defeats the purpose●

Aggravating sources of bias in practice○ model misspecification■ true generative model and functional form used by the learner differ○ covariate shift■ early adopters of an online course have different features than late adoptersBalanced contextual bandits●

Dimakopoulou, Zhou, Athey, Imbens (AAAI, 2019)●

Propensity score pt(at) the probability that context xt is assigned to arm at●

Balanced LinTS (BLTS) and balanced LinUCB (BLUCB)○ Weight each observation (xt, at, rt) by 1/pt(at)○ Use the weighted observations in ridge regression.

●

For Thompson sampling, propensity is known.
○●

Note: Formal Bayesian justification for weighting in Thompson sampling is not clear, similar to justification forusing the propensity score in observational studies.

For UCB, propensity is estimated (e.g. via logistic regression).
○

Note: The notion of “propensity” in UCB at a given time is contrived (either 0 or 1). Treating the arrival of a context
as random, we use the context’s ex ante propensity.

Why balancing helps?
●

In practice, balancing can help with covariate shift and model mis-specification.

●

Doubly-robust nature of of inverse propensity score weighted regression○ accurate value estimates either with a well-specified model of rewards or with awell-specified model of arm assignment policy.

●

Contextual bandits:○ generally, do not have a well-specified model of rewards○ even if they do, it cannot be estimated well with small datasets in the beginning○ but, they control arm assignment policy conditional on observed context○ hence, access to accurate propensities results in more accurate value estimates

State of the art regret guarantees, but better performance in practice.

A simple synthetic example
Well-specified reward model(include both linear and quadratic terms in context)

Expected reward of the arms
conditional on the context x = (x0, x1) ~ N(0, I)

Initial contexts come from a subset of thecovariate space around the global optima.

Mis-specified reward model(include only linear terms in context)

Experiments on 300 classification datasets●

A classification dataset can be turned into acontextual bandit○ labels → arms,○ features → context,○ accuracy → reward○ reveal only accuracy of chosen label●

300 datasets from Open Media LibraryStructural Models

Themes for ML + Structural Models
FROM STRUCTURAL LITERATUREFROM ML LITERATURE

Attention to identification, estimation using “good” exogenous
variation in data

More efficient computational tools◦ Supermarket application: Tues‐Wed comparisons when prices
change Tues night; attention to holiday purchases or highseasonality items

Adding sensible structure improves performance◦ Required for never‐seen counterfactuals◦ Increased efficiency for sparse data (e.g. longitudinal data)◦ E.g. stochastic gradient descent◦ E.g. variational inference

Dimension reduction for longitudinal data
◦ E.g. matrix factorization

Nature of structure◦ Learning underlying preferences that generalize to new situations◦ Incorporating nature of choice problem◦ Many domains have established setups that perform well in data‐
poor environments

Tune models for counterfactual performance◦ Focus on parameters of interest, not fit◦ Get a different answer depending on CF of interest

Formal model tuning on validation set◦ But with different objectives, e.g. counterfactual

Discrete Choice Models
User u, product i, time t

If sufficient exogenous variation in prices, canidentify & estimate distribution of 𝛼 .
With longitudinal data and sufficient price variation,
can estimate 𝛼 for each user. (Often Bayesian.)If
i.i.d. Type I extreme value,then∑

Revealed preference (users’ choices) allow us tounderstand welfare.
◦ Can solve for a firm’s optimal price, optimal coupon◦ Understand the impact on firm profits (given cost
information) and consumer welfare.

Can evaluate the impact of a new productintroduction or the removal of a product from choiceset.
Dan McFadden (early 1970s): Counterfactual
estimates of extending BART in San Francisco area.

Combining Discrete Choice Models with
Modern Machine Learning….
Ruiz, Athey, and Blei (2017), Athey, Blei, Donnelly, and Ruiz (2018), Athey, Blei, Donnelly,
Ruiz and Schmidt (2018)
Bring in matrix factorization, and apply to shopping for many items (baskets,
restaurants)
Incorporate choice to not purchase
Two approaches to product interactions◦ Use information about product categories, assume products substitutes within categories◦ Do not use available information about categories, estimate subs/complements

Can analyze counterfactuals◦ Personalized coupons◦ Restaurants opening and closing

The Nested
Logit
Factorization
ModelThe Nested
Logit
Factorization
Model• Counterfactual inference in nested logit
models uses structure• Model specifies how user substitutes if choice
set changes, e.g. product out of stock

The Nested
Logit
Factorization
Model• Conditional on purchasing a single item in a
category, choice probabilities redistributed inproportion to probabilities of other items• Model makes counterfactual predictions
about what happens when prices change• Given price sensitivity for a given product, model
makes sensible predictions about how purchaseprobabilities for other products change when theprice of the given product changes

Computational
ApproachGoodness of Fit (Tuned for CF)
Weeks where another product in category changed prices

Validation of Structural Parameter Estimates
Compare Tuesday‐Wednesday change in price to Tuesday‐Wednesday change in demand, in test set
Break out results by how price‐sensitive (elastic) we have estimated consumers to bePersonalized Pricing
Matrix Factorization Approach Allows Accurate Personalization
How much profit can be made by giving a 30% off coupon for a single product to a targeted
selection of 30% of the shoppers in the store? Compare uniform randomization, demographic, orindividual targeting policies based on structural estimates

Conclusions
Causal inference is key to using machine learning and artificial intelligence to make decisions◦ This is a tautological statement: but at the same time, not fully appreciated

Artificial intelligence agents will improve if they are good statisticians
AI based on causal modeling has desirable properties (stability, fairness, robustness, transferability, ….)There is an enormous literature on theory and applications of causal inference in many settings and with many
approaches
The conceptual framework is well worked out for both static and dynamic settings
Structural models enable counterfactuals for never‐seen worlds
Machine learning algorithms can greatly improve practical performance, scalability
Challenges: data sufficiency, finding sufficient/useful variation in historical data
◦ Recent advances in computational methods in ML don’t help with this◦ But tech firms conducting lots of experiments, running bandits, and interacting with humans at large scale can greatly expand
ability to learn about causal effects!

References

Selected References:
Traditional “Program Evaluation” or Treatment Effect
EstimationBOOKS

SURVEY AND NONTECHNICAL PAPERS

Guido W Imbens and Donald B Rubin. Causal Inference in Statistics,
Social, and Biomedical Sciences. Cambridge University Press, 2015.

Guido Imbens and Jeffrey Wooldridge. Recent developments in theeconometrics of program evaluation. Journal of Economic
Literature, 47(1):5–86, 2009.

◦ Summarizes literature from stats/econometrics/biostatistics
perspective in pre‐machine learning era

Angrist and Pischke, 2008, Mostly Harmless Econometrics◦ Informal introduction to causal inference

Cunningham, Causal Inference: The Mixtape◦ Applied economics perspective; recent and accessible, andavailable free online
http://scunning.com/cunningham_mixtape.pdf

Pearl and MacKenzie, Book of Why?
◦ Recent and accessible

Stephen L Morgan and Christopher Winship. Counterfactuals andcausal inference. Cambridge University Press, 2014

Susan Athey and Guido Imbens. “The state of applied
econometrics causality and policy evaluation.” Journal of Economic
Perspectives, 2017.

Selected References:
Randomization Approach to Causal Inference
Neyman [1923/1990] is a classic paper, reprinted in Statistical Science.
Fisher [1935] is another classic reference.
General statistics texts: Wu and Hamada [2011], Cook and DeMets [2007], Cox and Reid [2000],
Hinkelman et al. [1996]
Athey and Imbens [2016a] is a survey focused on an economics audience.
Bruhn and McKenzie [2009], Morgan and Rubin [2015, 2012] discuss re‐randomization.
Middleton and Aronow [2015], Murray [1998] discuss clustered randomized experiments.
The relation to regression is discussed in Abadie et al. [2014], Lin [2013], Freedman [2008], Samii
and Aronow [2012].
Imbens and Menzel [2018] develop a version of the bootstrap focused on causal effects.

Selected References:
ATE Under Unconfoundedness
Rosenbaum and Rubin [1983]: Potential outcomes, theory of propensity score weighting
Imbens [2004] presents a survey.
Matching estimators: Abadie and Imbens [2006, 2008], Rubin and Thomas [1996].
Hahn [1998] derives the efficiency bound and proposes an efficient estimator.
Robins and Rotnitzky [1995], Robins et al. [1995]: Doubly robust methods.
Hirano et al. [2003]: Weighting estimators with the estimated propensity score.
Crump et al. [2009] discuss trimming to improve balance.
Yang et al. [2016], Imbens [2000], Hirano and Imbens [2004] discuss settings with treatments taking on more than
two values
Hotz et al. [2005] discuss the role of external validity.
Applications to the Lalonde data: LaLonde [1986], Dehejia and Wahba [1999], Heckman and Hotz [1989].
Athey and Imbens [2016, AER], Athey, Imbens, Pham, Wager [2017], Athey and Imbens [2018, JEP] discussrobustness and supplementary analysis

Selected References:
Instrumental Variables
Imbens and Angrist [1994], Angrist et al. [1996]: LATE
Imbens [2014] presents a general discussion for statisticians
Classic applications: Angrist [1990], Angrist and Krueger [1991].
Staiger and Stock [1997], Moreira [2003] discuss inference with weak instruments.
Chamberlain and Imbens [2004] discuss settings with many weak instruments

Selected References:
Regression Discontinuity Designs
Thistlewaite and Campbell [1960]: original reference.
Imbens and Lemieux [2008], Lee and Lemieux [2010], Van Der Klaauw [2008], Skovron and Titiunik [2015], Choiand Lee [2016]: theory
Hahn et al. [2001]: fuzzy regression discontinuity
Imbens and Kalyanaraman [2012], Calonico et al. [2014]: optimal bandwidth choices.
Gelman and Imbens [2018] discuss the pitfalls of using higher order polynomials.
Bertanha and Imbens [2014], Battistin and Rettore [2008], Dong and Lewbel [2015], Angrist and Rokkanen [2015],
Angrist [2004] discuss external validity of regression discontinuity designs.
Applications: Angrist and Lavy [1999], Black [1999], Lee et al. [2010], Van Der Klaauw [2002]
Regression kink designs: Card et al. [2015].
Recent work focuses on settings where instead of choosing a bandwidth directly optimal weights arecalculated: Kolesar and Rothe [2018], Imbens and Wager [2017], Armstrong and Kolesar [2018].

Selected References:
Differences‐in‐Differences, Synthetic Controls
Angrist and Krueger [2000]: General discussion
Applications: Ashenfelter and Card [1985], Eissa and Liebman [1996], Meyer et al. [1995], Card
[1990], Card and Krueger [1994]
Nonlinear version: Athey and Imbens [2006]
Synthetic control methods: Abadie and L’Hour [2016], Abadie et al. [2010, 2015], Abadie andGardeazabal [2003], Doudchenko and Imbens [2016], Xu [2015], Gobillon and Magnac [2013],
Ben‐Michael et al. [2018], Athey and Imbens [2018].
Links between the matrix completion literature and the causal panel data literature are given in
Athey, Bayati, Doudchenko, Imbens, Khosravi [2017].

Selected References:
Econometrics and ML
Prediction v. Estimation◦ Mullainathan, Sendhil, and Jann Spiess. "Machine learning: an applied econometric approach." Journalof Economic Perspectives 31.2 (2017): 87‐106.

Prediction policy◦ Kleinberg, Jon, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer. “Prediction policy problems.”
The American Economic Review 105, no. 5 (2015): 491‐495.

Prediction v. Causal Inference◦ S. Athey. Beyond prediction: Using big data for policy problems. Science, 355 (6324):483‐485, 2017.
◦ A. Belloni, V. Chernozhukov, C. Hansen: “High‐Dimensional Methods and Inference on Structural andTreatment Effects,” Journal of Economic Perspectives, 28 (2), Spring 2014, 29‐50.
https://www.aeaweb.org/articles?id=10.1257/jep.28.2.29

Selected References:
Treatment Effect Estimation and Machine Learning
Survey: Athey, “The Impact of Machine Learning on Economics,” NBER
Volume, 2018
ATE
◦ McCaffrey et al. [2004] (propensity score)◦ Athey, Imbens and Wager [2018], Belloni et al. [2013], Chernozhukov et al.
[2016], Chernozhukov et al [2017], Chernozhukov et al [2018], van der Laan and
Rubin [2006] focus on doubly robust methods.

Dynamic Treatment Regimes◦ Chakroborty and Murphy (2014): Survey

Heterogeneous Treatment Effects◦◦◦◦

Imai and Ratkovic, 2013: LASSO
Zeiles et al [2008], Athey and Imbens [2016]: SubgroupsFriedberg et al. [2018]: Local linear forests
Wager and Athey [2018], Athey, Tibshirani, and Wager [2018]: Causal forestsand generalized random forests◦ Kunzel et al [2017]: Meta‐learners◦ Hartford et al [2017]: Deep IV◦ Chernozhukov et al [2018]: Testing top CATEs

Instrumental Variables◦ Chernozhukov et al (multiple papers)◦ Goldman and Rao [2017], Peysakhovich & Eckles [2018]:
experiments as instruments◦ Athey, Tibshirani, and Wager [2018]; Hartford et al [2017]:
heterogeneous treatment effects

Optimal Policy Estimation◦ Dudik et al [2011], Li et al [2012], Dudik et al [2014], Li et al[2014]◦ Thomas and Brunskill [2016], Kallus [2017]◦ Kitagawa and Tetenov [2016], Swaminathan and Joachim[2015], Zhao et al [2014]‐IPW◦ Athey and Wager [2017], Zhou, Athey, and Wager [2018],
CAIPW (doubly robust, efficiency with unknown propensity)

Contextual Bandits◦ Li et al [2010], Chapelle and Li [2011], Li et al [2017], Bastaniand Bayati [2015]◦ See Agarwal et al [2016] for a survey; John Langford for manytutorials and articles◦ Bakshy et al‐Bayesian optimization perspective◦ Dimakopoulou, Zhou, Athey and Imbens [2018]

Selected References:
Social Networks and Interference
Aronow [2018], Athey, Eckles and Imbens [2018]:
Randomization Inference Approach
Kizilcec, R.F., Bakshy, E., Eckles, D., & Burke, M. [2018]:
Social Influence
Eckles, D., Karrer, B., & Ugander, J. [2017]: Reducing Biasfrom interference
Eckles, D., Kizilcec, R. F. & Bakshy, E. [2016]

Selected References:
Structural Estimation
DISCRETE CHOICE/DEMAND SYSTEMS/
SUPPLY BEHAVIOR/WELFARE ESTIMATIONOLIGOPOLY/EQUILIBRIUM APPLICATIONSMcFadden [1972]

Porter and Zona [1999]

Deaton, A., and J. Muellbauer [1980]

Nevo [2000]

Berry [1994]

Busse and Rysman [2005]

Berry, Levinsohn, and Pakes [1995, 2004]

Dafny [2009]

Nevo [2000, 2001]

Marshall and Marx [2012]

Keane et al. [2013]
Elrod [1988]; Elrod and Keane, [1995];
Chintagunta [1994] (latent variable models)Selected References:
Structural Estimation and Market Design
TRADITIONAL AUCTIONSMARKET DESIGN

Laffont et al. (1995), Perrigne and Vuong: Identificationand estimation of first price auctions

Sponsored search auctions

Athey, Levin and Seira (2011), Athey, Coey and Levin(2013): counterfactual analysis of auction design andsmall business set‐asides in timber auctions
Hendricks, Pischke, and Porter: Identification andestimation with Common Values
Athey and Haile [2002]: Identification
Athey and Haile [2007]: Survey
Haile and Tamer [2003]: Bounds on counterfactualswith partial identification◦ Varian (2009)◦ Athey and Nekipelov (2012)◦ Bottou (2012)

Matching markets◦ Agarwal (2015): Medical match◦ Agarwal and Somaini (2018): School choice◦ Agarwal, Ashlagi, Rees, Somaini, Waldinger (2018): Kidney
allocation

Selected References:
Structural Estimation in Dynamic Settings
SINGLE PLAYER DYNAMIC OPTIMIZATION
Ackerberg, Daniel, “Advertising, Learning, and Consumer Choice in Experience Good Markets:A Structural Empirical Examination,”International Economic Review, 44: 1007‐1040, (2003).
Aguirregabiria, Victor, “The Dynamics of Markups and Inventories in Retailing Firms,” Review of Economic Studies 66(2): 275‐308,(1999).
Benkard, C. Lanier, “Learning and Forgetting: The Dynamics of Aircraft Production,” American Economic Review, 90(4): 1034‐1054,(2000).
Hitsch, Gunter, “An Empirical Model of Optimal Dynamic Product Launch and Exit Under Demand Uncertainty,” Marketing Science,25(1): 25‐50, (2006).
Hotz, Joseph and Robert Miller, “Conditional Choice Probabilities and the Estimation of Dynamic Models,” Review of Economic
Studies 60(3): 497‐530, (1993).
Hotz, Joseph, Robert Miller, Seth Sanders and Jeffrey Smith “A Simulation Estimator for Dynamic Models of Discrete Choice,” Review
of Economic Studies, 61: 256‐289, (1994).
Pakes, Ariel, “Patents as Options: Some Estimates of the Value of Holding European Patent Stocks,” Econometrica, 54(4): 755‐784,(1986).
Rust, John, “Optimal Replacement of GMC Bus Engines: An Empirical Model of Harold Zurcher,” Econometrica, 55(5): 999‐1033,(1987).

Selected References:
Structural Estimation in Dynamic Settings
MULTI‐PLAYER GAMES
Ackerberg, Daniel, Steven Berry, Lanier Benkard, and Ariel Pakes, “Econometric Tools for Analyzing Market Outcomes,” in Handbook of Econometrics. J.J. Heckman and E.E. Leamer(ed.), Elsevier. Edition 1, volume 6, (2007).
Bajari, Patrick, Lanier Benkard, and Jonathan Levin, “Estimating Dynamic Models of Imperfect Competition,”Econometrica, 75(5): 1331‐1370, (2007). 17
Benkard, Lanier, “Dynamic Analysis of the Market for Wide‐Bodied Commercial Aircraft,” Review of Economic Studies, 71(3): 581‐611, (2004).
Ericson, Richard and Ariel Pakes, “Markov‐Perfect Industry Dynamics: A Framework for Empirical Work,” Review of Economic Studies, 62(1): 53‐82, (1995).
Gowrisankaran, Guatam and Robert Town, “Dynamic Equilibrium in The Hospital Industry,” Journal of Economics and Management Strategy, 6(1): 45‐74, (1997).
Markovich, Sarit, “Snowball: The Evolution of Dynamic Oligopolies with Network Externalities,” Journal of Economic Dynamics and Control, 33(3): 909‐938, (2007).
Pakes, Ariel and Paul McGuire, “Computing Markov‐Perfect Nash Equilibria: Numerical Implications of a Dynamic Differentiated Product Model,” Rand Journal of Economic, 25(4):555‐589, (1994).
Pakes, Ariel and Richard Ericson, “Empirical Implications of Alternative Models of Firm Dynamics,” Journal of Economic Theory, 79(1): 1‐45, (1998).
Pakes, Ariel, Michael Ostrovsky, and Steven Berry, “Simple Estimators for the Parameters of Dynamic Discrete Games (with Entry/Exit Examples),” Rand Journal of Economics, 38(2):373‐ 399, (2007).
Pakes Ariel and U. Doraszelski, “A Framework for Applied Dynamic Analysis in IO”. In: Armstrong M, Porter R, The Handbook of Industrial Organization. Vol. 3. New York: Elsevier;
pp. Chapter 33 2183‐2162 (2007). Ryan, Stephen, “The Costs of Environmental Regulation in a Concentrated Industry,” Econometrica, 80(3): 1019‐1061, (2012).

Selected References:
Structural Estimation and Machine Learning
CONSUMER CHOICECounterfactual Inference for Consumer Choice
Across Many Product Categories (Susan Athey,David Blei, Rob Donnelly, Francisco Ruiz, inprogress)
SHOPPER: A Probabilistic Model of Consumer
Choice with Substitutes and Complements(Francisco Ruiz, Susan Athey, David Blei, 2017)
Estimating Heterogeneous Consumer
Preferences for Restaurants and Travel Time
Using Mobile Location Data (Susan Athey, David
Blei, Rob Donnelly, Francisco Ruiz, Tobias
Schmidt, AEA Papers and Proceedings, 2018)

Wan, Mengting, et al. "Modeling consumer
preferences and price sensitivities from large‐
scale grocery shopping transaction
logs." Proceedings of the 26th International
Conference on World Wide Web. International
World Wide Web Conferences Steering
Committee, 2017.