The Data Life Cycle

Jeannette M. Wing

Computer Science

Columbia University

The Lives and After Lives of Data

Christine L. Borgman

Information Studies, UCLA

Data governance is key to interpretation: 

Reconceptualizing data in data science

Sabina Leonelli

Sociology, Philosophy and Anthropology, 

University of Exeter

Generation Collection Processing Storage Management Analysis Visualization Interpretation 

Data Conceptualization Data Curation and Provenance 
2022/11/10
2022/11/10

5
2022/11/10

How my own research has been affected/evolved ...

• Data Quality (Multi-source Inference/Learning) 

• Data Privacy (Multi-phase Inference/Learning)

• Data Granularity (Multi-resolution Inference/Learning) 

Building Representative Miniatures out of

Non-representative Big Data: An Interplay of

Data Quantity, Quality, and Privacy

Xiao-Li Meng, Harvard University

Xiao-Li Meng, Harvard University Menu 1 1 / 26
Building Representative Miniatures out of

Non-representative Big Data: An Interplay of

Data Quantity, Quality, and Privacy

Xiao-Li Meng, Harvard University

• Meng (2022). Miniaturizing Data Defect Correlation: A Versatile Strategy

for Handling Non-Probability Samples. Survey Methodology.

• Meng (2018) Statistical Paradises and Paradoxes in Big Data (I): From

The Law of Large Populations to The Big Data Paradox. Annals of

Applied Statistics, 12, 685–726.

• Bradley, Sejdinovic, Meng, Kuriwaki, Isakov, Flaxman (2021)

Unrepresentative Big Surveys Significantly Overestimated COVID-19

Vaccination in the US. Nature Dec; 600(7890):695-700.

Xiao-Li Meng, Harvard University Menu 1 1 / 26
It has been a decade ...

• We know that a 5% random sample is better than a 5% non-random

sample in measurable ways (e.g., bias, predictive power).

Xiao-Li Meng, Harvard University Menu 2 2 / 26
It has been a decade ...

• We know that a 5% random sample is better than a 5% non-random

sample in measurable ways (e.g., bias, predictive power).

• But is an 80% non-random sample “better” than a 5% random

sample in measurable terms? 90%? 95%? 99%? (Jeremy Wu of

US Census Bureau, 2012, Seminar at Harvard Statistics)

Xiao-Li Meng, Harvard University Menu 2 2 / 26
It has been a decade ...

• We know that a 5% random sample is better than a 5% non-random

sample in measurable ways (e.g., bias, predictive power).

• But is an 80% non-random sample “better” than a 5% random

sample in measurable terms? 90%? 95%? 99%? (Jeremy Wu of

US Census Bureau, 2012, Seminar at Harvard Statistics)

• “Which one should we trust more: a 1% survey with 60%

response rate or a non-probabilistic dataset covering 80% of

the population?” (Keiding and Louis, 2015, Joint Statistical

Meetings; and JRSSB, 2016)

Xiao-Li Meng, Harvard University Menu 2 2 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

• Landmark paper: Jerzy

Neyman (1934)

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

• Landmark paper: Jerzy

Neyman (1934)

• The “revolution” lasted

about 50 years (Jelke

Bethlehem, 2009)

Xiao-Li Meng, Harvard University Menu 3 3 / 26
It has been a few centuries ..

• Law of Large Numbers:

Jakob Bernoulli (1713)

• Central Limit Theorem:

Abraham de Moivre (1733):

error ∝ √

1

n

: n − sample size

• Survey Sampling:

• Graunt (1662); Laplace (1882)

• The “intellectually violent

revolution” in 1895 by Anders Kiær,

Statistics Norway

• Landmark paper: Jerzy

Neyman (1934)

• The “revolution” lasted

about 50 years (Jelke

Bethlehem, 2009)

• First implementation in

US Census: 1940 led by

Morris Hansen

Xiao-Li Meng, Harvard University Menu 3 3 / 26
Triumphs and Troubles of Probability Sampling

Triumphs: Ensuring learning is feasible and cost effective

Xiao-Li Meng, Harvard University Menu 4 4 / 26
Triumphs and Troubles of Probability Sampling

Triumphs: Ensuring learning is feasible and cost effective

• Census is not feasible in most studies

• Sampling makes a study feasible and even cost effective

• Sampling also reduces privacy cost since it is a random suppression

Xiao-Li Meng, Harvard University Menu 4 4 / 26
Triumphs and Troubles of Probability Sampling

Triumphs: Ensuring learning is feasible and cost effective

• Census is not feasible in most studies

• Sampling makes a study feasible and even cost effective

• Sampling also reduces privacy cost since it is a random suppression

Troubles: Ensuring “repetitiveness” is increasingly challenging

Xiao-Li Meng, Harvard University Menu 4 4 / 26
Triumphs and Troubles of Probability Sampling

Triumphs: Ensuring learning is feasible and cost effective

• Census is not feasible in most studies

• Sampling makes a study feasible and even cost effective

• Sampling also reduces privacy cost since it is a random suppression

Troubles: Ensuring “repetitiveness” is increasingly challenging

• Non-response/coverage bias (not merely the rate) is on the rise

• Social media data are non-representative by design

• Administrative data are not probability samples

Thoughts: Handling data defects and privacy concomitantly

Xiao-Li Meng, Harvard University Menu 4 4 / 26
Triumphs and Troubles of Probability Sampling

Triumphs: Ensuring learning is feasible and cost effective

• Census is not feasible in most studies

• Sampling makes a study feasible and even cost effective

• Sampling also reduces privacy cost since it is a random suppression

Troubles: Ensuring “repetitiveness” is increasingly challenging

• Non-response/coverage bias (not merely the rate) is on the rise

• Social media data are non-representative by design

• Administrative data are not probability samples

Thoughts: Handling data defects and privacy concomitantly

• Can we utilize the inherent data defects as data privacy protections?

• Can we engineer “representative miniatures” that trade data utility

and privacy more sensibly than adding noises to defective data?

Xiao-Li Meng, Harvard University Menu 4 4 / 26
Statistical Thinking with R ...

Xiao-Li Meng, Harvard University Menu 5 5 / 26
Statistical Thinking with R ...

Recording (Reporting/Responding) Indicator R for a finite

population X = {X1, . . . , XN} :

Ri = 1, if Xiis included in a sample of size n, and Ri = 0 otherwise

Xiao-Li Meng, Harvard University Menu 5 5 / 26
Statistical Thinking with R ...

Recording (Reporting/Responding) Indicator R for a finite

population X = {X1, . . . , XN} :

Ri = 1, if Xiis included in a sample of size n, and Ri = 0 otherwise

Design Prob: Prob controlled and implemented by human

Ex: Prob sampling, Pr(Ri = 1|X) = n/N

More: Bootstraps, clinical trials, differential privacy, permutation test, MC.

Xiao-Li Meng, Harvard University Menu 5 5 / 26
Statistical Thinking with R ...

Recording (Reporting/Responding) Indicator R for a finite

population X = {X1, . . . , XN} :

Ri = 1, if Xiis included in a sample of size n, and Ri = 0 otherwise

Design Prob: Prob controlled and implemented by human

Ex: Prob sampling, Pr(Ri = 1|X) = n/N

More: Bootstraps, clinical trials, differential privacy, permutation test, MC.

Divine Prob: Existential Prob by God/Nature or by faith.

Ex: Real-World Evidence, but we assume R is random, and

Pr(Ri = 1|X) = π(Xi). [Needed for define “missing at Random” ]

Xiao-Li Meng, Harvard University Menu 5 5 / 26
Statistical Thinking with R ...

Recording (Reporting/Responding) Indicator R for a finite

population X = {X1, . . . , XN} :

Ri = 1, if Xiis included in a sample of size n, and Ri = 0 otherwise

Design Prob: Prob controlled and implemented by human

Ex: Prob sampling, Pr(Ri = 1|X) = n/N

More: Bootstraps, clinical trials, differential privacy, permutation test, MC.

Divine Prob: Existential Prob by God/Nature or by faith.

Ex: Real-World Evidence, but we assume R is random, and

Pr(Ri = 1|X) = π(Xi). [Needed for define “missing at Random” ]

Device Prob: Probability constructs invoked for analysis.

Ex: logit[π(x)] = α + βx

More: Prob distributions for expressing belief, prior knowledge,

assumptions, idealizations, compromises, desperation.

Xiao-Li Meng, Harvard University Menu 5 5 / 26
A Fundamental Identity for Estimation Error (Meng, 2018)

Xiao-Li Meng, Harvard University Menu 6 6 / 26
A Fundamental Identity for Estimation Error (Meng, 2018)

• Population {X1, ..., XN

}; Estimand μ ̄N =

PN

i=1 G(Xi )

N

;

Xiao-Li Meng, Harvard University Menu 6 6 / 26
A Fundamental Identity for Estimation Error (Meng, 2018)

• Population {X1, ..., XN

}; Estimand μ ̄N =

PN

i=1 G(Xi )

N

;

• Estimator: sample average

μ ̄n =

PN

i=1 RiG(Xi)

PN

i=1 Ri

≡

PN

i=1 RiG(Xi)

nR

where Ri = 1 if Xiis recorded, and zero otherwise.

• Neither X nor R is assumed random.

Xiao-Li Meng, Harvard University Menu 6 6 / 26
A Fundamental Identity for Estimation Error (Meng, 2018)

• Population {X1, ..., XN

}; Estimand μ ̄N =

PN

i=1 G(Xi )

N

;

• Estimator: sample average

μ ̄n =

PN

i=1 RiG(Xi)

PN

i=1 Ri

≡

PN

i=1 RiG(Xi)

nR

where Ri = 1 if Xiis recorded, and zero otherwise.

• Neither X nor R is assumed random.

Expressing the exact error w.r.t I ∼ U[1, . . . ,N] (a divine probability):

μ ̄n − μ ̄N =

EI[RIGI]

EI[RI]

− EI[GI] = CovI(RI

, GI)

EI[RI]

= ˆρR,G ×

r

N − n

n

× σG
A Fundamental Identity for Estimation Error (Meng, 2018)

• Population {X1, ..., XN

}; Estimand μ ̄N =

PN

i=1 G(Xi )

N

;

• Estimator: sample average

μ ̄n =

PN

i=1 RiG(Xi)

PN

i=1 Ri

≡

PN

i=1 RiG(Xi)

nR

where Ri = 1 if Xiis recorded, and zero otherwise.

• Neither X nor R is assumed random.

Expressing the exact error w.r.t I ∼ U[1, . . . ,N] (a divine probability):

μ ̄n − μ ̄N =

EI[RIGI]

EI[RI]

− EI[GI] = CovI(RI

, GI)

EI[RI]

= ˆρR,G ×

r

N − n

n

× σG

Xiao-Li Meng, Harvard University Menu 6 6 / 26
Why teaching is so important ...

Xiao-Li Meng, Harvard University Menu 7 7 / 26
Why teaching is so important ...

Xiao-Li Meng, Harvard University Menu 7 7 / 26
Xiao-Li Meng, Harvard University Menu 8 8 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

×

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

× σG

|{z}

Problem Difficulty

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

× σG

|{z}

Problem Difficulty

• Nothing is assumed random, only using divine probability via I.

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

× σG

|{z}

Problem Difficulty

• Nothing is assumed random, only using divine probability via I.

• Only assumption: sampled Xi are in the target population.

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

× σG

|{z}

Problem Difficulty

• Nothing is assumed random, only using divine probability via I.

• Only assumption: sampled Xi are in the target population.

Compare to the additive “Variance+Bias” decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

=  ̄μn − ER( ̄μn)

| {z }

Sampling Error

+ ER( ̄μn) − μ ̄N

| {z }

Bias

Xiao-Li Meng, Harvard University Menu 9 9 / 26
A Multiplicative Error Decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

= ˆρR,G

|{z}

Data Quality

×

r

N − n

n

| {z }

Data Quantity

× σG

|{z}

Problem Difficulty

• Nothing is assumed random, only using divine probability via I.

• Only assumption: sampled Xi are in the target population.

Compare to the additive “Variance+Bias” decomposition

μ ̄n − μ ̄N

| {z }

Exact Error

=  ̄μn − ER( ̄μn)

| {z }

Sampling Error

+ ER( ̄μn) − μ ̄N

| {z }

Bias

• Must involve prob on R, design, divine, or device.

• It’s “phenotype” decomposition, not “genotype.”

Xiao-Li Meng, Harvard University Menu 9 9 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

• Stir it well, then a few bits

are sufficient regardless of

the size of the container!

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

• Stir it well, then a few bits

are sufficient regardless of

the size of the container!

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

• Stir it well, then a few bits

are sufficient regardless of

the size of the container!

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

• Stir it well, then a few bits

are sufficient regardless of

the size of the container!

⇐⇒

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Why and when can we ignore the population size N?

• Think about tasting soup

• Stir it well, then a few bits

are sufficient regardless of

the size of the container!

⇐⇒

• But what happens when we fail to stir (well)?

Xiao-Li Meng, Harvard University Menu 10 10 / 26
Probability Sampling miniaturizes the ddc

For SRS, because VR(Z) = 1, where

Z =

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1 ⇒ VR(ˆρR,G) = 1

N − 1

Hence ˆρR,G ∈



− √

3

N−1

, √

3

N−1



99% of the time regardless of G.

Xiao-Li Meng, Harvard University Menu 11 11 / 26
Probability Sampling miniaturizes the ddc

For SRS, because VR(Z) = 1, where

Z =

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1 ⇒ VR(ˆρR,G) = 1

N − 1

Hence ˆρR,G ∈



− √

3

N−1

, √

3

N−1



99% of the time regardless of G.

μ ̄n − μ ̄N = ˆρR,G

|{z}

1/

√

N

×

r

N − n

n

| {z }

√

N

×σG

Xiao-Li Meng, Harvard University Menu 11 11 / 26
Probability Sampling miniaturizes the ddc

For SRS, because VR(Z) = 1, where

Z =

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1 ⇒ VR(ˆρR,G) = 1

N − 1

Hence ˆρR,G ∈



− √

3

N−1

, √

3

N−1



99% of the time regardless of G.

μ ̄n − μ ̄N = ˆρR,G

|{z}

1/

√

N

×

r

N − n

n

| {z }

√

N

×σG

• This cancellation is THE reason that we can ignore N for any G.

Xiao-Li Meng, Harvard University Menu 11 11 / 26
Data Defect Correlation (ddc) and Data Defect Index (ddi)

ddc: ˆρR,G

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1

ddi: DI = ER(ˆρ

2

R,G

)

Deff ≡

MSE( ̄μn)

SRS MSE = DI(N − 1)

Xiao-Li Meng, Harvard University Menu 12 12 / 26
Data Defect Correlation (ddc) and Data Defect Index (ddi)

ddc: ˆρR,G

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1

ddi: DI = ER(ˆρ

2

R,G

)

Deff ≡

MSE( ̄μn)

SRS MSE = DI(N − 1)

ddi = Deff

N−1 = Design effect per subject in the population

• For SRS: DI = (N − 1)−1

Xiao-Li Meng, Harvard University Menu 12 12 / 26
Data Defect Correlation (ddc) and Data Defect Index (ddi)

ddc: ˆρR,G

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1

ddi: DI = ER(ˆρ

2

R,G

)

Deff ≡

MSE( ̄μn)

SRS MSE = DI(N − 1)

ddi = Deff

N−1 = Design effect per subject in the population

• For SRS: DI = (N − 1)−1

• Probability sample =⇒ DI ∝ N

−1

Xiao-Li Meng, Harvard University Menu 12 12 / 26
Data Defect Correlation (ddc) and Data Defect Index (ddi)

ddc: ˆρR,G

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1

ddi: DI = ER(ˆρ

2

R,G

)

Deff ≡

MSE( ̄μn)

SRS MSE = DI(N − 1)

ddi = Deff

N−1 = Design effect per subject in the population

• For SRS: DI = (N − 1)−1

• Probability sample =⇒ DI ∝ N

−1

• MSE( ̄μn) ∝ n

−1 ⇐⇒ DI ∝ N−1

Xiao-Li Meng, Harvard University Menu 12 12 / 26
Data Defect Correlation (ddc) and Data Defect Index (ddi)

ddc: ˆρR,G

μ ̄n − μ ̄ q N

1−f

n

SG

= ˆρR,G

√

N − 1

ddi: DI = ER(ˆρ

2

R,G

)

Deff ≡

MSE( ̄μn)

SRS MSE = DI(N − 1)

ddi = Deff

N−1 = Design effect per subject in the population

• For SRS: DI = (N − 1)−1

• Probability sample =⇒ DI ∝ N

−1

• MSE( ̄μn) ∝ n

−1 ⇐⇒ DI ∝ N−1

Deep Trouble

• when DI does not vanish with N

−1

;

• or equivalently when ˆρG,R

does not vanish with N

−1/2

...

Xiao-Li Meng, Harvard University Menu 12 12 / 26
How Small is Our “Big Data”?

Xiao-Li Meng, Harvard University Menu 13 13 / 26
How Small is Our “Big Data”?

The Effective Sample Size of a “Big Data” in terms of SRS size

neff =

n

1 + (1 − f )[(N − 1)ER[ˆρ

2

R,G

] − 1] ≈

f

1 − f

1

ρˆ

2

Xiao-Li Meng, Harvard University Menu 13 13 / 26
How Small is Our “Big Data”?

The Effective Sample Size of a “Big Data” in terms of SRS size

neff =

n

1 + (1 − f )[(N − 1)ER[ˆρ

2

R,G

] − 1] ≈

f

1 − f

1

ρˆ

2

Why do we need random testing? (Walter Dempsey, Twitter)

• NY State: N ≈ 20M;

Xiao-Li Meng, Harvard University Menu 13 13 / 26
How Small is Our “Big Data”?

The Effective Sample Size of a “Big Data” in terms of SRS size

neff =

n

1 + (1 − f )[(N − 1)ER[ˆρ

2

R,G

] − 1] ≈

f

1 − f

1

ρˆ

2

Why do we need random testing? (Walter Dempsey, Twitter)

• NY State: N ≈ 20M;

• Suppose we conduct n = 10, 000 COVID test: f = 1/2000

Xiao-Li Meng, Harvard University Menu 13 13 / 26
How Small is Our “Big Data”?

The Effective Sample Size of a “Big Data” in terms of SRS size

neff =

n

1 + (1 − f )[(N − 1)ER[ˆρ

2

R,G

] − 1] ≈

f

1 − f

1

ρˆ

2

Why do we need random testing? (Walter Dempsey, Twitter)

• NY State: N ≈ 20M;

• Suppose we conduct n = 10, 000 COVID test: f = 1/2000

• Suppose the selective testing resulted in ˆρ = 0.005;

neff =

0.0005

0.9995

×

1

0.0052

≈ 20

Xiao-Li Meng, Harvard University Menu 13 13 / 26
How Small is Our “Big Data”?

The Effective Sample Size of a “Big Data” in terms of SRS size

neff =

n

1 + (1 − f )[(N − 1)ER[ˆρ

2

R,G

] − 1] ≈

f

1 − f

1

ρˆ

2

Why do we need random testing? (Walter Dempsey, Twitter)

• NY State: N ≈ 20M;

• Suppose we conduct n = 10, 000 COVID test: f = 1/2000

• Suppose the selective testing resulted in ˆρ = 0.005;

neff =

0.0005

0.9995

×

1

0.0052

≈ 20

• Hence ˆρ = 0.005 implies a 99.80% loss of sample size!

Xiao-Li Meng, Harvard University Menu 13 13 / 26
Where Did ˆρ = 0.005 Come From? (Meng, 2018)

Cooperative Congressional Election Study by Ansolabehere, Schaffner, Luks, Rivers on Oct 4 -

Nov 6, 2016 (YouGov); Analysis assisted by Shiro Kuriwaki

Xiao-Li Meng, Harvard University Menu 14 14 / 26
Where Did ˆρ = 0.005 Come From? (Meng, 2018)

Cooperative Congressional Election Study by Ansolabehere, Schaffner, Luks, Rivers on Oct 4 -

Nov 6, 2016 (YouGov); Analysis assisted by Shiro Kuriwaki

Poll underestimated

Clinton support

Poll overestimated

Clinton support

0%

50%

100%

0% 50% 100%

Final Clinton Popular Vote Share

Turnout−adjusted Poll Estimate,

Clinton Support

Root Mean Squared Error: 0.06

−0.00012 ± 0.00059

0

4

8

12

−0.010 −0.005 0.000 0.005 0.010

Clinton ρNvv

Count

Clinton: ˆρvvt ≈ −0.0001 ± 0.0006

Xiao-Li Meng, Harvard University Menu 14 14 / 26
Where Did ˆρ = 0.005 Come From? (Meng, 2018)

Cooperative Congressional Election Study by Ansolabehere, Schaffner, Luks, Rivers on Oct 4 -

Nov 6, 2016 (YouGov); Analysis assisted by Shiro Kuriwaki

Poll underestimated

Clinton support

Poll overestimated

Clinton support

0%

50%

100%

0% 50% 100%

Final Clinton Popular Vote Share

Turnout−adjusted Poll Estimate,

Clinton Support

Root Mean Squared Error: 0.06

−0.00012 ± 0.00059

0

4

8

12

−0.010 −0.005 0.000 0.005 0.010

Clinton ρNvv

Count

Clinton: ˆρvvt ≈ −0.0001 ± 0.0006

Poll underestimated

Trump support

Poll overestimated

Trump support

0%

50%

100%

0% 50% 100%

Final Trump Popular Vote Share

Turnout−adjusted Poll Estimate,

Trump Support

Root Mean Squared Error: 0.15

−0.0043 ± 0.00059

0

4

8

12

−0.010 −0.005 0.000 0.005 0.010

Trump ρNvv

Count

Trump: ˆρvvt ≈ −0.0043 ± 0.0006

Xiao-Li Meng, Harvard University Menu 14 14 / 26
Studies of COVID-19 vaccine uptake vary in design

Axios-Ipsos

Coronavirus Index

Census Household

Pulse

Facebook-Delphi

COVID Symptom

Survey

Mode online online online

Sampling

frame

Ipsos

KnowledgePanel

Census Master

Address File

Facebook Active User

Base

n 1,000/wave 65,000/wave 250,000/wave

Question

Wording

“Do you personally

know anyone who has

already received the

COVID-19 vaccine?”

Answers include “Yes,

I have received the

vaccine”

“Have you received a

COVID-19 vaccine?”

“Have you had a

COVID-19

vaccination?”

Target

Population

2019 CPS March

Supplement, US 18+

2018 ACS, 1-year

est., US 18+

2018 CPS March

Supplement, US 18+

Weighting

Variables

gender x age, race,

education, Census

region, metropolitan

status, household

income, partisanship

metropolitan

statistical area

(MSA), state x

education x gender x

age, state x Hispanic

ethnicity x gender x

age

state x age x gender

and “proprietary

covariates”

Xiao-Li Meng, Harvard University Menu 15 15 / 26
Big Data Paradox: The Bigger the Data, The Surer We

Fool Ourselves

Xiao-Li Meng, Harvard University Menu 16 16 / 26
Estimated Effective Sample Sizes (dropping three 0s!)

Xiao-Li Meng, Harvard University Menu 17 17 / 26
Dramatic Reduction in Effective Sample Size (> 99.9%)

Xiao-Li Meng, Harvard University Menu 18 18 / 26
Composition of Survey Respondents

Xiao-Li Meng, Harvard University Menu 19 19 / 26
“Weight, Weight, Don’t tell me ...”

There are always those “messy” weights ...

μ ̄w =

PN

i=1 RiwiG(Xi)

PN

i=1 Riwi

Xiao-Li Meng, Harvard University Menu 20 20 / 26
“Weight, Weight, Don’t tell me ...”

There are always those “messy” weights ...

μ ̄w =

PN

i=1 RiwiG(Xi)

PN

i=1 Riwi

Let CVWbe the coefficient of variation of WI given RI = 1

μ ̄w − μ ̄N = ˆρRw ,G ×

r

1 − fw

fw

× σG

where

fw =

nw

N

, nw =

n

1 + CV2

w

(Kish, 1965)

Xiao-Li Meng, Harvard University Menu 20 20 / 26
“Weight, Weight, Don’t tell me ...”

There are always those “messy” weights ...

μ ̄w =

PN

i=1 RiwiG(Xi)

PN

i=1 Riwi

Let CVWbe the coefficient of variation of WI given RI = 1

μ ̄w − μ ̄N = ˆρRw ,G ×

r

1 − fw

fw

× σG

where

fw =

nw

N

, nw =

n

1 + CV2

w

(Kish, 1965)

• Seeking w to make ˆρRw ,G < ρˆR,G

and compensate for nw < n.

Xiao-Li Meng, Harvard University Menu 20 20 / 26
Estimation Methods for Non-probability Samples

• A non-probability sample {(yi

, Xi), i ∈ S}, S = {i : Ri = 1}

• An auxiliary probability sample {Xi

, i ∈ S

∗}, S∗ = {i : R∗

i = 1}

• Key Assumption: yi ⊥ Ri

|Xi (Missing at Random)

• A device model: p(y, R|x) = p(y|x)p(R|x)

Xiao-Li Meng, Harvard University Menu 21 21 / 26
Estimation Methods for Non-probability Samples

• A non-probability sample {(yi

, Xi), i ∈ S}, S = {i : Ri = 1}

• An auxiliary probability sample {Xi

, i ∈ S

∗}, S∗ = {i : R∗

i = 1}

• Key Assumption: yi ⊥ Ri

|Xi (Missing at Random)

• A device model: p(y, R|x) = p(y|x)p(R|x)

Quasi-randomization: Estimate π(x) = Prp(R = 1|x)

μˆ =

PN

i=1 Riwi yi

PN

i=1 Riwi

, wi ∝ πˆ

−1

(Xi)

Xiao-Li Meng, Harvard University Menu 21 21 / 26
Estimation Methods for Non-probability Samples

• A non-probability sample {(yi

, Xi), i ∈ S}, S = {i : Ri = 1}

• An auxiliary probability sample {Xi

, i ∈ S

∗}, S∗ = {i : R∗

i = 1}

• Key Assumption: yi ⊥ Ri

|Xi (Missing at Random)

• A device model: p(y, R|x) = p(y|x)p(R|x)

Quasi-randomization: Estimate π(x) = Prp(R = 1|x)

μˆ =

PN

i=1 Riwi yi

PN

i=1 Riwi

, wi ∝ πˆ

−1

(Xi)

Super-population/model assisted: also fit y = m(x)

μˆ+ =

PN

i=1 Riwi(yi − mˆ(Xi))

PN

i=1 Riwi

+

PN

i=1 R

∗

i mˆ(Xi)

PN

i=1 R

∗

i

,

Xiao-Li Meng, Harvard University Menu 21 21 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Under a divine model for (R, Y |X) and sampling model for R

∗

E[ˆμ+] − μ ∼ Ex {CovI[πIwI, δI]} , where δI ≡ E(yI|XI) − mˆ(XI)

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Under a divine model for (R, Y |X) and sampling model for R

∗

E[ˆμ+] − μ ∼ Ex {CovI[πIwI, δI]} , where δI ≡ E(yI|XI) − mˆ(XI)

• Quasi-randomization: making πIwI ∝ 1, (Q)

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Under a divine model for (R, Y |X) and sampling model for R

∗

E[ˆμ+] − μ ∼ Ex {CovI[πIwI, δI]} , where δI ≡ E(yI|XI) − mˆ(XI)

• Quasi-randomization: making πIwI ∝ 1, (Q)

• Super-population: making δI ≡ E(yI

|XI) − mˆ(XI) = 0, (S)

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Under a divine model for (R, Y |X) and sampling model for R

∗

E[ˆμ+] − μ ∼ Ex {CovI[πIwI, δI]} , where δI ≡ E(yI|XI) − mˆ(XI)

• Quasi-randomization: making πIwI ∝ 1, (Q)

• Super-population: making δI ≡ E(yI

|XI) − mˆ(XI) = 0, (S)

• Doubly robust: either (Q) or (S) makes Cov(πIwI

, δI) = 0, but we

don’t need to know which one.

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Why Do They Work?

μˆ+ − y ̄N =

CovI(RIwI, yI − mˆ(XI))

EI(RIwI)

+

CovI(R

∗

I

, mˆ(XI))

EI(R∗

I

)

,

Under a divine model for (R, Y |X) and sampling model for R

∗

E[ˆμ+] − μ ∼ Ex {CovI[πIwI, δI]} , where δI ≡ E(yI|XI) − mˆ(XI)

• Quasi-randomization: making πIwI ∝ 1, (Q)

• Super-population: making δI ≡ E(yI

|XI) − mˆ(XI) = 0, (S)

• Doubly robust: either (Q) or (S) makes Cov(πIwI

, δI) = 0, but we

don’t need to know which one.

• “Double+ robustness”: the validity holds if and only if

Ex {CovI[πIwI, δI]} = 0

Xiao-Li Meng, Harvard University Menu 22 22 / 26
Counterbalance Sub-sampling (CBS)

• If Pr(Ri = 1|X, Y ) = πi

, sub-sample with Pr(Si = 1|Ri = 1) ∝ π

−1

i

Xiao-Li Meng, Harvard University Menu 23 23 / 26
Counterbalance Sub-sampling (CBS)

• If Pr(Ri = 1|X, Y ) = πi

, sub-sample with Pr(Si = 1|Ri = 1) ∝ π

−1

i

• Hence Pr(SiRi = 1) ∝ 1, creating an equal-probability sample

Xiao-Li Meng, Harvard University Menu 23 23 / 26
Counterbalance Sub-sampling (CBS)

• If Pr(Ri = 1|X, Y ) = πi

, sub-sample with Pr(Si = 1|Ri = 1) ∝ π

−1

i

• Hence Pr(SiRi = 1) ∝ 1, creating an equal-probability sample

The Effective Sample Size of “Big Data” in terms of SRS size

nB ≈

fB

1 − fB

1

ρˆ

2

B

,

where fB = n/N is the relative size of sample B.

Xiao-Li Meng, Harvard University Menu 23 23 / 26
Counterbalance Sub-sampling (CBS)

• If Pr(Ri = 1|X, Y ) = πi

, sub-sample with Pr(Si = 1|Ri = 1) ∝ π

−1

i

• Hence Pr(SiRi = 1) ∝ 1, creating an equal-probability sample

The Effective Sample Size of “Big Data” in terms of SRS size

nB ≈

fB

1 − fB

1

ρˆ

2

B

,

where fB = n/N is the relative size of sample B.

Trading quantity for quality: create an unweighted sub-sample D such

that

fB

1 − fB

1

ρˆ

2

B

<

fD

1 − fD

1

ρˆ

2

D

,

where fD = fB fS , and fS is the sub-sampling rate.

Xiao-Li Meng, Harvard University Menu 23 23 / 26
Estimand p = P(Y = 1); estimable p

∗ = P(Y = 1|R = 1)

Xiao-Li Meng, Harvard University Menu 24 24 / 26
Estimand p = P(Y = 1); estimable p

∗ = P(Y = 1|R = 1)

• Reporting rates:

ry = Pr(R = 1|Y = y); r =

r1

r0

• Sub-sampling rates:

sy = Pr(S = 1|R = 1, Y = y); s =

s1

s0

Xiao-Li Meng, Harvard University Menu 24 24 / 26
Estimand p = P(Y = 1); estimable p

∗ = P(Y = 1|R = 1)

• Reporting rates:

ry = Pr(R = 1|Y = y); r =

r1

r0

• Sub-sampling rates:

sy = Pr(S = 1|R = 1, Y = y); s =

s1

s0

;

Counterbalancing: (r − 1)(s − 1) < 0

(i) If δ = r − 1 > 0, then take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1 (1)

(ii) If δ = r − 1 < 0, then take any s

1 < s ≤

1 − (1 − p

∗

)δ

[1 + (1 + p

∗)δ]+

(2)

Xiao-Li Meng, Harvard University Menu 24 24 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

Xiao-Li Meng, Harvard University Menu 25 25 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

We do not know r, but

• Suppose a previous survey had r = 1.5

Xiao-Li Meng, Harvard University Menu 25 25 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

We do not know r, but

• Suppose a previous survey had r = 1.5

• We might feel comfortable to assume that the current r ∈ (1.2, 1.8)

Xiao-Li Meng, Harvard University Menu 25 25 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

We do not know r, but

• Suppose a previous survey had r = 1.5

• We might feel comfortable to assume that the current r ∈ (1.2, 1.8)

• Suppose we observe p

∗ = 0.6

Xiao-Li Meng, Harvard University Menu 25 25 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

We do not know r, but

• Suppose a previous survey had r = 1.5

• We might feel comfortable to assume that the current r ∈ (1.2, 1.8)

• Suppose we observe p

∗ = 0.6

• Then the max of the lower bound is 0.7

Xiao-Li Meng, Harvard University Menu 25 25 / 26
An miniature illustration

Counterbalancing: (r − 1)(s − 1) < 0

When δ = r − 1 > 0, take any s

[1 − (1 − p

∗

)δ]+

1 + (1 + p

∗)δ

≤ s < 1

We do not know r, but

• Suppose a previous survey had r = 1.5

• We might feel comfortable to assume that the current r ∈ (1.2, 1.8)

• Suppose we observe p

∗ = 0.6

• Then the max of the lower bound is 0.7

• If r = 1.5, then s = 1/1.5 = 0.67 will be optimal, but any s ∈ [0.7, 1)

will lead to smaller MSE compared to not using CBS

Xiao-Li Meng, Harvard University Menu 25 25 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

Xiao-Li Meng, Harvard University Menu 26 26 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

• Data cleanness and data relevance trade-off

Xiao-Li Meng, Harvard University Menu 26 26 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

• Data cleanness and data relevance trade-off

• Data utility and data privacy trade-off

Xiao-Li Meng, Harvard University Menu 26 26 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

• Data cleanness and data relevance trade-off

• Data utility and data privacy trade-off

• ...

Xiao-Li Meng, Harvard University Menu 26 26 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

• Data cleanness and data relevance trade-off

• Data utility and data privacy trade-off

• ...

Xiao-Li Meng, Harvard University Menu 26 26 / 26
For those who just woke up ...

There is no free lunch

• Data quality and data quantity trade-off

• Data cleanness and data relevance trade-off

• Data utility and data privacy trade-off

• ...

But let’s not overpay either

• Avoid add more noise than necessary

• Avoid weighted data when an unweighted sub-sample can provide

similar statistical information

• ...

Xiao-Li Meng, Harvard University Menu 26 26 / 26