Six Crises One Dozen Opportunities in Public-Stewardship Statistics

John L. Eltinge

Assistant Director for Research and Methodology

John.L.Eltinge@census.gov

FCSM Virtual Conference – November 2, 2021

Session C-2: Evidence and Data Policy

2

Acknowledgements and Disclaimer: 

The speaker thanks many colleagues in government statistical agencies, academia and the private sector for years of very helpful discussions of the topics discussed in this presentation.     

The views expressed in this presentation are those of the speaker and do not represent the policies of the United States Census Bureau.  

Overview: Six Crises One Dozen Opportunities

3

Public-Stewardship Statistics, Stakeholders and Methodology

Six Perceived Crises

One Dozen Opportunities

Improve Understanding of Our Environment & Procedures

Improve Design of Our Procedures

I. Public-Stewardship Statistics & Methodology – What?

4

“Official Statistics” - information to address key societal needs: 

	Economic, public-health and demographic conditions

	Ex: Population Counts, Consumer Price Index, Unemployment 	      Rate, Disease Prevalence Rates, National Accounts

	Static product form: Tables, maps and related analyses

	NASEM (2021) – Principles and Practices, Seventh Edition

I. Public-Stewardship Statistics – How?

5

	Conceptual and methodological basis: 

      	Primarily sample surveys (some administrative records)

	1. Design: Allocate resources (sample units): 

	    optimize sampling variance, conditional on (approximate) 

	    unbiasedness, plus cost & operational constraints

	2. Rich literature: Neyman (1934), Hansen et al (1953), 

          Cochran (1977), Binder (1983), Fuller (1975, 2009), others

I. Public-Stewardship Statistics – Why?   (1) 

6

Perceived Stakeholder Value: Often via Concrete Use Cases Based on: 

Quality (Brackstone, 1999; others): Accuracy; 

Relevance; Granularity; Punctuality; Comparability; 

Interpretability; Accessibility; Credibility

Risk: Prospective failure points; cumulative effects; 

trajectory of recovery; fault-tolerant designs

Cost: All Resources – Cash, Data, Time, Systems, Skills, Burden

I. Public-Stewardship Statistics – Why?   (2) 

7

Generally managed via “public stewardship”

      - Related to “public goods” – some private-sector variants

Multiple stakeholders  - competing priorities

	- Trade-offs among multiple performance criteria: 

	   often complex, dynamic and conditional on environment

	- cf. “wicked problems” in design, e.g., Rittel & Webber (1973), 	  Buchanan (1992), Lindberg et al. (2012), Thienen et al. (2014)

II. Six Perceived Crises in Public-Stewardship Statistics (1)

8

Degradation of some dimensions of point estimation data quality

	   Sample survey case: declining response rates (Tourangeau, 2017)

        Broader issues with some “organic data” (e.g., admin records):

	   Population coverage, unit problems, timeliness, temporal and 

	   cross-sectional comparability – multiple components of MSE

(Groves, 2012;  Couper, 2013; Citro, 2014; NASEM, 2017; Meng, 2018)

II. Crises (2): Reproducibility & Other Inferential Issues

9

Substantive Inference and Fundamentals of the Scientific Process:

How much does it cost to be sure enough? Sharing intellectual property?

	Ioannidis (2005), Stodden et al. (2014), Wasserstein and Lazar (2016), 

      Vilhuber (2018), NASEM (2019), Efron (2020), many others

	Trade-offs among costs, inferential quality (current & future):  

	study registration; data & code curation; verification/validation servers

	Some similarities to issues with methodological protections against bias 	in epidemiological studies, e.g.,  Keiding and Louis (2016) 

II. Crises (3): Risks to Privacy and Confidentiality

10

	Database reconstruction theorem (Dinur and Nissim, 2003; 

	related comments in Abowd and Schmutte, 2019)

	Differential privacy and allocation of privacy budgets 

	Tiered access (Clark, 2020; others) – account for item sensitivity?

	Secure multiparty computing and secure memory encryption

	Nuanced assessment of impact on all dimensions of data quality

II. Crises (4): Reduction in Discretionary Resources

11

Resources: Many Components – Intangible-Capital Intensive

Ex: Data (both for production and internal controls), cash, 

calendar time, specified skills, systems, institutional capital, 

respondent burden, privacy budgets 

Often have high fixed costs with ambiguous attribution; indeterminate amortization; exacerbated by unfunded mandates

Issue: Binding constraints on which resources?  Feedback loops? 

II. Crises (5): Changing Expectations on 		       Public Stewardship and Public Goods 

12

Public goods: Weisbrod (1964), Arrow & Fisher (1974), Groshen (2018)

“non-exclusive and non-rivalrous” – unattainable (?) ideal

Public stewardship: broader definition; expectations of generally level playing field and duty of care for long-term public benefit 

Implicitly based on positive-sum approach to societal (govt?) decisions

Changes arising from societal heterogeneity → (?) increased tendency toward zero-sum and negative-sum decision processes and outcomes

II. Crises (6): General Decline in Trust in Science, 		      Expertise and Public Institutions

13

Bates et al. (2012); Bauer et al. (2019; Fobia et al. (2019); 

Hunter-Childs et al. (2019); Pew (2019)

   - Per crisis #4, impact on survey response rates, access to admin records

  -  Increased contention over use & interpretation of stat information

In part: increased public recognition of context & conditioning

In part: erosion of expectation of common base of facts

  →  Methods to measure underlying cognitive and social processes?

II. Six Perceived Crises: Diagnosis → Prescription?  

14

Terminology: Perceived Crises and Crossroads:

Crisis = Experiences do not match current/prior expectations

e.g., Schlesinger (1957), Nixon (1962), Deming (1986)

Related: “Crossroads” - Small et al. (2019), Abowd (2021)

Implies path dependence and impact on societal norms 

Analyze impact → Options for changes in design (writ large)

III.  One Dozen Opportunities: Structure of “Crises”? (1)

15

Perceived value for stakeholders (often via concrete use cases) 

will depend on target parameters   and “performance profile”

           =   Design vector (targeted resource decisions)

	Environment (observed, uncontrolled) 

	e = residual effects (uncontrolled, unobserved)

	 = parameters of performance profiles, dispersion

III. One Dozen Opportunities: Structure of “Crises”? (2)

16

Conjecture: Characterize “crises” – connections with changes in: 

1.  Environment   :  different point in space → stability of ? 

2.  Constraints on design factors  , performance profiles 

	 (e.g., reduced variance, cost, operational risk) 

3.  Functions for performance (quality, risk, cost)   or 

     stakeholder value  (depends on and environment ) 

Suggestions for mitigation (and improvement) through changes in 

In principle, each of these possibilities could help to explain each of our six perceived crises.  

And next, we will explore some of those cases in additional depth.  

16

III. Opportunities: Improve Understanding of Procedures

17

The market: Who are the key stakeholders and what are their statistical information needs?

	Set of estimands ,  and stakeholders 

     Conditional (on  distributions of

	(cf. variability of quality over small domains)

17

III. Opportunities: Improve Understanding of Procedures 

18

B. Realistic schematic and empirical descriptions of our   

     stakeholder utility functions?   Concrete use cases?

Extend (A): Conditional distributions of utility functions ?  Dominant terms , , , ?

Methodological question:  Empirical information on stakeholder utility through Bayesian elicitation methods, 

e.g., Garthwaite et al. (2005), O’Hagan et al. (2006)?

18

III. Opportunities: Improve Understanding of Procedures 

19

C. What are realistic schematic models for our production 	

	function and for our performance dimensions  , 

	including quality, risk and cost?  

	Also: To what extent can we quantify these models?

	Dominant (global) factors?  Local approximations? 

	Cf. “rational groupings” in formation of variance function 	models

19

III. Opportunities: Improve Understanding of Procedures 

20

D. Finding the best levers: 

	Which design factors  can we control? 

	How well can we really control them, at what cost, and 

	within what operating constraints? Characterize effects of:

	- Slippage issues

	- Distinctions between one-time experimental results and 

		robust production-level performance at scale 

20

III. Opportunities: Improve Understanding of Procedures 

21

Roles of “public stewardship” and “public goods” requirements?

	Public expectations on specific high-profile statistical information

 	- Groshen (2018), Hess and Ostrom (2006), Rolland (2017) 

	   Summers (2016), Taylor (2016), Teoh (1977) and Trivellato (2017)

     → Quantify: Environmental variables Z  for V ? Constraints on  or  

	Pattern of statistical information developed in response to specific 	high-profile needs, e.g., Hughes-Cromwick and Coronado (2019)

21

III. Opportunities: Improve Understanding of Procedures 

22

What do we not know, and how do we learn more?

	Right choices for crucial dimensions of quality, risk, cost, 	stakeholder value, design factors, environmental 	conditions?   

	Realistic framing of decisions based on this information?  

	Quality of information required for realistic decision?

22

III. Opportunities: Improve Design of Procedures 

23

Change the target stakeholder groups and product mix:

	Areas consistent with positive-sum public stewardship?

	Realistic trajectories for changes in usage patterns

	Severe asymmetries in perceived utility effects:

     Addition of new data series vs. loss of previous series

	Methodological note:  Extend elicitation methods 

	to evaluate gains and losses? (O’Hagan et al., 2006)

III. Opportunities: Improve Design of Procedures 

24

Align commitments on quality/risk/cost profiles with operating 

      constraints and stakeholder priorities (NASEM, 2021)

	Ex: Priorities related to classes of inferential questions:

	- Simple descriptive statements (e.g., means, totals)

     - Association (correlation; contingency tables)

	- Satisfactory predictive models? – Congressional Budget Office

	- Causality (e.g., Imbens and Rubin, 2016)

	- Beyond causality to outright control? (cf. Goroff, 2020; 

        Foundations of Evidence-Based Policymaking Act of 2018)

III. Opportunities: Improve Design of Procedures 

25

Move the lever: 

      Incremental changes of settings of design factors 

	Methodological question: 

	Designs to capture realistic information on

	- Constraints and costs of specified prospective changes

	- Risks incurred with specified changes & options to 

	   mitigate?

→ Extend usual evolutionary operation & adaptive designs

III. Opportunities: Improve Design of Procedures 

26

Improve the lever: 

	More refined control over design factors, accounting for 	slippage issues and adaptive design options

	Methodological note:

	Characterize and measure ways in which adaptive, 	responsive and agile procedures substantially improve 	quality, or reduce costs and risks

III. Opportunities: Improve Design of Procedures 

27

Produce more fundamental change: 

	Add entirely new design factors, with fundamentally changed 	cost structures, quality effects and risk profiles 

      Ex: More administrative records; enhanced online surveys

	Major change in functional form or parameters of 

	performance profile :  Need extensive exploratory work

	“Disruptive innovation” ≠  “Haphazard”

III. Opportunities: Improve Design of Procedures 

28

F. Improve stakeholder communication and negotiation

	Reality check: Zones for clarity, consensus & limitations

	- Practical measures of performance profile    truly aligns 

	   with key stakeholder value function   ?

	- Communicate alignment – and uncertainties – to resonate 

	   deeply with stakeholders?  “Connect numbers with stories”

IV. Closing Remarks 

29

Six Perceived Crises

Framing Through Schematic and Empirical Models 

	for Quality, Risk, Cost, Stakeholder Value 

	and Related Constraints

C.  One Dozen Opportunities – Research & Operations

Improve Understanding of Environment & Procedures

Improve Design of Our Procedures

30

Thank You!

References:

31

Abowd, John M. (2021).  Official Statistics at the Crossroads: Data Quality and Access in an Era of Heightened Privacy Risk.  The Survey Statistician 83, 23-26.  

Abowd, John M. and Ian M. Schmutte (2019) “An Economic Analysis of Privacy Protection and Statistical Accuracy as Social Choices,” American Economic Review, Vol. 109, No. 1 (January):171-202, DOI:10.1257/aer.20170627. 

Bates, Nancy, Monica J. Wroblewski, and Joanne Pascale (2012).  Public Attitudes Toward the Use of Administrative Records in the U.S. Census: Does Question Frame Matter?  Proceedings of the 2012 FCSM Conference.  Available through: https://nces.ed.gov/FCSM/pdf/Wroblewski_2012FCSM_III-A.pdf

Bauer, Paul C., Florian Keusch and Frauke Kreuter (2019). Trust and Cooperative Behavior: Evidence from the Realm of Data-Sharing.  PLOS One.  https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0220115

Binder, David A. (1983).  On the Variances of Asymptotically Normal Estimators from Complex Surveys.  International Statistical Review 51, 279-292.

32

Box, George E.P. and N.R. Draper (1969).  Evolutionary Operation: A Statistical Method for Process Improvement.  New York: Wiley

Buchanan, Richard (1992).  Wicked Problems in Design Thinking.  Design Issues, 8 (2), 5-21

Cho, Moon Jung, Eltinge, John L., Gershunskaya, Julie and Huff, Larry (2014).  “Evaluation of Generalized Variance Functions in the Analysis of Complex Survey Data.”  Journal of Official Statistics, 30, 63-90. 

Citro, Constance F. (2014).  From Multiple Modes for Surveys to Multiple Sources for Estimates.  Survey Methodology Journal 40, 137-161.

Clark, Cynthia Z.F. (2020).  COPAFS-Hosted Tiered Access Workshops.  Presentation to the Council of Professional Associations on Federal Statistics, March 6, 2020.  Available through: https://copafs.org/wp-content/uploads/2020/03/CLARK-COPAFS-hosted-Tiered-Access-Workshops-rev.pdf

Cochran,  W.G.  (1977).  Sampling  Techniques,  Third Edition,  New  York: Wiley. 

Deming, W. Edwards (1986).  Out of the Crisis.  Cambridge, Massachusetts: Massachusetts Institute of Technology Center for Advanced Engineering Study.

Dillman, Don A. (1996). Why Innovation is Difficult in Government Surveys (with discussion).  Journal of Official Statistics, 12, 113-197.  

33

Dinur, I. and Nissim, K. (2003). Revealing information while preserving privacy,  Proceedings of the Twenty-second ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, PODS ’03, ACM, New York, NY, USA, pp. 202–210.

Efron, Bradley (2020) Prediction, Estimation, and Attribution, Journal of the American Statistical Association, 115:530, 636–655, DOI: 10.1080/01621459.2020.1762613.  

Fobia, A.C. J. Holzberg, C. Eggleston, J. Hunter-Childs, J. Marlar and G. Morales (2019).  Attitudes Towards Data Linkage for Evidence-Based Policymaking.  Public Opinion Quarterly 83 (S1), 264-279.

Fuller, W.A. (1975). Regression analysis for sample survey. Sankhyā, Series C, 37, 117­-132.

Fuller, W.A. (2009).  Sampling Statistics.  New York: Wiley. 

Garthwaite, Paul H., Kadane, Joseph B. and O'Hagan, Anthony (2005). Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100, 680–701. 

34

Goroff, Daniel (2020).  Data Dreams and the Everyday Economics of Evidence, Inference and Governance.  ASA Links Award Lecture, December 8, 2020.  Available through:  https://www.amstat.org/ASA/Your-Career/Awards/Links-Lecture-Award.aspx

Groshen, Erica L. (2018).  Views on advanced economy price and wage-setting from a reformed central bank researcher and national statistician.  Proceedings of the Conference on Price and Wage-Setting in Advanced Economies, ECB Forum on Central Banking, pp. 267-283.  Available through:  https://www.ecb.europa.eu/pub/pdf/sintra/ecb.forumcentbank201810.en.pdf

Groves, Robert M. (1989). Survey Errors and Survey Costs.  New York: Wiley.  

Groves, Robert M. and S.G. Heeringa (2006).  Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs.  Journal of the Royal Statistical Society, Series A 169, 439-457.  

Hansen, Morris H., William N. Hurwitz and William G. Madow (1953).  Sample Survey Methods and Theory, Volumes 1 and 2.  New York: Wiley.  

Hess, Charlotte and Elinor Ostrom, (eds)  (2006).  Understanding Knowledge as a Commons: From Theory to Practice.  Cambridge, Massachusetts: MIT Press.

35

Hunter-Childs, Jennifer, Fobia, Aleia C., King, R. and Morales, G. (2019). Trust and Credibility in the U.S. Federal Statistical System. Survey Methods: Insights from the Field. Survey Insights: Methods from the Field. Retrieved from https://surveyinsights.org/?p=10663 DOI:10.13094/SMIF-2019-00001

Imbens, Guido W. and Donald B. Rubin (2015).  Causal Inference for Statistics, Social and Biomedical Sciences: An Introduction.  New York: Cambridge University Press.  

Ioannidis, John P.A. (2005).  Why Most Published Research Findings Are False.  PLOS Medicine 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124

Jordan, Michael I. (2019).  Artificial Intelligence – The Revolution Hasn’t Happened Yet. (with discussion and rejoinder).  Harvard Data Science Review 1 (1).  https://doi.org/10.1162/99608f92.f06c6e61

Keiding, Niels and Thomas A. Louis (2016).  Perils and Potentials of Self‐Selected Entry to Epidemiological Studies and Surveys (with discussion).  Journal of the Royal Statistical Society, Series A 179, 319-376.  https://doi.org/10.1111/rssa.12136

Kish, Leslie (1965).  Survey Sampling.  New York: Wiley.

Lindberg, T., Köppen, E., Rauth, I. & Meinel, C. (2012). On the Perception, Adoption and Implementation of Design Thinking in the IT Industry. In H. Plattner, C. Meinel and L. Leifer (Eds.), Design Thinking Research. Studying Co-Creation in Practice (229-240). Berlin: Springer.  

36

Meng, Xiao-Li (2018).  Statistical Paradises and Paradoxes in Big Data (I): Law of Large Populations, Big Data Paradox and the 2016 U.S. Presidential Election.  Annals of Applied Statistics 1-42.  

National Academies of Sciences, Engineering, and Medicine (2017). Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps. Washington, DC: The National Academies Press. https://doi.org/10.17226/24893.

National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303

National Academies of Sciences, Engineering, and Medicine (2021). Principles and Practices for a Federal Statistical Agency: Seventh Edition. Washington, DC: The National Academies Press. https://doi.org/10.17226/24810

Neyman, J. (1934).  On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection.  Journal of the Royal Statistical Society 97, 558-625.

Nixon, R.M. (1962).  Six Crises. New York: Simon and Schuster.

37

O’Hagan, A., C.E. Buck, A. Daneshkhah, J.R. Eiser, P.H. Garthwaite, D.J. Jenkinson, J.E. Oakley and T. Rakow (2006).  Uncertain Judgements: Eliciting Experts' Probabilities.  Chichester: Wiley. 

Pew Research Center (2019).  Americans and Privacy: Concerned, Confused and Feeling Lack of Control Over Their Personal Information.  November, 2019.  Available through:  https://www.pewresearch.org/internet/2019/11/15/americans-and-privacy-concerned-confused-and-feeling-lack-of-control-over-their-personal-information/

Rigby, Darrell K., Jeff Sutherland and Hirotaka Takeuchi (2016).  The Secret History of Agile Innovation.  Harvard Business Review https://hbr.org/2016/04/the-secret-history-of-agile-innovation

Rittel, H. W. J. &  Webber, M. M. (1973). Dilemmas in a General Theory of Planning, Policy Sciences, 4, 155-169.

Rosenblum, Michael, Peter Miller, Benjamin Reist, Elizabeth Stuart, Michael Thieme and Thomas Louis (2019).  Adaptive Design in Surveys and Clinical Trials: Similarities, Diffferences, and Opportunities for Cross-Fertilization.  Journal of the Royal Statistical Society, Series A 

Schlesinger, Jr., Arthur M. (1957). The Age of Roosevelt: The Crisis of the Old Order, 1919-1933.  New York: Houghton Mifflin.  

38

Simon, H. A. (1956). "Rational Choice and the Structure of the Environment."  Psychological Review 63 (2), 129–138.  

Small, D., D. Banks, B. Yu, X. He, M. Jordan, D. Madigan, M. Markatou (2019).  Statistics at a Crossroads: Who Is for the Challenge?  Presentations at JSM 2019 Late-Breaking Invited Session.  Materials available through: 

https://ww2.amstat.org/meetings/jsm/2019/onlineprogram/AbstractDetails.cfm?abstractid=307986

Stodden, V, F. Leisch and R.D. Peng (2014).  Implementing Reproducible Research.  London: CRC Press

Summers, L. (2016).  The Future of Price Statistics.  Available through: http://larrysummers.com/2016/04/01/world-bank-price-stats/

Taylor L. (2016).  The Ethics of Big Data as a Public Good: Which Public? Whose Good? Philosophical Transactions of the Royal Society, A 374: 20160126.  http://dx.doi.org/10.1098/rsta.2016.0126

Teoh, Siew Hong (1997).  “Information Disclosure and Voluntary Contributions to Public Goods.” RAND Journal of Economics 28, 385-406.

Thienen, J. P. A. von, Meinel, C. & Nicolai, C. (2014). How design thinking tools help to solve wicked problems. In H. Plattner, C. Meinel and L. Leifer (eds.), Design thinking research. Building innovation eco-systems (97-102). Berlin: Springer.

39

Tourangeau, Roger (2017).  Presidential Address, American Association for Public Opinion Research.  May 19, 2017, New Orleans, Louisiana. 

Trivellato, Ugo (2017) : Microdata for Social Sciences and Policy Evaluation as a Public Good, IZA Discussion Papers, No. 11092, Institute of Labor Economics (IZA), Bonn.  Available through: https://www.econstor.eu/bitstream/10419/174002/1/dp11092.pdf

Valliant, Richard (1987). Generalized Variance Functions in Stratified Two-Stage Sampling. Journal of the American Statistical Association 82, 499-508.  

Vilhuber, Lars (2018).  Reproducibility and Replicability in Economics.  White paper prepared for the National Academies' Committee on Reproducibility and Replicability in Science.  

Wasserstein, Ronald L. and Nicole A. Lazar (2016). The ASA Statement on p-Values: Context, Process, and Purpose, The American Statistician, 70:2, 129 133,  DOI: 10.1080/00031305.2016.1154108https://amstat.tandfonline.com/doi/full/10.1080/00031305.2016.1154108#.X9P1ifk3nIU

# image2.png

# image3.png

# image4.png

# image5.png

# image6.png

# image7.png

# image8.png

# image9.png

# image10.png

# image11.png

# image1.png