FCSM Best Practices for Nonresponse Bias Reporting

Morgan Earp (US National Center for Health Statistics)

Jennifer Madans (US National Center for Health Statistics)

Stephen Blumberg (US National Center for Health Statistics)

Elise Christopher (US National Center for Education Statistics)

Tala Fakhouri (US Food and Drug Administration) 

Kathryn Downey Piscopo (US Substance Abuse and Mental Health Services Administration) 

Joseph Schafer (US Census Bureau) 

Robert Sivinski (US Office of Management and Budget) 

Jenny Thompson (US Census Bureau)

Official Disclaimers

The best practices and guidelines presented in this session are an interim product. 

Target final version:  March 2022

The FCSM welcomes comments

Any views expressed on statistical issues or operational procedures are those of the authors and not necessarily those of the U.S. Census Bureau. The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (Approval ID:CBDRB-FY22-ESMD001-002)

The Best Practices and Guidelines

Provide a common framework for reporting and disseminating nonresponse bias analysis research conducted by federal agencies

General language designed to cater to the variety of surveys administered by federal agencies

Discuss what should be included in a nonresponse bias analysis report 

Provides basic information on nonresponse bias analysis methods 

Provides references where additional information can be found

Not a primer on methods of nonresponse bias analysis

Purpose of presentation

Introduce (proposed) best practices and guidelines

Illustrate selected guidelines using fictional examples

Conceptual/clarifying

Not intended as prototypes 

Not intended to serve as examples for style, formatting, etc.

Obtain informed feedback on (proposed) best practices and guidelines

FCSM Best Practices for Nonresponse Bias Reporting

Best Practice 1: Describe the survey that is the subject of the nonresponse bias analysis

Best Practice 2: Provide unit response rates for the survey and discuss potential for nonresponse bias

Best Practice 3: Describe the plan for evaluating and quantifying nonresponse bias and for any mitigation strategies to be employed

Best Practice 4: Describe and justify all available sources of auxiliary data used in the nonresponse bias analysis

Best Practice 5: Describe results of nonresponse bias analysis for all key survey items

Best Practice 6: Summarize the major conclusions of the analyses

Best Practice 7: Discuss recommendations for data collection methods and adjustment strategies to mitigate nonresponse bias in future waves of data collection

Best Practice 1:  Describe the surveyBest Practice 1:  Describe the Survey the survey

Guidelines

1.1. Describe the survey target population

1.2. Describe the survey sample frame

1.3.  Discuss the potential for coverage error of the frame

1.4.  Describe the survey sample design

1.5.  Describe the sample unit

1.6.  Describe the survey data collection modes

1.7. Describe the survey’s key survey items, and identify  those that will be used in the nonresponse bias analysis

1.8.  Describe any nonresponse mitigation strategies employed during data collection

Provide detailed information on the target population, survey unit, sample design, and data collection modes, as well as the sponsoring agency, the year the study was conducted, and the years of survey data used for the analyses. 

Checklist for NRB Analysis Report

Survey description included

Sponsoring agency acknowledged

Year of study provided

Years of survey data used provided

Best Practice 1: Fictional Example 1Time Use Survey (Students) – 2019
|                                                                   |                                                                                                                                                                        |
|-------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Target Population                                                 | High school students (grades 9-12)                                                                                                                                     |
| Sampling frame                                                    | Listings of public and private high schools from state governments Registries within sampled schools                                                                   |
| Frame coverage issues                                             | Home schooled studentsTransfer students/dropouts                                                                                                                       |
| Sample design                                                     | Two stage cluster sample:PPS selection of schools within state, stratified by public versus private statusSystematic sample of students within grade in sample schools |
| Sample unit                                                       | PSU: High schools USU: Student                                                                                                                                         |
| Data collection modes                                             | Personal interview, Telephone follow-up, Web collection                                                                                                                |
| Key items                                                         | Time usage: classroom, homework, sports & leisure activities (group), social media, other                                                                          |
| Nonresponse mitigation strategies employed during data collection | Field Experiment:  Incentive offered ($10 VISA card) for students in historically low-responding schools                                                               |

Only report mitigation activities that include an evaluation of bias reduction. 

Best Practice 1: Fictional Example 2Athletic Shoe Sales (Stores) - 2019
|                                                                   |                                                       |
|-------------------------------------------------------------------|-------------------------------------------------------|
| Target Population                                                 | Stores that sell athletic shoes                       |
| Sampling frame                                                    | Business register                                     |
| Frame coverage issues                                             | Misclassification on frameNew stores (births)         |
| Sample design                                                     | One-stage stratified simple-random sample             |
| Sample unit                                                       | (Business) establishments (Stores)                    |
| Data collection modes                                             | Web collection (primary); telephone follow-up         |
| Key items                                                         | Total revenue, total revenue from athletic shoe sales |
| Nonresponse mitigation strategies employed during data collection | Not reported (no special evaluations or experiments)  |
|                                                                   |                                                       |
|-------------------------------------------------------------------|-------------------------------------------------------|
| Target Population                                                 | Stores that sell athletic shoes                       |
| Sampling frame                                                    | Business register                                     |
| Frame coverage issues                                             | Misclassification on frameNew stores (births)         |
| Sample design                                                     | One-stage stratified simple-random sample             |
| Sample unit                                                       | (Business) establishments (Stores)                    |
| Data collection modes                                             | Web collection (primary); telephone follow-up         |
| Key items                                                         | Total revenue, total revenue from athletic shoe sales |
| Nonresponse mitigation strategies employed during data collection | Not reported (no special evaluations or experiments)  |

Best Practice 2:  Provide unit response rates for the survey and discuss potential for nonresponse bias

Guidelines

Guideline 2.1.  Identify and report unit response rates based on weights that adjust for selection probabilities

Guideline 2.2.  Report response rates for regions or key subgroups for which national estimates are published

Guideline 2.3.  Report response rates for all key survey items described in Guideline 1.7

Guideline 2.4.  Report response rates for each sampling stage that provides actionable information on survey participation and/or coverage

Guideline 2.5.  Report response rates for each wave of data collection in longitudinal surveys

Response rates are unit-level performance metrics that measure the proportion of the eligible sample units that responded to survey or census. 

Checklist for NRB Report

Method of calculating response rates provided. 

Response rates are reported for the survey (total) and by subdomains of interest for all study years

Guideline 2.6.  For cases where responsive design or treatment groups for other experiments designed to reduce bias were used during data collection, report response rates for each of the treatment groups used

Guideline 2.7.  Discuss the potential for nonresponse bias given unweighted and weighted response rates, response rates by sampling stage, waves of data collection, and key subgroups

Best Practice 2: Fictional Example 1 (Part 1)Time Use Survey

Key Items:

Time spent in classroom (classes)

Time spent on homework

Time spent on sports & leisure activities

Time spent on social media

Key subdomains

Region (NE, MW, S, W)

Grade (9, 10, 11, 12)
|        | Unit Response Rate | Unit Response Rate | Weighted Item Response Rates |          |                      |              |
|--------|--------------------|--------------------|------------------------------|----------|----------------------|--------------|
|        | Unweighted         | Weighted           | Classroom                    | Homework | Sports & Leisure | Social Media |
| Total  | 64                 | 70                 | 70                           | 75       | 30                   | 15           |
| Region |                    |                    |                              |          |                      |              |
| NE     | 82                 | 85                 | 85                           | 83       | 25                   | 12           |
| MW     | 66                 | 68                 | 66                           | 68       | 20                   | 14           |
| S      | 29                 | 45                 | 43                           | 45       | 27                   | 9            |
| W      | 60                 | 71                 | 70                           | 71       | 35                   | 7            |
| Grade  |                    |                    |                              |          |                      |              |
| 9      | 73                 | 87                 | 87                           | 80       | 80                   | 76           |
| 10     | 71                 | 79                 | 78                           | 74       | 68                   | 73           |
| 11     | 70                 | 72                 | 72                           | 74       | 60                   | 58           |
| 12     | 22                 | 35                 | 34                           | 35       | 30                   | 15           |

Guideline 2.2

Guideline 2.3

Guideline 2.7 
|        | Unit Response Rate | Unit Response Rate | Weighted Item Response Rates |          |                      |              |
|--------|--------------------|--------------------|------------------------------|----------|----------------------|--------------|
|        | Unweighted         | Weighted           | Classroom                    | Homework | Sports & Leisure | Social Media |
| Total  | 64                 | 70                 | 70                           | 75       | 30                   | 15           |
| Region |                    |                    |                              |          |                      |              |
| NE     | 82                 | 85                 | 85                           | 83       | 25                   | 12           |
| MW     | 66                 | 68                 | 66                           | 68       | 20                   | 14           |
| S      | 29                 | 45                 | 43                           | 45       | 27                   | 9            |
| W      | 60                 | 71                 | 70                           | 71       | 35                   | 7            |
| Grade  |                    |                    |                              |          |                      |              |
| 9      | 73                 | 87                 | 87                           | 80       | 80                   | 76           |
| 10     | 71                 | 79                 | 78                           | 74       | 68                   | 73           |
| 11     | 70                 | 72                 | 72                           | 74       | 60                   | 58           |
| 12     | 22                 | 35                 | 34                           | 35       | 30                   | 15           |
|        | Unit Response Rate | Unit Response Rate | Weighted Item Response Rates |          |                      |              |
|--------|--------------------|--------------------|------------------------------|----------|----------------------|--------------|
|        | Unweighted         | Weighted           | Classroom                    | Homework | Sports & Leisure | Social Media |
| Total  | 64                 | 70                 | 70                           | 75       | 30                   | 15           |
| Region |                    |                    |                              |          |                      |              |
| NE     | 82                 | 85                 | 85                           | 83       | 25                   | 12           |
| MW     | 66                 | 68                 | 66                           | 68       | 20                   | 14           |
| S      | 29                 | 45                 | 43                           | 45       | 27                   | 9            |
| W      | 60                 | 71                 | 70                           | 71       | 35                   | 7            |
| Grade  |                    |                    |                              |          |                      |              |
| 9      | 73                 | 87                 | 87                           | 80       | 80                   | 76           |
| 10     | 71                 | 79                 | 78                           | 74       | 68                   | 73           |
| 11     | 70                 | 72                 | 72                           | 74       | 60                   | 58           |
| 12     | 22                 | 35                 | 34                           | 35       | 30                   | 15           |

Best Practice 2:  Fictional Example 1 (Part 2)Time Use Survey

Two-stage sample

Schools within region

Public and Private

NE, MW, S, W

Students within school
|         | Stage 1:  Schools | Stage 2:  Students|School |
|---------|-------------------|---------------------------|
| Total   | 86                | 70                        |
| NE      | 90                | 85                        |
| Public  | 95                | 71                        |
| Private | 93                | 80                        |
| MW      | 75                | 68                        |
| Public  | 79                | 75                        |
| Private | 63                | 62                        |
| S       | 60                | 45                        |
| Public  | 61                | 35                        |
| Private | 23                | 50                        |
| W       | 80                | 71                        |
| Public  | 79                | 70                        |
| Private | 81                | 72                        |

Guideline 2.4
|         | Stage 1:  Schools | Stage 2:  Students|School |
|---------|-------------------|---------------------------|
| Total   | 86                | 70                        |
| NE      | 90                | 85                        |
| Public  | 95                | 71                        |
| Private | 93                | 80                        |
| MW      | 75                | 68                        |
| Public  | 79                | 75                        |
| Private | 63                | 62                        |
| S       | 60                | 45                        |
| Public  | 61                | 35                        |
| Private | 23                | 50                        |
| W       | 80                | 71                        |
| Public  | 79                | 70                        |
| Private | 81                | 72                        |

High nonresponse! Targeted nonresponse follow-up? Experiment?
|         | Stage 1:   Schools | Stage 2:  Students|School |
|---------|--------------------|---------------------------|
| Total   | 86                 | 70                        |
| NE      | 90                 | 85                        |
| Public  | 95                 | 71                        |
| Private | 93                 | 80                        |
| MW      | 75                 | 68                        |
| Public  | 79                 | 75                        |
| Private | 63                 | 62                        |
| S       | 60                 | 45                        |
| Public  | 61                 | 35                        |
| Private | 23                 | 50                        |
| W       | 80                 | 71                        |
| Public  | 79                 | 70                        |
| Private | 81                 | 72                        |

Best Practice 2:  Fictional Example 2Athletic Shoe Sales Survey
|                               | Unit Response Rate |          | Weighted Item Response Rates |            |
|-------------------------------|--------------------|----------|------------------------------|------------|
|                               | Unweighted         | Weighted | Total Sales                  | Shoe Sales |
| All Stores                    | 60                 | 81       | 85                           | 82         |
| Sporting Goods Stores (45111) | 55                 | 70       | 81                           | 80         |
| Department Stores(45221)      | 65                 | 86       | 92                           | 75         |

Include a Measure-of-Size variable to account for skewed populations

Low response from smaller units

High response from largest units

Best Practice 3: Describe the plan for evaluating and quantifying nonresponse bias and for any mitigation strategies to be employed

Guidelines

Guideline 3.1. Describe and justify all nonresponse bias evaluation methods used

Guideline 3.2.  Describe the analysis plan for comparing and assessing nonresponse bias levels across key stages and subgroups as described by Guidelines 2.2 to 2.6.

Guideline 3.3.  Describe post data collection nonresponse bias mitigation strategies that were employed and how they might reduce nonresponse bias 

The report should include a complete discussion of all evaluation methods used, including the rationale for the selection of each method along with an explanation of why some standard methods are not used.  The discussion should address the aspect of nonresponse that the method will address. 

Checklist for NRB Report

Description of activities used to reduce nonresponse bias during data collection included

The analytic plan for evaluating nonresponse bias in the final data file provided

The post-data collection mitigation strategies (if applicable) described

Guideline 3.1 Direct Versus Indirect Analysis

Direct Assessment Methods

Use (micro)data from the data collection 

sampling frame

matched auxiliary data

“Sophisticated” analyses at survey level and by subdomains

Alternative weighting methods

Regression trees

R-indicators

Balance and distance indicators

Proxy-pattern mixture models

Indirect Assessment Methods

Compare estimates from the studied program to similar estimates from other sources

Assume the “similar” other source estimates have high accuracy and precision 

May be confounded by differences in definition and timing (among others)

Analyses limited to same level of aggregation

Best Practice 3:  Fictional Example 1Time Use Survey (Indirect Analysis)

Our (fictional) survey

Limited to grades 9-12 (approximate ages 14-18)

Limited available estimates from American Time Use Survey 

Civilian noninstitutional population 15+

Different subdomains

Best Practice 3:  Fictional Example 2Athletic Shoe Sales Survey Analysis Plan
| Evaluation method                                                                                           | Data                                                                       | Objective/Interpretation                                      |
|-------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|---------------------------------------------------------------|
| Compare estimates of total sales in Sporting Goods Stores (NAICS 45111) and Department Stores (NAICS 45221) | Athletic Shoe Sales Survey (Subject)Annual Retail Trade Survey (Benchmark) | Comparable levels may indicate low impact of nonresponse bias |
| Evaluation method                                                                                           | Data                                                                              | Objective/Interpretation                                                                                                                                                                                        |
|-------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Compare estimates of total sales in Sporting Goods Stores (NAICS 45111) and Department Stores (NAICS 45221) | Athletic Shoe Sales Survey (2021) (Subject)Annual Retail Trade Survey (Benchmark) | Comparable levels may indicate low impact of nonresponse bias                                                                                                                                                   |
| Balance and Distance Indicators using linked survey response data and frame measure-of-size                 | Athletic Shoe Sales Survey (2021) Business Register (sampling frame)              | Assess existence and extent of systematic differences in composition of full sample and respondent sample (balance)between respondents and nonrespondents in full sample using proxy (measure-of-size) variable |
| Evaluation method                                                                                           | Data                                                                              | Objective/Interpretation                                                                                                                                                                                                   |
|-------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Compare estimates of total sales in Sporting Goods Stores (NAICS 45111) and Department Stores (NAICS 45221) | Athletic Shoe Sales Survey (2021) (Subject)Annual Retail Trade Survey (Benchmark) | Comparable levels may indicate low impact of nonresponse bias                                                                                                                                                              |
| Balance and Distance Indicators using linked survey response data and frame measure-of-size                 | Athletic Shoe Sales Survey (2021) Business Register (sampling frame)              | Assess existence and extent of systematic differences in composition of full sample and respondent sample (balance)between respondents and nonrespondents in full sample using proxy (measure-of-size) variable (distance) |
| Fraction of Missing Information (FMI) from Proxy Pattern-Mixture Model Analysis for Athletic Shoe Sales     | Athletic Shoe Sales Survey (2021 and 2022)Business Register (sampling frame)      | Assess effects of nonresponse on nonresponse adjusted estimates of athletic shoe sales under alternative response mechanisms                                                                                               |

Indirect analysis – potential evidence of credibility

Direct analysis with linked data – potential evidence of missing-at-random response mechanism

Direct analysis with linked data – potential justification of imputation method for key item (and assesses robustness)

Sampling/nonsampling errors in both sources

Does not assess NR bias effect on key item

Does not assess NR bias effect on key item

Measure-of-size variable may be out of date

Relies on potentially weak imputation model

Best Practice 4:  Describe and justify all available sources of auxiliary data used in the nonresponse bias analysis

Guidelines

Guideline 4.1.  Describe all sources of auxiliary data used for the nonresponse bias analysis

Guideline 4.2.  Discuss the characteristics, quality, and completeness of auxiliary data in terms of coverage, item missingness, and measurement error, and timeliness

Guideline 4.3.  Discuss the relationship between auxiliary data being used to evaluate nonresponse bias and the key survey items

The type of nonresponse bias analysis as well as the type of inferences that can be drawn depend on what auxiliary sources are available at the time of analysis. The availability of auxiliary data will impact the methods selected, how the analysis is done, and the interpretation of the results. 

Checklist for NRB Report

All key data items should be associated with one or more sources of auxiliary data

List key data items for which auxiliary data are not available (limitation in study)

Descriptions for each source of auxiliary data include

Overlap with study program e.g., timing, coverage

Quality assessments e.g., measurement error, sampling error, nonresponse error

A Few Things to Look For in Auxiliary Data

Missingness 

Extent of missing data by data item

Assess whether auxiliary data are missing at random 

Extent of imputation 

Coverage 

Overlap of auxiliary data population and survey target population

Linkage rates for microdata

Definitional differences survey and auxiliary data

Timeliness 

Time difference (survey and auxiliary data)

Determine when auxiliary data were last updated

Accuracy 

Comparisons of matched survey and auxiliary data on same variables

Sampling error measurements (or coefficients of variation) of auxiliary data

Association with survey data 

Correlation or regression analysis  

Best Practice 5: Describe results of nonresponse bias analysis for all key survey items

Guidelines

Guideline 5.1. Provide and discuss the results of the nonresponse bias analyses specified in the methods section, including all identified key items and evaluation methods (see Guideline 1.7 and 3.1) across all key stages and subgroups (see Guideline 3.2)

Guideline 5.2.  Describe the impact of post data collection mitigation strategies on reducing nonresponse bias 

Guideline 5.3.  Describe nonresponse bias before and after adjustment across key stages and subgroups as described by Guideline 2.2 to 2.6. 

The results of the nonresponse bias analysis should be reported for each method used in terms of the extent of bias identified, even if the results were inconclusive.  If mitigation strategies are implemented or adjustment methods were applied, the results of those strategies on bias indicators and key survey items should also be reported. 

Checklist for NRB Report

Reporting of the results of the analysis refers to the analytic plan and addresses the key items for the survey by stage and subgroup and identified indicators of bias. 

All analyses reported (even if inconclusive)

Assessment of mitigation strategies included, overall and by applicable sampling stage or wave.

Best Practice 5: Fictional Example 1Annual Shoe Sales Survey

Comparison to benchmark estimate (Total Sales)

Balance and distance measures 

(Measure-of-size = 2017 Sales)

Point estimates from Annual Shoe Sales Survey not statistically different from ARTS estimates

Ideal values:  Balance = 1 and Distance = 0

45111 – Indications of potential NR bias

45221 – No indications of NR bias with these indicators

Best Practice 5: Fictional Example 1 (Cont’d)Annual Shoe Sales Survey

Measure: Fraction of Missing Information (FMI)

Proxy Pattern-Mixture Model Analysis

Assessed nonresponse bias in variable after adjustment (imputation)

FMI  0 indicative of low nonresponse bias

FMI  1 indicative of high nonresponse bias

Variable of Interest:  Total shoe sales

Proxy: Predicted value (all units) from imputation model
| Industry         | FMI (MAR) | FMI (NMAR) | Nonresponse Rate |
|------------------|-----------|------------|------------------|
| 45111(R2 = 0.90) | 0.70      | 0.75       | 45%              |
| 45221(R2 = 0.95) | 0.13      | 0.18       | 39%              |
| Industry         | FMI (MAR) | FMI (NMAR) | Nonresponse Rate |
|------------------|-----------|------------|------------------|
| 45111(R2 = 0.90) | 0.70      | 0.75       | 45%              |
| 45221(R2 = 0.95) | 0.13      | 0.18       | 39%              |

Evidence of NR Bias 

EVEN AFTER IMPUTATION

FMI near 1 (both response mechanisms)

FMI > Nonresponse rate

Best Practice 6: Summarize the major conclusions of the analyses

Guidelines

Guideline 3.1. Summarize the results of the nonresponse bias analysis, any post-data collection mitigation strategies, and the final assessment of potential nonresponse bias for the full data collection and for key indicators

Guideline 6.2.  Discuss the implications and potential causes of contradictory results

Guideline 6.3.  Discuss potential implications as they relate to fitness of purpose

Provide an executive summary that provides high level findings for the full program and for the key survey items, overall and by studied subgroup.  If possible, highlight post-data collection mitigation strategies that appear to be effective, as well as those that are less effective. 

Checklist for NRB Report

Executive summary provided

Assessment of post data collection migration strategies highlighted

Implications of assessments provided

Best Practice 6:  Fictional Example 1Time Use Survey

Key Findings

Low response from sampled schools in the South

Levels of estimated totals vary greatly depending on alternative weighting adjustments, providing indications of nonresponse bias

Low response across-the-board for 12th grade students

Evidence that 12th grade respondents are not a random sample

Higher than (national) average time reported spent on homework by 12th grade students

Less than (national) average time reported spent on sports and leisure activities

Low response across-the-board for reporting time on social media

No benchmarks or auxiliary data available for comparison

Best Practice 6: Fictional example 2Athletic Shoe Sales Survey

Key Findings

Larger businesses more likely to report than smaller businesses

Imputation models may not be adequately correcting for nonresponse bias, as response mechanism is not missing-at-random

Nonresponse bias impacts estimates in from sporting goods stores (NAICS 45111)

Respondents primarily large businesses

FMI near 1 

Caution advised with estimates from this industry, as the analysis indicates that shoe sales may be overestimated (fitness for use)

Best Practice 7: Discuss recommendations for data collection methods and adjustment strategies to mitigate nonresponse bias in future waves of data collection

Guidelines

Guideline 7.1.  Discuss recommendations for modifications to sample design, questionnaires, or data collection strategies that may reduce the bias in collected data. 

Guideline 7.2.  Discuss recommendations that may improve post-data collection adjustment strategies.

A thorough nonresponse bias analysis that follows the previous guidelines should provide useful insight into the likely sources of nonresponse bias. In recurring surveys, this information can be used to improve future data collection. 

Checklist for NRB Report

Recommendations are justified by results provided

Recommendations can include future research or embedded experiments

Best Practice 7:  Fictional Example 1Time Use Survey

Finding:   Low response from sampled schools in the 	South

Finding:	 Low response across-the-board for 12th 	grade students

Finding:  	Low response across-the-board for 	reporting time on social media

Recommendations:

Review the sampling design, focusing on first stage allocations in South

If funds permit, consider nonresponse follow-up study

Recommendation: 

Consider offering incentives if funding permits provided that ongoing pilot test shows positive incentive effects in low-responding schools

Recommendation: 

Do not publish these measures from current collection; fitness for use compromised

High nonresponse indicative of potential issue with question (questionnaire). Conduct focus groups or other cognitive testing, if funding permits

Final Remarks

Reminder that presentation discusses interim report

Comments welcome

In that vein, looking forward to comments from discussant

Presentation today given by a messenger

However, feel free to follow-up 

katherine.j.thompson@census.gov

# image1.png

# image2.png

# image3.png

# image4.png

# image5.png

# image6.svg

# image7.png

# image8.svg

# image9.png