# Presentation: 033981

## Data Processing, Nonresponse and Quality Control

- Dr. Nancy Kirkendall
- Energy Information Administration
- February, 2003
- Kathmandu, Nepal

## Outline

- Processing Error
- Nonresponse and Imputation
- Quality Control Methods
- Evaluating Survey Error

## Minimizing Processing Errors

- To minimize processing errors use edits to identify potential errors
- Contact respondents to correct large errors (those that have an impact on key estimates)

## Data Processing Steps--Paper

**1.****	****Check-in**: questionnaires are collected, time of receipt noted on forms
- 2.	**Scan edit**: entries are inspected to assure legibility
- 3.	**Data entry**: data are captured via keying, scanning or other
- 4.	**Editing**: outliers are identified and corrected; missing items on form may be “imputed.”

## Data Processing Error

- Relatively sparse literature
- Some steps are very error prone 	(e.g., coding and editing)
- Errors are both systematic and 	variable
- Increased automation and integration generates reduced variable error while increasing bias

## Data Capture Errors

- Telephone
- Interviewer usually has on-line edits to check data entry
- Keying errors
- Discovered by verification keying, or  edits during data entry
- Error rates usually small
- The vital few large errors can have large effects on MSE

## Electronic Data Collection

- Fax – errors same as mail
- Email – errors same as mail if data keyed, no keying error if data automatically read into databases
- Internet – with on-line edits so respondents correct data is ideal
- EIA moving to have respondents and data entry clerks use same form with same edits

## Scanning in forms

- Intelligent Character Recognition
- Error types are substitution and rejection
- Substitution errors can be systematic
- Condition of incoming documents and the equipment is crucial

## Keying error rates

- In one study agreement rates for keying closed-ended questions was about 99.7%.  After verification, the keying error rate was about .07%.
- Keying error rates on open-ended questions may be much higher (up to 5%)

## Definition

- Editing - identification of logical inconsistencies and implausible values in individual data
- Computer editing is optimal

## Types of Edits

- Micro editing (at record level)
- Consistency or logical checks, eg. BTU’s in >=BTUs out; zero now, zero before
- Range Edits
- Ratio edits (e.g., net profit/total revenue  must be within a specified range)
- Comparisons to past data
- Balance edits (totals = sum of parts)

## Types of Edits

- Macroediting: Editing at aggregate 	level (needs data from many 	respondents.)
- Usually based on models of relationships between variables, or market share
- Used to identify impact of specific respondent data on estimates

## When Edits are Applied

- Data check-in or just prior to keying (scan editing) Correction of unkeyable responses
- Micro editing – used on data entry
- Warning flags (suspicious entries)
- Fatal flags (definite errors)
- Make sure data base distinguishes between no data (blanks) and reported zeros

## Editing to the point of adding more errors than are being corrected
Granquist and Kovar (1997) 
suggest that 50% of the warning flags have little or no effect on the final estimates.  They advocate using “selective editing” to reduce editing costs.

- Granquist and Kovar (1997)
- suggest that 50% of the warning flags have little or no effect on the final estimates.  They advocate using “selective editing” to reduce editing costs.

- Overediting

- 29oct01

## Selective editing – a fancy term for identifying the most important errors based on
unit size, error size, survey weight, 	importance of the variable, and cost
to decide which warning edits will be 	followed up with respondents.

- unit size, error size, survey weight, 	importance of the variable, and cost
- to decide which warning edits will be 	followed up with respondents.

- Selective Editing

- 29oct01

## Editing and Data Quality

- Engstr****m (1997) found 90% of errors due to comprehension errors
- Questionnaire redesign eliminated the errors
**Go back to section on Measurement Error and follow the guidelines**

## Definition

- Imputation -- Estimation of missing survey information.  May be used to replace data that fail edits.  In EIA used primarily to estimate for nonresponse.

## Types of Imputation

- Model-based imputation
- Deck imputation
- cold deck – external donor
- Hot deck – internal donor
- Weight adjustment (for sample surveys)

## Nonresponse error

## To Minimize Nonresponse 

- Follow-up with nonrespondents that do not provide data
- Send reminder letter
- Followup call(s)
- Send letters to more senior people in company signed by more senior people in your agency

## Two types of nonresponse

- Unit nonresponse – failure for a respondent to send in a form
- Item nonresponse – respondent does not completely fill in form, leaves some items blank.

## Unit nonresponse

- If skewed population and periodic survey, use estimate based on past reported data from respondent as imputed value.
- If nonrespondent similar to respondents, use estimate based on other respondents (eg. Hot deck)
- Error is difference between imputed value and value respondent should have reported.

## Computing response ratesNotation

- I = number completed interviews
- RF= number refusals
- NC=number noncontacts
- NI=number of other interviews not completed
- IE=number of ineligibles

## Response rate

## Example of Survey Response Rates

_**	**__**	**__**Unweighted**_
_**Survey**_**	**_**Year**_**	**_**Rates**_
**Agricultural Resources Management Study****	****1999****	****66.1**
**Agricultural Survey****	****1999****	****80.0**
**Annual Survey of Manufacturers****	****1998****	****72.6**
**Census of Agriculture****	****1997****	****86.2**
**Commercial Building Energy Consumption**
**Survey:  Building Characteristics****	****1995****	****87.5**
**Economic Census****	****1997****	****86.1**
**National Ambulatory Medical Care Survey****	****1998****	****68.0**

- Shimizu, I.M.  (2000)  “Response in Federal Establishment Surveys.”  Paper presented at the International Conference on Establishment Surveys, June 2000, Buffalo, NY.

## Summary of Response Rates and Nonresponse Bias

- Bias is a function of both:
- Response rate
- Difference between respondent data and imputed data.  (Can check by using simulation.)

## Dealing with nonresponse

- Increasing contacts (Number of call backs, call scheduling)
- Increasing cooperation
- Advance letters and information, letters to higher authorities in company
- Adjust estimates (if necessary)
- Impute for missing data using past data or data from other respondents, as appropriate
- Can subsample nonrespondents to determine similarity to respondents

## Both kinds of nonresponse

- Higher if information requested is not available, question is difficult to understand, or data are difficult to find.
- Higher if data requested are considered sensitive.  Some price information is considered sensitive.
- Usually lower if surveys are mandatory

## Quality Control Methods

## Quality Control Methods

- Inspection Methods
- Continuous quality improvement
- Acceptance sampling
- Statistical process control

## Inspection Methods for Data Processing Quality

- 1.	Inspect sample (or entire set) of items in the data base
- 2.	Compare to questionnaires, or contact respondents to verify
- 3.	*Optional:* correct nonconformities
- 4.	Pass or fail
- 5.	*Optional: *provide feedback to operators

- 29oct01

## Problems with Inspection

- Costly
- 100% inspection required for control at very small error levels
- Responsibility for improving quality given to inspectors
- Implies operators are responsible for all errors
- Feedback is usually ineffective

- 29oct01

## Continuous Quality Improvement Strategy

- Use of Pareto principle
- Focus on root causes
- Team approach

- 29oct01

## The Generic Coding Process

**The Generic Coding Process**

## The Continuous Quality Improvement Method

- 							Example
- 1.	Perform coding operation twice and observe disagreements
- 2.	Perform Pareto analysis
- 3.	Meet in teams to identify root causes
- 4.	Implement corrective measures and repeat

- 29oct01

## Example of Pareto Chart

**Example of Pareto Chart**

## Coding Error Rates for Occupation

**Coding Error Rates for Occupation**

## Evaluating Survey Error

## Survey Error Evaluation

- Compare data collection modes or methods
- Optimize allocation of resources
- Error reduction for specific survey processes
- Provide users with information on data quality
- Adjust estimates for nonsampling error

## Record Check Studies

*Objective:*
- 	To obtain the true value of characteristics by accessing administrative or official records (microlevel), or returning to respondent to verify data

- 29oct01

## Time periods may not coincide
Record variable definition and survey variable definition differ
Matching errors
Records not available for all variables and all respondents

- Record variable definition and survey variable definition differ
- Matching errors
- Records not available for all variables and all respondents

- Limitations:

- 29oct01

## Data Comparisons

- Comparisons of related estimates from different sources (macro level)
- Time periods may not coincide
- Definitions may not be the same
- Help to understand strengths and weaknesses of data
- Monitor over time to detect systematic changes

## Some Techniques for Survey Evaluation

| Evaluation Method | Stage | Purpose |
| --- | --- | --- |
| Debriefings
Interviewers group
Respondent focus groups | Post-survey | Evaluate questionnaire and data collection procedures  |

## Some Techniques for Survey Evaluation

| Evaluation Method | Stage | Purpose |
| --- | --- | --- |
| Post-Survey Analysis
Experiments
Comparisons | Post-survey | Compare alternative methods of data collection,estimate MSE components, validate survey estimates |

## Some Techniques for Survey Evaluation

| Evaluation Methods | Stage | Purpose |
| --- | --- | --- |
| Post-survey Data collection
Reinterview
Non-response followup
Record check studies | Post-Survey | Estimate one or more components of  MSE |

## The Balance

- Quality dimensions conflict
- Accuracy vs timeliness
- Accuracy vs relevance
- Comparability vs accuracy
- Cost vs error

## Measuring and Documenting Quality

- Accuracy can be measured
- Other quality dimensions are 	qualitative and can be seen as constraints
- Quality reports
- Performance measures

## Documentation

- Survey processes
- Survey plan, process details
- Quality reports
- Use framework based on quality dimensions
- Report estimates of MSE components
- In absence of MSE component estimates provide indicators of quality
- Implement an evaluation scheme