# Presentation: 408853

## Design a High Quality Energy Survey

- Dr. Ruey-Pyng Lu
- January 2005
- South Asian Regional Initiative –
- Energy

## Exit Polls for 14th Lok Sabha 

- Election 2004

| Media Group | NDA | Cong+ | Others |
| --- | --- | --- | --- |
| Sahara Samay | 263-278 | 92-102 | 171-181 |
| Star News | 263-275 | 174-186 | 86-98 |
| Aaj Tak | 248 | 189 | 105 |
| Zee News | 249 | 117 | 176 |
| NDTV-IE | 230-250 | 190-205 | 100-130 |
| Final Results | 185 | 220 | 137 |

- Most analysts believed the NDA would win the elections:
- this assessment was also supported by opinion polls. But ...

## Types of Survey Error

- Coverage error ---
- Nonresponse error --- the error due to failure of some respondents to submit information.
- Sampling error --- due to interviewing sample rather than entire population
- Measurement error
- Data processing error

## Designing a Survey

- Identify needed information
- Identify the survey population
- Access to the **frame** (e.g. a list of the population entities.)
- Watch for** Frame Errors** (omissions) -- usually contribute to understatement of totals
- Describe attributes of the population by conducting a survey
- A sample design with a probability mechanism?
**Sampling Error** results from estimating at total from a sample

## Collection and Processing

- Identify respondents
- Watch for **Measurement error **
- Design form, send to respondents, collect by mail or telephone
- Enter data into Processing System.
- Watch for **Processing Error**

## Total Survey Error

**Large variance, **
**small bias**

**Small variance,**
**large bias**

## Bias and Variance by Error Source

## A survey from a process perspective

- Define research objectives

- Choose mode Of Collection

- Choose sampling frame

- Construct and pretest
- questionnaire

- Design and
- Select sample

- Recruit and measure sample

- Code and edit data

- Make postsurvey adjustment

- Perform analysis

## Scope of Survey and Frame

- Define what you want to measure, determine  which part of industry is optimal, where data exists in company
- 	-- custody versus equity?
- 	-- company versus establishment?
- 	-- specific person who knows or has data
- 			knowledge versus willingness 			versus authority to cooperate
- 	-- when are data available?

**Notes:**
Work with experts to understand the issues
Work with industry to gain their approval

## Energy Industries Highly Skewed

- 11% of all establishments account for 70% of all business
- Sample design must take frame size (counts) and respondent size (skewness) into account
- Size is an important analysis variable
- May need more than name and address of energy companies – e.g. industry type, geography, size
- Difficult to track “births” and “deaths” of establishments
- When is an entity “in operation”?
- How should mergers be handled?
- Frames get out-of-date

## Types of Frame Errors

-  Coverage errors
- Missing units, duplicates, extraneous units
-  Classification errors
- Industry type (SIC, NAICS)
- Geography
- Size
- Contact errors
- Address incomplete or incorrect
- Contact name incorrect
- Phone number incorrect
- Current frame?

## Collection Mode

- Self-administered questionnaire most common
- Mail
- Electronic (including fax, email)
- Internet
- Telephone interviewing
- For survey with few questions
- For nonresponse followup

## Respondent Process

- Identify appropriate respondent at establishment.
- Send materials to respondent
- Conduct Cognitive test

## A fancy way to say talk to people and use the information they provide 

Work with the person who fills out the form to make sure they understand and have data

- Work with the person who fills out the form to make sure they understand and have data

- Cognitive Methods

**Notes:**
Identify appropriate respondent at establishment.
Send materials to respondent
Respondent knowledge, authority, desire to respond
Respondent’s assignment of priority for this response (voluntary versus mandatory, competing tasks, sensitivity of information)
Respondent’s comprehension of requested information (unit of response, specific data)
Ease of data retrieval from records, number of others involved in getting data
Company policy on response

## Editing and Data Quality

- Engstr****m (1997) found 90% of errors due to comprehension errors
- Questionnaire redesign eliminated the errors
**Go back to section on Measurement Error and follow the guidelines**

## Types of Edits (1)

- Editing - identification of logical inconsistencies and implausible values in individual data
- Micro editing (at record level)
- Consistency or logical checks, eg. BTU’s in >=BTUs out; zero now, zero before
- Range Edits
- Ratio edits (e.g., net profit/total revenue  must be within a specified range)
- Comparisons to past data
- Balance edits (totals = sum of parts)
- Computer editing is optimal

## Types of Edits (2)

- Macroediting: Editing at aggregate 	level (needs data from many respondents.)
- Usually based on models of relationships between variables, or market share
- Used to identify impact of specific respondent data on estimates

## When Edits are Applied

- Data check-in or just prior to keying (scan editing) Correction of unkeyable responses
- Micro editing – used on data entry
- Warning flags (suspicious entries)
- Fatal flags (definite errors)
- Make sure data base distinguishes between no data (blanks) and reported zeros

## Types of Imputation

- Imputation -- Estimation of missing survey information.  May be used to replace data that fail edits.  In EIA used primarily to estimate for nonresponse.
- Model-based imputation
- Deck imputation
- cold deck – external donor
- Hot deck – internal donor
- Weight adjustment (for sample surveys)

## Types of nonresponse

- Unit nonresponse – failure for a respondent to send in a form
- If skewed population and periodic survey, use estimate based on past reported data from respondent as imputed value.
- If nonrespondent similar to respondents, use estimate based on other respondents (eg. Hot deck)
- Error is difference between imputed value and value respondent should have reported.
- Item nonresponse – respondent does not completely fill in form, leaves some items blank.

## To Minimize Nonresponse 

- Follow-up with nonrespondents that do not provide data
- Send reminder letter
- Followup call(s)
- Send letters to more senior people in company signed by more senior people in your agency

## Dealing with nonresponse

- Increasing contacts (Number of call backs, call scheduling)
- Increasing cooperation
- Advance letters and information, letters to higher authorities in company
- Adjust estimates (if necessary)
- Impute for missing data using past data or data from other respondents, as appropriate
- Can subsample nonrespondents to determine similarity to respondents

## Both kinds of nonresponse

- Higher if information requested is not available, question is difficult to understand, or data are difficult to find.
- Higher if data requested are considered sensitive.  Some price information is considered sensitive.
- Usually lower if surveys are mandatory

## Computing response rate R

- I = number completed interviews
- RF= number refusals
- NC=number noncontacts
- NI=number of other interviews not completed
- IE=number of ineligibles

## Example of Survey Response Rates

_**	**__**	**__**Unweighted**_
_**Survey**_**	**_**Year**_**	**_**Rates**_
**Agricultural Resources Management Study****	****1999****	****66.1**
**Agricultural Survey****	****1999****	****80.0**
**Annual Survey of Manufacturers****	****1998****	****72.6**
**Census of Agriculture****	****1997****	****86.2**
**Commercial Building Energy Consumption**
**Survey:  Building Characteristics****	****1995****	****87.5**
**Economic Census****	****1997****	****86.1**
**National Ambulatory Medical Care Survey****	****1998****	****68.0**

- Shimizu, I.M.  (2000)  “Response in Federal Establishment Surveys.”  Paper presented at the International Conference on Establishment Surveys, June 2000, Buffalo, NY.

## Summary of Response Rates and Nonresponse Bias

- Bias is a function of both:
- Response rate
- Difference between respondent data and imputed data.  (Can check by using simulation.)

## Inspection Methods for Data Processing Quality

- 1.	Inspect sample (or entire set) of items in the data base
- 2.	Compare to questionnaires, or contact respondents to verify
- 3.	*Optional:* correct nonconformities
- 4.	Pass or fail
- 5.	*Optional: *provide feedback to operators

## Continuous Quality Improvement Strategy

- Use of Pareto principle
- Focus on root causes
- Team approach

## The Generic Coding Process

**The Generic Coding Process**

## Survey Evaluation

| Evaluation Methods | Stage | Purpose |
| --- | --- | --- |
| Debriefings
Interviewers group
Respondent focus groups | Post-survey | Evaluate questionnaire and data collection procedures  |
| Post-Survey Analysis
Experiments
Comparisons | Post-survey | Compare alternative methods of data collection, estimate MSE components, validate survey estimates |
| Post-survey Data collection
Reinterview
Non-response followup
Record check studies | Post-Survey | Estimate one or more components of MSE |

## The Balance

- Quality dimensions conflict
- Accuracy vs timeliness
- Accuracy vs relevance
- Comparability vs accuracy
- Cost vs error
- Comprehensiveness vs timeliness