# Presentation: 523248

## Workshop 4bQuestionnaire Design

- Stephanie J. Battles
- Energy Information Administration
- Dhaka, Bangladesh
- September 21, 1999

## Questionnaire Design

- What Are the Data Needs?
- Are the Data Obtainable?
- How Do You Find Out ?
- How Do You Collect Your Data?
- How Do You Write the Questions?
- What are Sources of Nonsampling Error in the Questionnaire?
- Why Pretest?

## What Are the Data Needs?

**What Are the Data Needs?**

- Publish Federal Register Notice
- Public Notice in Newsletters
- Mail Request for Comments to Groups and Organizations
- Target Key Individuals/Groups for Formal Input
- Hold User Group Meetings
- Internet Survey

_**Users**_**: Data Needs and Suggest Changes; **
_**Developers**_**: Feedback on Proposed Changes,**
** ****Additions, and Deletions**)

## Data Needs Example One1993 RECS User-Needs Study

**Data Needs Example One*****1993 RECS User-Needs Study***

- Written Response
- Federal Register Notice (40 Letters Received) (April 14, 1992)
- Mass Mailing to Users (750): Letter, Federal Register Notice, Overview to the RECS (April 1992)
- Requested 16 Energy-Related Magazines or Newsletters Carry a Notice About the User-Needs Study (April 1992)

**56 Letters in Response**

## Data Needs Example One1993 RECS User-Needs Study

- Target Group Meetings (Summer 1992)
- 15 Meetings (Including ACEEE Summer Study)
- Forecasting
- Policy
- Home Builders
- Conservation Group
- Bureau of Labor Statistics
- Minority Impact Group
- Low-Income Group

## Working Group Meetings (3): Members Representing DOE Users Responsible for:
Directing Research Funds 
Administering National Programs
Proposing Government Policy on Energy-Related Issues
Forecasting Energy Consumption
Setting Standards for Appliances and Buildings

- Directing Research Funds
- Administering National Programs
- Proposing Government Policy on Energy-Related Issues
- Forecasting Energy Consumption
- Setting Standards for Appliances and Buildings

**Data Needs Example One*****1993 RECS User-Needs Study***

## Results
After the Study--New Questions Such As:
Consumer-Decision Making
More Questions on Windows
Addition of Lighting Supplement
Difficulties in Meeting User Needs
Time Constraint- Respondent Burden (1 Hour)	
If Add...Had to Delete
Hundreds of Suggested Changes
Cost

- After the Study--New Questions Such As:
- Consumer-Decision Making
- More Questions on Windows
- Addition of Lighting Supplement
- Difficulties in Meeting User Needs
- Time Constraint- Respondent Burden (1 Hour)
- If Add...Had to Delete
- Hundreds of Suggested Changes
- Cost

**Data Needs Example One*****1993 RECS User-Needs Study***

## Target Group Meetings (Summer 1998)
4 Meetings
Information Center (NEIC)
International Statistics Office
Forecasting Office
Domestic Energy Statistics

- Target Group Meetings (Summer 1998)
- 4 Meetings
- Information Center (NEIC)
- International Statistics Office
- Forecasting Office
- Domestic Energy Statistics

_**Purpose**_**: Needed to Discuss Data Collection Proposals for Changes Due to Budget Reductions and Changing Natural Gas and Electricity Markets.**

**Data Needs Example Two*****1998 MECS User-Needs Study***

## Written Response
Federal Register Notice
Mail Survey (June 1998)
Mass Mailing to Users (239)
21% Response
Internet Survey (May to July 1998)
207 Visits
15 Completed Surveys
Results--Examples
Removed Fuel Switching
Support for Less Geographic Detail
Numerous Requests for Other Data 

- Federal Register Notice
- Mail Survey (June 1998)
- Mass Mailing to Users (239)
- 21% Response
- Internet Survey (May to July 1998)
- 207 Visits
- 15 Completed Surveys
- Results--Examples
- Removed Fuel Switching
- Support for Less Geographic Detail
- Numerous Requests for Other Data

**Data Needs Example Two*****1998 MECS User-Needs Study***

## Are the Data Obtainable?

- Cost--Survey Administrator and Respondent
- How Difficult to Obtain
- Adds to the Collection/Data Processing Time?
- Adds Response Burden?
- Respondent Knowledgeable, e.g. insulation in walls
- Saturation of Data of Interest, e.g. solar roofs

## Federal Registry Notice
Site Visits
Pretests/Pilots
Focus Groups
Cognitive Studies

- Site Visits
- Pretests/Pilots
- Focus Groups
- Cognitive Studies

**How Do You Find Out?**

## MECS 1985: New Survey Development
Federal Register Notice, Feb. 1984 (40 From Data Providers) 
Site Visits to 20 Manufacturing Establishments, June to November 1984:
Prearranged Agenda Included:
Description of the Establishments’ Energy Using Processes
Recordkeeping Practices and Availability of Recorded Data on a List of Specific Data Items that EIA was Considering
Respondents’ Conception of Certain Complex Subject Areas e.g. Capacity Utilization
Establishment End Uses and Fuel Switching Capability

- Federal Register Notice, Feb. 1984 (40 From Data Providers)
- Site Visits to 20 Manufacturing Establishments, June to November 1984:
- Prearranged Agenda Included:
- Description of the Establishments’ Energy Using Processes
- Recordkeeping Practices and Availability of Recorded Data on a List of Specific Data Items that EIA was Considering
- Respondents’ Conception of Certain Complex Subject Areas e.g. Capacity Utilization
- Establishment End Uses and Fuel Switching Capability

**How Do You Find Out?*****Example: MECS***

## MECS 1985 (Con’t)
Personal Interviews/Correspondence (105)
DOE, Other Federal Agencies
States and Universities
Pilot Test  on a Nonrandom Sample (100) (Between Oct. 1985 and Jan. 1986)
Establishments Chosen Based on Complexity of Operation
Respondents were Interviewed to Evaluate the Questionnaire

- Personal Interviews/Correspondence (105)
- DOE, Other Federal Agencies
- States and Universities
- Pilot Test  on a Nonrandom Sample (100) (Between Oct. 1985 and Jan. 1986)
- Establishments Chosen Based on Complexity of Operation
- Respondents were Interviewed to Evaluate the Questionnaire

**How Do You to Find Out? *****Example: MECS***

## MECS 1991: National Energy Strategy (1990)--Increased Data Needs
Focus Groups Used to 
Identify Needs of New and Expanded Energy-Related Data (10)
Determine the Availability of these Data from Potential Manufacturing Respondents (7)
Does Requested Information Exist ?
Could Approximations Be Provided ?
Would Approximations be Fairly Reliable ?
Would Respondent Burden be Substantially Increased ?

- MECS 1991: National Energy Strategy (1990)--Increased Data Needs
- Focus Groups Used to
- Identify Needs of New and Expanded Energy-Related Data (10)
- Determine the Availability of these Data from Potential Manufacturing Respondents (7)
- Does Requested Information Exist ?
- Could Approximations Be Provided ?
- Would Approximations be Fairly Reliable ?
- Would Respondent Burden be Substantially Increased ?

_**Focus Groups**_**: Qualitative interactive research using 6-12 participants with knowledge about the topic.**

**How Do You to Find Out? *****Example: MECS***

## MECS 1991 (Con’t)
Focus Group Attendees
Fertilizers, Pulp and Paper, Petroleum Refining, Motor Vehicle Manufacturer, Steel, Chlor-alkalies, and Olefins
Results of the Industry Focus Groups
Consumption of Specific Fuels and Electricity was Generally Available
Some Manufacturers, Especially Those with Newer Plants, Collected Limited End-Use Data for Selected Process Stage, e.g. process heating
Consumption of Energy by Specific End Uses is Generally Not Tracked
Energy and Plant Managers Would Have an Intuitive Understanding of How Energy Was Used

- Focus Group Attendees
- Fertilizers, Pulp and Paper, Petroleum Refining, Motor Vehicle Manufacturer, Steel, Chlor-alkalies, and Olefins
- Results of the Industry Focus Groups
- Consumption of Specific Fuels and Electricity was Generally Available
- Some Manufacturers, Especially Those with Newer Plants, Collected Limited End-Use Data for Selected Process Stage, e.g. process heating
- Consumption of Energy by Specific End Uses is Generally Not Tracked
- Energy and Plant Managers Would Have an Intuitive Understanding of How Energy Was Used

**How Do You to Find Out? *****Example: MECS***

## MECS 1998: Questionnaire Redesign 
Cognitive Study: Why ?
Significant Questionnaire Content Changes
Concern Over the Quality of Some Data
Make More User-Friendly
Help Maintain Response Rates

- MECS 1998: Questionnaire Redesign
- Cognitive Study: Why ?
- Significant Questionnaire Content Changes
- Concern Over the Quality of Some Data
- Make More User-Friendly
- Help Maintain Response Rates

_**Cognitive Study**_**: The approach used a one on one structured **
**interview with respondents.  Information was obtained about **
**their “thinking processes” undertaken to answer survey **
**questions.  Results are anecdotal, not statistical but provide**
**some insights.**

**How Do You to Find Out? *****Example: MECS***

## How Do You Collect Your Data?

- Personal Interview
- Paper (PAPI), e.g. 1993 RECS and Earlier, 1992 CBECS and Earlier
- Computer Assisted (CAPI), e.g. 1995 CBECS, 1997 RECS
- Mail, e.g. MECS, Energy Supplier Surveys
- Telephone
- Paper
- Computer Assisted (CATI), e.g. 1983 and 1999 CBECS
- Touchtone Response, BLS
- Voice Recognition
- Electronic, e.g. Fax or Diskette
- Internet, e.g. Dept. of Education Customer Satisfaction Survey

## Personal InterviewPAPI Versus CAPI

- Reduction of All Types of Administrative Errors
- Easily Handle Questionnaire Complexities
- Run Edit Checks Immediately and Allow Respondent to Correct or Clarify
- Incorporate Skip Patterns
- Eliminate Manual Data Keying

## Personal InterviewPAPI Versus CAPI:Example RECS

- Why Use CAPI instead of PAPI
- RECS Household Questionnaire is a Long, Complex Interview with Many Opportunities for Error
- Question Routing
- Real-Time On-Line Editing
- Range Checks
- Consistency Checks
- External Look-Up Lists
- Customized Questions
- Reduce the Time Between Completion of Data Collection and Availability of Data Set

## How Do You Write the Questions? 

- Questions Should:
- Use Simple Language
- Example, *main versus principal *
- Use Common Concepts
- Example, *amount of time spent cooking a meal versus *
- p*ercent of time cooking a meal *
- Should Only Ask for Manageable Tasks
- Example, *the walls are insulated* *versus *
- a*mount of Insulation in the walls *
- Contain Clear Definitions for Shared Meanings
- Example, *your utility* *versus* *local distribution company*

## How Do You Write the Questions? 

- Specific Questions Are Better than General Questions
- Specific Questions Help in Recall and Provide a Narrower Range of Interpretation
- Closed Questions
- May Be Designed More Specifically
- Open Questions
- Use When Unfamiliar with All the Answers
- Use as a Follow-up to a Closed Question
- Question Order May Matter
- Example, sensitive questions such as personal income...order them later in the questionnaire

## How Do You Write the Questions?Example: MECS 1998 Redesign

- Series of EIA Team Meetings
- Significant Questionnaire Content Changes
- Make More User-Friendly
- Dropped Matrix Format
- Adopted Booklet-Style, Question Format
- 8 1/2  by 11 (inches)

## Expert Review Opinion
Portrait Booklet Style
Use Screener Questions
Make Sure There is Sufficient Redundancy in Skip Questions So They will be Easily Followed
Definitions With the Questions Instead in a Separate Booklet
Question Text in Bold
Instructions in Italics
Sequentially Number Questions
Number Important Instructions

- Portrait Booklet Style
- Use Screener Questions
- Make Sure There is Sufficient Redundancy in Skip Questions So They will be Easily Followed
- Definitions With the Questions Instead in a Separate Booklet
- Question Text in Bold
- Instructions in Italics
- Sequentially Number Questions
- Number Important Instructions

**How Do You Write the Questions?*****Example: 1998 MECS Redesign***

## Cognitive Study: “Thinking Approach”
One-on-One Structured Interviews with Respondents
Obtain Information about the “Thinking Processes” Undertaken to Answer Survey Questions
Check on Respondent’s Understanding of terms, Survey Flow, and Sequence
Semi-Structured Approach Allowed Respondents to Talk About Areas/Issues Important to Them
Results are Anecdotal, Not Statistical
Results Provide Insights

- One-on-One Structured Interviews with Respondents
- Obtain Information about the “Thinking Processes” Undertaken to Answer Survey Questions
- Check on Respondent’s Understanding of terms, Survey Flow, and Sequence
- Semi-Structured Approach Allowed Respondents to Talk About Areas/Issues Important to Them
- Results are Anecdotal, Not Statistical
- Results Provide Insights

**How Do You Write the Questions?*****Example: 1998 MECS Redesign***

## Cognitive Study--Selected Topics
Natural Gas Section
Technology
Skip Patterns and Directions
Appearance and Formatting
Collection Information to Complete Form
Comparison of 1994 and 1998 Forms

- Natural Gas Section
- Technology
- Skip Patterns and Directions
- Appearance and Formatting
- Collection Information to Complete Form
- Comparison of 1994 and 1998 Forms

**How Do You Write the Questions?*****Example: 1998 MECS Redesign***

## Cognitive Study: Methodology
Population: 1994 MECS Respondents
Selection Criteria:
Geographic Location
Used Natural Gas
7 Respondents--Based on type of Manufacturing

- Population: 1994 MECS Respondents
- Selection Criteria:
- Geographic Location
- Used Natural Gas
- 7 Respondents--Based on type of Manufacturing

**How Do You Write the Questions?*****Example: 1998 MECS Redesign***

## What are Sources of Nonsampling Error in the Questionnaire? 

- Sources of Error
- Ambiguous Definitions
- Poor Instructions
- Questionnaire Wording
- Questionnaire Format
- Questionnaire Length

- Control of Error
- Glossary of Terms
- Unambiguous Instructions
- Expert Review Panel
- Cognitive Studies
- Questionnaire Pretest

## Why Pretest?

- Pretest Questions
-  Acceptable Level of Variation
-  Meaning the Same For All Respondents
- Task Difficulty
- Do the Questions Ask for Familiar Information?
- Respondent Interest and Attention
- Does Varying the Format Help?
- Does the Use of Booklets Help?

## Pretest Questionnaire
Flow and Naturalness
Order of Questions
Skip Patterns
Timing
Fatigued--Satisficing Results
Respondent Interest
Pretest Data Collection Procedures

- Flow and Naturalness
- Order of Questions
- Skip Patterns
- Timing
- Fatigued--Satisficing Results
- Respondent Interest
- Pretest Data Collection Procedures

**Why Pretest?**

## Why Pretest?

- Pretest Issues
- Costly
- Time Consuming
- No Clear Consensus as to Good Principles of a Pretest
- Pretest Results May be Different if Respondents Know it is a Pretest

## Why PretestExample: 1999 CBECS

- Purpose of the Pretest:
- Can We Collect Data by Telephone
- Past CBECS Collected Data by Personal Interview
- Can Respondents Provide Their Energy Consumption and Expenditure Data
- Past CBECS Obtained Data From Energy Suppliers
- Does the Dodge Sample of New Construction Work
- New Construction Updates: Telephone and Mail
- Past CBECS Listed New Construction

## Main Goal of Pretest One: Understand the Effectiveness of the Proposed 1999 CBECS Data Collection Procedures
Final Results
95 Completed Interviews from a Sample of 167 (57%)
Major Recommendation
Speak With Knowledgeable Person as Part of Prescreening Effort Before Sending Out Advanced Materials	

- Final Results
- 95 Completed Interviews from a Sample of 167 (57%)
- Major Recommendation
- Speak With Knowledgeable Person as Part of Prescreening Effort Before Sending Out Advanced Materials

**Why Pretest*****Example: 1999 CBECS***

## Main Goal of Pretest Two: Test Changed Respondent Contact Procedures as a Result of Pretest One Findings
Final Results
164 Completed Interviews from a Sample of 196 (84%)

- Final Results
- 164 Completed Interviews from a Sample of 196 **(84%)**

**Why Pretest*****Example: 1999 CBECS***