# Presentation: 265407

## Workshop 4a--Survey Design

- Stephanie J. Battles
- Energy Information Administration
- Dhaka, Bangladesh
- September 20, 1999

## Framework for Workshops

**Framework for Workshops**

- US and Bangladesh--How Do We Compare?
- What are Statistics?
- Collecting the Data
- Methodology for the Collection of Usable Survey Data
- Workshop Examples: EIA Energy Consumption Surveys
- Why Does EIA Collect Energy Consumption Data?

## US and Bangladesh--How Do We Compare?

**United States**
- Population
- 264 Million (1998)
- Land Area
- 9.8 Million km2

**Bangladesh**
- Population:
- 125 Million (1998)
- Land Area
- 145 Thousand km2
- Size of Wisconsin
- 1.5 % of U.S.

## What are Statistics ?

- With Statistics We Can Describe the Data, for Example: “ In the U.S. Between 1991 and 1994, the average price of natural gas increased at an annual rate of 1.2 %.” ...Descriptive Statistics (Summaries, Mean, Tables, Graphs)
- With Statistics We Can Quantify Uncertainty--Make it Precise, for Example: “In the U.S. in 1995, with 95% confidence, the true amount of natural gas used in retail and service buildings was between 299 to 491 trillion Btu.” (CBECS 1995) ...Statistical Inference
- American Statistical Association: “Statistics is a set of methods used to collect, analyze, present, and interpret data.”

## Collecting the Data

- Why Not Have a Census
- Population is Large, Not Easily Accessible
- Costly
- Require Timely Data
- Rapid Rate of Change
- Why Use a Sample
- Limited Budget
- Speed...Less Data Needs to Be Collected
- Can Do A Better Job of Data Collection

## Methodology for the Collection of Usable Survey Data

- Survey Design
- Questionnaire Design
- Data Collection
- Data Processing and Validation

**Methodology for the Collection of Usable Survey Data**

## Workshop Examples:EIA Consumption Surveys 

- Manufacturing Energy Consumption Survey (MECS)
- Commercial Buildings Energy Consumption Survey (CBECS)
- Residential Energy Consumption Survey (RECS)

**Note: Residential Transportation Energy Consumption**
** ****Survey (RTECS): Discontinued**
**http://www.eia.doe.gov/emeu/rtecs/contents.html**

## MECS

** **
** **
** **

**http://www.eia.doe.gov/emeu/mecs/contents.html**

**For More Information:**

- Sponsored by the Energy Information Administration (EIA)
-  Sample Survey of approximately 18,000 establishments
-  Conducted Presently Every 4 Years
-  Mandatory Mail Survey
-  Previous Surveys, 1985, 1988, 1991, 1994,   and Now Collecting 1998
- Confidentiality..Data Collected by the U.S. Bureau of the Census

## MECS Data Examples

**MECS Data Examples**

- First Use of Energy for All Purposes
- Total Inputs of Energy for Heat, Power, and Electricity Generation
- Feedstock and Byproducts Used for Energy
- Energy Purchased/Sold/Generated Onsite
- Energy Expenditures
- End Uses, e.g. Boiler Fuel
- End Uses/Fuel Switching
- Characteristics: Number of Establishments, Floorspace, etc

## CBECS

- Sponsored by EIA
- Sample Survey of Approximately 5,700 Commercial Buildings
- Voluntary Personal Interview/Mandatory Energy Supplier Survey
- Conducted Presently Every 4 Years
- Previous Surveys, 1979, 1983, 1986, 1989, 1992, 1995 (Jan. 2000 Will Start Collecting 1999 Data)

**http://www.eia.doe.gov/emeu/cbecs/contents.html**

**For More Information:**

## CBECS Data Examples

- Physical Characteristics of the Building
- Age, Size, Construction
- Usage of the Building
- Ownership and Occupancy Characteristics
- Heating and Cooling Systems
- Lighting Systems
- Energy Conservation Practices
- Electricity Cogeneration
- Energy Sources Used and End Uses
- Energy Consumption and Expenditures

## RECS

- Sponsored by EIA
- Sample Survey of approximately 5,800 households
- Voluntary Personal Interview/Mandatory Energy Supplier Survey/Rental Agent Survey
- Conducted Presently Every 4 Years
- Previous Surveys, 1978 to 1982 (annually),1984, 1987, 1990, 1993, and 1997

**http://www.eia.doe.gov/emeu/recs/contents.html**

**For More Information:**

## RECS Data Examples

- Energy Consumption and Expenditures
- Housing Characteristics, e.g. housing type, floorspace, and age
- Energy Sources and End Uses
- Equipment and Appliances
- Demographic Characteristics
- Low-Income Assistance
- Energy Conservation

## Why Does EIA Collect Energy Consumption Data?

- Provides Information for Program Development, e.g. Energy Efficiency Programs
- Assists in the Development of Energy-Related Standards, e.g. Appliance Standards
- Data are Inputs into Forecasting Model (NEMS)
- Assists in the Policy Decision Process

**Data Users: Federal, State, and **
**Local Government, Private Industry,**
** ****Universities, etc.**

## Collection of Usable Survey DataSurvey Design

- Defining the Sampling Unit--Our Target Population
- Develop a Definition of Chosen Sampling Unit
- Types of Sampling Frames
- Sampling Frame Data
- Sampling Methodology
- Sampling Error
- Nonsampling Error
- Frame Error
- Survey Redesign

## Defining the Sampling Unit

**Defining the Sampling Unit**

- Manufacturing Sector Choices:
-  Company
-  Establishments
- Residential Sector Choices:
- Housing Unit
- Households
- Buildings
- Commercial Sector Choices:
- Establishments
- Utility Customer Records
- Buildings

_**Target Population**_**: The population to be studied.**
_**Sampling Unit**_**: Parts into which the Target Population**
** ****is divided.**

## MECS Sampling Unit

- Company: Industry’s Suggestion
- collect data for locations
- aggregate it through Trade Associations to the Company level
- data compilation complex
- linking problem for widespread companies
- Establishment:
- confidentiality concerns could be met
- establishment list available
- subsampling from ASM
- transferring economic data was available

## RECS Sampling Unit

- Housing Unit
- Estimate of Households Based On the Current Population Survey
- Use Census Results
- Energy Policy is Related to Structures
- Households
- Mobile: Too Difficult to Follow
- Costly
- Use Services, e.g. Heat
- Buildings
- Multiple Units in a Building

## CBECS Sampling Unit

- Establishments
- Standardized National List
- Pilot Test --Census Bureau (1978): Not Feasible
- Many Establishments in the Same Building, e.g. Shopping Center
- One Establishment with Many Buildings, e.g. College
- Utility Customer Records
- Excellent Measure of Size
- Hundreds of Records--Different Formats, Computerized Billing Not Uniform
- Mixed Classification
- Lack of Cooperation/Slow Response

## CBECS Sampling Unit (Con’t)

-  Buildings
- Could Obtain Information on Building Characteristics to Obtain a Direct Connection Between Energy Use and Building Characteristics
-  No Benchmark Available
-  No National List of Buildings

## Develop a Definition of Chosen Sampling Unit: MECS

**Develop a Definition of Chosen Sampling Unit: MECS**

- Manufacturing Establishment
- Working Definition Available from the Office of Management and Budget’s *Standard Industrial Classification Manual, 1987*

_**Manufacturing Establishments**_**: [Manufacturing establishments are]... engaged in the mechanical or chemical transformation of materials or substances into new products. These establishments are usually described as plants, factories, or mills and characteristically use power driven machines and materials handling equipment.  Establishments engaged in assembling component parts of manufactured products are also considered manufacturing if the new product is neither a structure nor other fixed improvement.  Also included is the blending of materials such as lubricating oil, plastics, resins, or liquors**.
- .

## Develop a Definition of Chosen Sampling Unit: RECS

_**Housing Unit**_**: A house, an apartment, a group of rooms, or single room if it is occupied, or intended for occupancy, as separate living quarters by a family, an individual, or a group of one to nine unrelated persons. Separate living quarters means the occupants (1) live and eat separately from other persons in the house or apartment and (2) have direct access from the outside of the building or through a common hall--that is, they can get to it without going through someone else’s living quarters.**

**Develop a Definition of Chosen Sampling Unit: RECS**

- Housing Unit
- U.S. Census Bureau Definition Adding
- Occupied
- Primary Residence (50% or Longer Annually)

## Develop a Definition of Chosen Sampling Unit: CBECS

_**Building**_**: ****A structure totally enclosed by walls extending from the foundation to the roof, containing over ****1,000 square feet**** of floorspace, and intended for human occupancy**

**Develop a Definition of Chosen Sampling Unit: CBECS**

- Commercial Building
- Working Definition Not Available from Other Sources--Needed a Definition of “Building” and “Commercial Building

## Develop a Definition of Chosen Sampling Unit: CBECS (Con’t)

_**Commercial Building**_**: A commercial building is a building with more than 50% of its floorspace used for commercial activities. Commercial buildings include, but not limited to, stores, offices, schools, religious worship buildings, gymnasiums, libraries, museums, hospitals, clinics, warehouses, and government buildings not on restricted access sites.**

**Develop a Definition of Chosen Sampling Unit: CBECS (Con’t)**

## Types of Sampling Frames

**Types of Sampling Frames**

_**List Frame**_** is a list of sampling units from which the sample is chosen.**

_**Area Frame**_** is a set of geographic areas from which **
**areas are selected from which the sample is chosen.**

## Sampling Frame Data

- Unique Identifiers, e.g. name, address
- Variables Used to Stratify the Population and Choose the Sample, e.g. size, location
- Contact Information, e.g. mailing addresses, telephone numbers
- Dates of Additions and Changes to the Frame and Linkages Between Frame Units

## Sampling Frame Example

- MECS Frame
- Prior to 1994, the Annual Survey of Manufactures’ (ASM) Mail File (Approximately 55,000 Establishments)
- 1994, the Updated 1992 Census of Manufactures’ (CM) Mail File (Approximately 380,000 Establishments in CM File-200,000 in CM Mail File Representing 98% of Value of Shipments)
- 1998, the 1997 CM Mail File

## Sampling Methodology

- A Survey May Have More Than One Frame
- A List Frame and Area Frame Can be Used in the Same Survey
- A Multistage Design Has a Frame for Each Stage
- A Multiphase Design’s Second Stage Frame is a Subset of Units From the Sample Chosen In The First Stage

## Sampling Methodology Example: MECS

- Pre-1994
- MultiStage Design
- First Stage: Selection of ASM Sample from the CM Frame
- Selection of MECS Sample From ASM Mail Sample
- 1994
- Single Stage Design
- Selection of MECS Sample from the Updated 1992 CM Frame

**Important Classification Variables:**
**Standard Industrial Classification Codes **
**and Census Divisions (1994)**

## Sampling Methodology Example:RECS

- Since 1980--Multistage Area Probability Cluster Sample Design
- Stage 1
-  50 States and D.C. Total Land Area is Divided into 1,786 Primary Sampling Units (PSU)
- PSU’s Were Grouped Into 116 strata with 1 PSU Selected from Each Strata

_**PSU**_**: PSU’s are either metropolitan areas containing a**
**central city of 50,000 or larger, or they are counties **
**containing small cities and rural areas.**

## Sampling Methodology Example:RECS (Con’t)

- Stage 2
- From Each of the Selected PSU’s, 8 or More Secondary Sampling Units (SSU) were Selected

_**SSU**_**: SSU’s consist of one or more Census blocks,**
** ****selected directly from the Census statistics.  Blocks**
** ****are combined as necessary, to create SSU’s that **
**contained at least 50 housing units.**

## Sampling Methodology Example:RECS (Con’t)

**Updated for New Construction**

**Sampling Methodology ****Example:RECS (Con’t)**

- Stage 3
- A Detailed Listing of Street Addresses Is Created for each SSU.  (SSU’s With Large Numbers of Housing Units  Are Divided Into Smaller Listing Segments And one is chosen for the detailed address listing.)
- Stage 4
- An Ultimate Cluster of approximately 5 housing units was Selected from the list of housing Units for Each of the Sampled Listing Segments.

## Sampling Methodology Example:CBECS

**Large Impact on Variances if Only**
**Area Sampling: Need Relatively Rare**
**Large Energy Users**** **

**Sampling Methodology ****Example:CBECS**

- Dual Frame Design
- Multistage **Area** Probability Cluster Sample Design
- Four Stage Cluster Design, PSU,SSU,Listing Segments, and the Ultimate Cluster of Buildings
- Supplemental **List** of:
- “Large” Buildings (250,000 Square Feet or Larger),
- Recently Constructed Large Buildings, and
- “Special” Buildings (Federal Government Buildings, Post Offices, Hospitals, Colleges, and Universities)

## Sampling Error

- Causes
- Chance May Result in Untypical Choices
- Sampling Bias, e.g. Sample from an Energy Trade Association Mailing List and Assuming that the Sample is Representative of Everyone Working in the Energy Field.
- It is a Sample and Not a Census

_**Sampling Error**_**: Sampling error is the differences between the sample and the population that are due solely to the particular elementary units that happen to be selected.**

_**Sampling Bias**_**: Sampling bias is a tendency to favor the selection of sampled units that have particular characteristics.**

## Sampling Error (Con’t)

- Sample Design Allows the Sampling Error to Be Estimated
- Sampling Error Is Measured by the Standard Error of the Estimate.
- Standard Errors are Calculated for the MECS, RECS, and CBECS

**Notes:**
Std errors for the CBECS large...no population controls for the weights...a building with a big weight..can play havoc to the variances

## Nonsampling Error

**Nonsampling Error**

- Causes: Different Sources
- Examples
- Defects in the Frame
- Defects in the Selection of the Sample
- Mistakes in the Collection of the Data
- Nonresponse
- Mistakes at the Processing Stage

_**Nonsampling Error**_**: The error in an estimate which**
** ****cannot be attributed to sample fluctuations.**

## Frame Error

- Use Stratification in Choosing the Sample, e.g. In Large PSU’s, the SSU’s were first stratified by predicted energy expenditures, and within these strata, by geographic location (RECS)
- Supplemental Lists, e.g. CBECS’s Area Sample is supplemented within each PSU with a Sample of Large and Specialized Buildings

_**Frame Error**_**: A frame error occurs when the frame**
** ****does not represent the target population. Frame Error**
** ****is considered a nonsampling error due to omission, **
**inclusion, or duplication of your target population.**

## Frame ErrorExamples

- RECS
- Data are Not Collected for Vacant Housing Units/Seasonal/Second Homes (11% in 1993)
- Effects of Omission Are:
- Underestimation
- Total Housing Units
- Number of Units in Subcategories
- Amount of Energy Consumed
- MECS
- Only 98% of Manufacturing as Measured by Payroll
- CBECS
- Buildings Over 1,000 Square Feet

## Survey Redesign

- Why Redesign
- Different Target Population, e.g. Building as the Target Population Instead of Separate RECS and CBECS
- New Construction, e.g. 1999 CBECS Using 1986 Updated Frame (Frame Based on 1980 Population Measures)---Redesign is Planned in the Near Future
- Different Mode of Data Collection, e.g. Personal Interview to a Telephone Interview