# Presentation: 394036

## SURVEY SOFTWARE DESIGN

- Jacob Bournazian
- May 2000
- Bangladesh Ministry of Energy and Mineral Resources

## Typical Survey System

- <date/time>

**Data **
**Entry**

**Name, **
**address**
**Code**

**Edit,**
** ****Impute,**
**Code**

**Respondent**
**Data**

**Energy**
**Data**

**Est., Var.,**
** ****Supp.,  **
**Code**

**Data**
** ****Files**

**Desk Top **
**Code**

**Publi-**
**cations**

**Typical Survey System**

## Criteria and Goals for a Survey Processing System

-  What are the goals for the system?
- Design Quality
- Data Integrity, Accessibility, Timeliness
- What criteria will be applied?
- E.g. edit criteria
- What are the costs to develop and maintain the system?

**Notes:**
How accessible is the data base going to be.  Can survey analysts do queries on the data base.  Can only canned reports be done.  Can both survey designers and analysts access the data base.
Can you change the code, how accessible is that

How and where is the data base stored.  Is there just one data base, or multiple versions stored in separate places.  How are changes to the data tracked.  How do we know we are working with the final version or that all changes made it in.

Costs to develop
Do we need to write custom software is there any off the shelf software.  If you can use off the shelf software, are there any modules that may need cusomizing, e.g., estimation and variance formulas, etc.

Cost to maintain - Do need to be updating certain data elements all the time, what about adding/deleting a product, sales category, etc.

## Developing Automated Collection Systems

- CASIC (Computer Assisted Survey Information Collection) is an all inclusive term for several automated data collection systems
- Technology and supporting software has been developing and rapidly expanding for the past 20 years
- Grew out of need for handling large surveys, reducing costs of data collection, speeding up data collection, editing, processing, and attempts to improve response rates in mail surveys.

**Notes:**
In the past 20 years technology has been developing for automating the collection of data.

Recent software enhancements have led to several software products that can handle all survey processing from collection to publication.

## Types of (CASIC) Computer-Assisted Survey Information Collection Systems

- CATI (Computer-Assisted Telephone Interviewing)
- Interviewers generally cluster at one or more central locations and contact telephone respondents, Interviewer reads questions displayed by a computer, and enters the answers into the computer.
- E.g., retail price surveys for gasoline and diesel fuels use to measure to price per gallon paid by the average U.S. consumer

## Types of CASIC systems continued

- CAPI (Computer-Assisted Personal Interviewing)
- Interviewers go to the respondents home or offices with a laptop PC and read the questions from, and record the answers into the computer.
- E.g., Consumer Price Index survey used to measure the rate of inflation in the U.S.

## CASI (Computer-Assisted Self Interviewing)
Techniques include:
PDE (Prepared Data Entry) Respondents use a PC or terminal themselves to fill out interactively the survey questionaire
TDE (Touchtone Data Entry) Respondents answer computer generated questions by pressing buttons on a telephone
VRE (Voice Recognition Entry) Respondents answer questions by speaking directly into a telephone

- Techniques include:
- PDE (Prepared Data Entry) Respondents use a PC or terminal themselves to fill out interactively the survey questionaire
- TDE (Touchtone Data Entry) Respondents answer computer generated questions by pressing buttons on a telephone
- VRE (Voice Recognition Entry) Respondents answer questions by speaking directly into a telephone

- Types of CASIC systems continued

**Notes:**
PDE is appropriate for long forms or surveys where respondents have to search for info in order to fill out the form

TDE is appropriate for short forms with a few data elements that contain numerical data

VRE - several health surveys use this.  Most popular use of this technology is respondents calling to request copies of forms or publications

## EIA’s CASIC-related Efforts

- End Use [Consumption] Surveys
- PC Electronic Data Reporting Option
- Electronic Filing Pilots
- Weekly CATI Surveys
- CATI Frame Update Test

## End Use Surveys

- Surveys at end users--households and commercial buildings
- Between 5,000 and 6,000 respondents
- Detailed Questionnaires
- 30-45 minute interviews
- Changed mode from CAPI to CATI for Commercial Buildings Energy Consumption Survey in 1999

**Notes:**
Residential Energy Consumption Survey
Commercial Buildings Energy Consumption Survey

Use BLAISE for Questionnaire (in house programming)
30 to 40-minute Telephone Interview 
Data collection is going on now 		(Jan-Aug ‘00
1992 survey done by mail Minimum # of questions 66, Max 330  (13 weeks 91%)
1995 survey done by CAPI Min # of questions 52, Max 199, easier way to do skip patterns (16 weeks 87%)
2000 survey done by CATI min # question 21, max 171, if don’t have cooking, or certain fuel, skips to next section  (28 weeks)
Data collection extended - pre screening, necessary to find the right building and right contact person, and then send them worksheets with the survey so they will have info available when contacted by telephone.

## PC Electronic Data Reporting Option (PEDRO)

- Windows-based Application
- 11 monthly surveys
- 5 weekly surveys
- Runs on Respondent’s Machine
- Includes Edit Capability
- Data transmit Directly to EIA via Modem or over the Internet
- Pretty Good Privacy (PGP) software for encryption

**Notes:**
One of EIA’s initial efforts in  the CASIC area was the PC Electronic Data Reporting Option or PEDRO.  
This system was first introduced over 10 years ago.  It was originally DOS-based, now  a Windows application is being used.  PEDRO is used throughout EIA and  now has about 800 respondents.
No passwords.  The respondent ID serve as the unique identifier of the respondent data.
Allows users to import data easily from a company data base or spreadsheet.   No changes are needed to the software if downloading from a company’s system into PEDRO.
On-line edits.  Mandatory data entry error checks require the user to correct errors before proceeding to the next cell. Certain fields can’t be blank (name, address) Data validation can be performed at any time during or after data entry or file importation.  Run edits on volume to check math.
It was set up to allow respondent companies to easily  report their data.  The application runs on the respondent’s Pentium machine.  Need 20 Mb for PEDRO and 10 Mb for each survey.  The Respondent  then transmits the encrypted data to EIA , via Internet or modem.
Transmitted data is encrypted using “Pretty Good Privacy”, a software package which has become the worldwide de facto standard for email encryption.  Available as freeware on the Internet, runs on either MacIntosh or IBM.  Marketed through Phil’s Pretty Good Software.

## Electric Power Survey	Electronic Filing Pilots

- Using 3 Monthly Electric Power Surveys
- Companies Send  Encrypted Data via Internet Directly to Servers Outside of EIA’s Firewall
- Each company receives a password
- Read Information From Previous Submissions, Bring in Blank Forms, and File Current Report
- Also Testing Filing Data via E-mail

**Notes:**
RW-859--Electric Utilities, 350 respondents, collect data on spent nuclear fuel, used to be annual, now every 3 years.  Collected 1996, 97, 98 data in 1999.  Originally used FOXpro but now the reporting system has been rewritten in Access. 

EIA-900--monthly Nonutility generation, used to be TDE, 500 respondents, how much power is generated, what fuel consumption is and fuel stocks
EIA-759- monthly utility power generation survey, collects same data as EIA-900
EIA-826--Sales, Revenues, sample survey160 respondents, electric utilities, how much electricity they sold and revenue, average revenue per kilowatt for each end use sector, outputs to ASCII file.  

For the financial survey, EIA-412, allowing respondents to file via e-mail, rather than sending in diskettes. Lengthy 16 page survey Financial data on electric utilities. [Also trying  this with a natural gas  survey where R. typically respond via diskettes. 

contact:  Roger Sacquety--202/426-1160

## Weekly CATI Surveys

- Retail Gasoline Prices	(approx. 1000 outlet prices)
- Retail Diesel Fuel Prices (350 outlet prices)
- Data Collected on Monday morning
- Publish retail price on Monday afternoon
- regional and US pricing on a weekly basis expanding coverage to 5 cities, 5 states,

**Notes:**
EIA conducts weekly retail energy prices surveys using Computer-assisted Telephone Interviewing.

The first survey collects retail gasoline prices from approx 1000 companies.  
The second  collects retail diesel fuel prices from 350 companies.  
Data for both surveys are collected in the morning between 8:00 am- 11:00 am,   The interviewer enters the data while talking to the respondent.  If the reported price is more than 5 cents different than last week, the system prompts the interviewer to ask for the reason for the price change.  validation, nonresponse follow up between 11:00 am - 1:30 am, close out and production 2:00 pm - 3:00 pm.  Publish data by 5:00 pm on the Internet and telephone hotline on Mondays of each week.

The computer software used is Sawtooth Software’s Ci2 CATI, Version 3.

[additional info:  Survey numbers are EIA-878 for Gasoline Prices and EIA-888 for Diesel Prices.  Data are released at PADD  and SubPADD level and for CA.  Data are collected by MACRO in VT.]

## Petroleum Product Sales Survey 		Frame Construction 

- Quadrennial Survey of Approx. 22,000 Companies
- Tested CATI versus Mail Updates, using Split Sample
- Conducted survey in 1999

**Notes:**
Last year, EIA conducted a study of  the use of CATI versus mail survey in order to update a large establishment frame.

## Petroleum Product Sales Frame  	Test Results

**CATI**
- Had Higher Response Rates
- Reports Were Filed Sooner
- Resulted in Cleaner Data
- Cost Less

**Notes:**
Test results show that Computer Assisted Telephone Interviewing led to slightly higher response rates, early filing of data, and less money and time spent on editing the reported data.

## Background of the EIA-863

- ~22,000 companies
-  mail survey, every 3 years
-   starting with 1998 reference, major budget reduction--- every 4 years--- other steps necessary to reduce cost
- reduce cost by 1) less manual review in list construction 2) less edit failures  3) increase initial response rates

**Notes:**
--22,000 mailed to, 821 respondents exempt from reporting
Analysis of previous reference period showed:-->most respondents medium to small size (large companies treated differently) -->91% reported in only 1 state-->96% reported 5 or fewer cells-->phone numbers available for 93% of the cos.

Small companies may be interested in reported through the computer assisted interview.

Large companies are selling many products and often in many states,  they may want to report electronically or by fax.

## Edit Profile

- 40-50% of companies fail:1) control data quality (CDQ) edit or2) volume data quality (VDQ) edit
- High rate of edit failures, and high initial nonresponse result in extensive phone follow-up
- Potential area of cost savings

**Notes:**
PREVIOUS RESEARCH SHOWED THAT APPROX. 50% OF THE REPORTING COMPANIES FAIL EITHER A CONTROL DATA QUALITY EDIT OR VOLUME DATA EDIT.

Examples of Control data quality edits
company says they are in business but do not report data
company says they sold their business or no longer in business but report data
company says they sold but do not report who they sold to
Most survey do not do these

Examples of Volume data quality edits
Cells in a row must add to the total

## Slide 17

**Notes:**
Respondents allowed to report either way even they were selected to report by CATI
CATI: 91.2% within 15 weeks
Mail: 87.7% within 23 weeks

## Response Rates

- CATI               MAIL            P-value
- TOTAL	91.20%	       87.72%	0.056
- One State	89.33%	       86.33%	0.262
- Two States	93.33%	       88.67%	0.159
- Three States	93.22%	       90.00%	0.372
- Zero Items	73.75%	       66.25%   	0.304
- One Items	96.47%	       87.21%  	0.027
- Two Items	91.82%	       91.82%  	1.000
- Three Items	95.83%	       93.75%	0.519
- Four Items	93.81%	       92.86%  	0.513
- Five or more	93.00%	       90.00%   	0.449

**Notes:**
The difference significant for total number or respondent report by CATI and Mail at .056
significant difference for one cell reported, at .027
about 60% of respondents selected to report using CATI actually used it.  Remainder preferred to report by mail.

## Slide 19

## Average Costs as Response Originally Designated 

-                          MAIL        CATI    DIFFERENCE    Pilot Survey     $6.20          $5.67          $.53*    Full Survey       $5.56         $5.20          $.36                                                * p value of .0002Break even point to recover programming costs                             	---> 14,000 respondents

## Summary

- CATI designated had higher response rates, reported sooner, and had cleaner data
- 31% of CATI designated are likely to choose to report by mail
- High proportion on full survey of one state and/or two item responses (lowest cost savings) & high percentage reporting by mail--> savings of only $.36/response--->break even of 14,000

## The Blaise Family of Softwarehttp://www.westat.com/blaise

**Notes:**
Developed by Statistics Netherlands.  Marketed by Westat corporation in the U.S.
The program was named after Blaise Pascal, the French philosopher and scientist in the 17th century who invented the first mechanical adding machine.  The Blaise system is programmed in Pascal; hence the name Blaise.
Widely used in Western Europe.  Increasing use in U.S.  Also in use in Eastern Europe, former Soviet Union countries, China, New Zealand and Australia.
Software package has 13 modules to handle all aspects of data collection and processing, survey administration and management. Especially useful for complicated and long surveys, with many routing conditions, calculations, and edits.  Also good for surveys for hierarchical data structures.  The package covers 
Supports CAPI, CATI, and CASI
Handles all survey processing activities; interactive editing, high-speed data entry, performs data manipulation and tabulation functions and survey management measures.
Offers 3 methods for adjustment weighting
International Blaise User’s Group based in London and they host user conferences every 1-2 years
Energy/environment/agriculture surveys
Business and economic surveys
Household surveys
Labor force surveys
Surveys with multiple collection modes

## Computer-Assisted Survey Execution System (CASES)http://socrates.berkeley.edu:7500/casesforcal2.html

**Notes:**
Originally developed for US Dept of Agriculture, Bureau of Labor Statistics, and US Census
Supports up to 9 languages, handles all survey processing activities 
Extensive case management capability
can handle complex forms with multiple answers
data outputted in ASCII format for use in any statistical analysis program
Alphabetic responses can be defined and stored as numeric data. For example, interviewers might enter "D" for a "Don't know" response, but the value stored in the data could be "-1."
will accept or reject data based pre-determined edit criteria
runs on Windows 3.1, 95, 98, NT

Used frequently on simple small surveys

## SAWTOOTH’S Ci2CATI for Windowshttp://www.sawtooth.com

**Notes:**
Sophisticated features for designing questionnaires
Provides full range of graphical and sound capabilities with Window features

## SNAP survey softwarehttp://www.mercator.co.uk/products.htm

**Notes:**
SNAP software handles entire survey process, a complete survey data processing service covering all aspects of survey design, data preparation and data collection through analysis and results presentation, using both paper and electronic questionnaires
Designed to run on all types of surveys: customer satisfaction, employee attitudes, consumer opinion, quality control issues.
In use in over 50 countries

## The Survey Systemhttp://www.surveysystem.com

**Notes:**
Another survey management package.  Handles complete survey process from multiple modes of collection, editing, aggregate statistics, and generate reports
In use in over 40 countries, easy to use and learn.  The Survey System is developed from Creative Systems in California.
Uses standard Windows pull-down menus
Allows up to 500 answer choices per each multiple choice question.  Good for survey questions with 11-500 answer choices.  E.g. automobiles
Handles up to 9 digits long

## Surveycrafthttp://www.surveycraft.com/Products.html

**Notes:**
Widely used in marketing and social research
Handles the entire survey process, all double byte languages are supported in English, Chinese, Japanese or Korean
Uses a central data dictionary to hold all the information about your data.  Eliminates the need to duplicate the definition of data items and speeds up the analysis process
standalone system runs on IBM compatible 486 or higher processor, 4MB memory

## Ronin’s Results for Researchhttp://www.ronin.com/rforrprod.htm

**Notes:**
Ronin translates as “wave man.” They were loyal sumarai to the Jananese emperor, that were released from their pledge of allegiance, became free thinking spirits, using only their intelligence, discipline and courage to try bold new methods.  Some became scholoars, some monks, and other mercenaries.   The founder of industrial giant Mitsubishi Trading Company, Yataro Iwaski, was a ronin.  Some founded software companies
Fully integrated PC, LAN and Internet based CATI software product
Handles full range of survey processing
multi-language support
full service company, operate as an integrated group, develop and build systems along with selling software.

## TOUCHTONE DATA ENTRY (TDE)

- Current Employment Statistics Program
- Bureau of Labor Statistics, US Dept of Labor
- Monthly survey of employment, payroll, and hours
- Sample of 390,000 business establishments
- Publish data two and a half weeks from collection
- Provides key economic indicators
- Employment by industry, state, and area
- Average hourly earnings
- Average weekly hours

## Automated Collection Methods

- CES offers a variety of automated reporting methods:
-         _New Automated collection mode_                  _Year_
- Computer Assisted Telephone Interview . . . . . 1984
- Touchtone Data Entry (TDE). . . . . . . . . . . .. . .1986
- Voice Recognition . . . . . . . . . . . . . . . . . . . . ..1988
- Electronic Data Interchange (EDI) . . . . . . . . .  1994
- FAX  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  . .. 1995
- World Wide Web . . . . . . . . . . . . . . . . . . .  . ..  1996

**Notes:**
This is a time sensitive survey.  Each month, ther are only 10-15 days to collect and process the data before publishing preliminary estimates.  Historically, always done by mail, struggled with response rates of 55% for preliminary estimates, resulted in large revisions.  Data watched closely by bueinesses, financial markets and policy makers.  Data collected by mail was slow and labor intensive.  In order to improve estimates, raise response rates and reduce programming costs.  First used CATI in 1984 and response rates rose to over 80%
Touchtone Data Entry (TDE) was first used as a data collection method for the Current Employment Statistics (CES) program in 1986

CES was the first Federal statistical survey to use TDE for data collection

CES is now the largest Federal statistical survey to use TDE

## Distribution of CES Sample by Reporting Method

- <date/time>

**Notes:**
There are multiple modes for collecting data
For respondents without touchtone service, Voice recognition (VR) system is developed which recognizes spoken responses to the automated interview.  For respondents who report for dozens of establishments each month, Electronic Data Interchange (EDI) they transmit their data once.

## Data Collection: CATI to TDE

- <date/time>

- Most respondents report via Computer Assisted Telephone Interviewing (CATI) for five months
- CATI Interviewers use this opportunity to:
- Solidify CES reporting relationship
- Ensure data quality
- Serve as a resource for information
- Prepare respondent for TDE self-reporting
- After five months, respondents are requested to begin reporting by TDE, and are sent an _[introductory package](file:///C:/WINDOWS/Desktop/TDEletter.doc)__ in the mail _
- If the respondent does not wish to use TDE, they are offered alternate reporting methods

## TDE Methodology

- Respondents receive a monthly advance reminder FAX or postcard, according to prompt code
- Respondents call a toll-free number and enter data using the keypad of their telephone
- As the monthly deadline approaches, delinquent respondents receive a nonresponse prompt (NRP) by FAX or telephone, according to prompt code
- A second, “Last Chance” NRP message may be sent on the morning of the deadline
- Questions/concerns are addressed by a Help Desk staff
- Data are extracted daily and uploaded to a main server
- Registry information updates are sent to the States weekly

## One-Point TDE Features

- TDE collection for 40 states in one office
- 120 phone lines for data collection
- 48 lines for outbound FAXes
- Postcard generation for prompts
- NRP telephone prompts made at calling centers
- Back-up site with 48 phone lines and 24 FAX lines
- Help Desk

## Example of Advance Notice FAX

## One Point TDE Help Desk Staff

- On duty 7:00AM to 8:00PM EST to answer questions
- Adept at answering questions and dealing with reluctance
- Use a specially designed Help Desk system that aids in customer service
- Help Desk staff have the ability to:
- Collect data
- Update registry information
- Print and mail “respondent packages”
- FAX “respondent packages”

## Special Uses of TDE System

- In addition to data collection, the TDE system may be modified to contain:
- Contact update questions
- FAX availability questions
- FAX prompts may be customized by industry,  month, and establishment
- Tips on touchtone reporting
- Rounding instructions
- Past due reports needed
- Special messages

## Data Collection Cost Components

## Summary

- A viable collection mode for short, numeric surveys
- Convenience of self-response
- Maximizes response within cost limitations
- Requires ongoing nonresponse follow-up to maintain high response rates

## Audio Computer Assisted Survey Interviewing

- Automating data collection on Self administered surveys

## Benefits of self administered surveys 

- Enhanced privacy for the respondent
- Reduction in interviewer effects
- Greater control given to the respondent
**In Health surveys, there is a higher reporting of sensitive behaviors including:**
- smoking, drinking, drug use
- sexual activities
- abortion

## Benefits of Computerized self administered survey interviewing (CASI)

- Routing controlled by the computer
- More complex questionnaires possible
- Questions cannot be skipped inadvertently
- Out-of-range responses are eliminated
- Customized wordings possible
- Questions presented in the same order to all respondents
- Visual aids incorporated directly into the instrument

## Audio Computer-Assisted Self-Interviewing (ACASI)

- Above and beyond CASI, ACASI allows for:
- no requirement of literacy
- fully standardized question presentation
- fully private administration

## Issues for Audio Computer Assisted Surveys

- What to record
- Length of the ACASI survey
- Implications for question wording
- Choice of a voice
- Preparing the audio files
- Testing the audio
- Translations
- Loading the laptops

## What to Record...

- Some instruments have audio for the questions but not for the response categories
- Remember why you’re including the audio component
- If response categories are read, include instruction for how to enter an answer:
- Yes, press 1
- No, press 2

**Notes:**
Recommend including response categories

## How Long is Too Long?

- Consider the length for a variety of reading skills
- With respondents “in control” of the interview, there is less an interviewer can do to either slow a respondent down or speed him/her up
- Some evidence that respondents listen to less of the audio the further they get into the questionnaire
- To date, no evidence that there is an upper limit on the length of an ACASI instrument
- Interviewer reactions somewhat mixed

## Question Construction

- Consider impact of “fills” on the audio component of ACASI
- The computer recorded that you were [AGE 1st USE] the first time you used marijuana.  Earlier, you told the interviewer that your date of birth was [MONTH] [DAY] [YEAR].  That would make you [AGE] which is [# YEARS] younger than the first time you used marijuana.  That is not possible.  Which answer is correct?
- I was [AGE 1st USE] the first time I used marijuana
- I am [AGE] years old
- Neither answer is correct

## Question Construction cont’d

- Revised:
- The answer to the last question and an earlier question disagree.  Which answer is correct?
- I was [AGE 1st USE] the first time I used marijuana
- I am [AGE] years old
- Neither answer is correct
- Open-ended responses cannot be used as “fills”
- Consider the impact of tailoring questions on the audio recording.
- One question per screen

## Choosing a Voice for ACASI 

- Is the voice important, and if so what type of voice is best?
- Turner, et al. found no difference in data when male or female voice was used
- Pleasant is good, but promoting honest and accurate reporting is better!
- Laboratory testing at RTI indicates respondents do attribute varying degrees of “integrity” to a voice and have preferences for particular voices

## Practical Considerations in Choosing a Voice 

- Consider the topic of the study
- For longitudinal or ongoing surveys, determine long term availability
- Don’t select someone who has other competing priorities on the project
- Familiarity with interviewing is an asset
- “Professional” voices are an asset (singers, actors, public speakers, disc jockeys, etc.)

## Preparing the Audio Files 

- Allow sufficient time, but don’t start until you’ve finalized the question wording!
- Determine the optimal recording level
- Trade-off between quality and size of the files
- Assign unique file name to each audio file
- Prepare “scripts” for recording
- Options for recording audio
- Digitizing the audio files
- Goal:  Remove as much “dead air” from each file as possible -- particularly for “fills”

## Testing the Audio 

- On-the-spot testing
- Goal:  Identify mistakes in wording or pronunciation at the time of recording
- Database testing
- Goal:  Identify mistakes in wording or pronunciation as well as audio files that are missing
- Integrated file testing
- Goal:  Verify audio and screen text match; identify unusual intonation and long pauses

## Translations 

- Re-visit space constraints:
- each translation will at least double the amount of space needed for the audio
- subject/object agreement and degree of formality
- Select voice
- Don’t start until the translation is finalized!
- (Don’t translate until the English is finalized)

## Loading the Laptops 

- Allow sufficient time to load
- Experience with conducting health surveys:
- 1 hour interview / ACASI portion runs 40 - 45 minutes
- English and Spanish, 2 CDs needed
- Use of Head phones
- Hygiene issues
- Laptop speakers

**Notes:**
Typical load time: 20 minutes

## In Summary...

- ACASI does improve reporting of sensitive data, but, “the devil is in the details”
- Allow sufficient time for:
-  testing
- translations
- loading

## Slide 56

## COOL EDIT PRO www://syntrillium.com

**Notes:**
2 audio editing applications for Windows 95/98/NT
Cool Edit Pro offers 64-track mixing, 50 sound effects, 
Cook Edit 2000 - provides professional audio recording.  Single input/output pair for recording and playback.  20 sound effects
only 2,500 taccas

## Automating a processing system

- <date/time>

**Name, **
**address**
**Code**

**Edit,**
** ****Impute,**
**Code**

**Respondent**
**Data**

**Energy**
**Data**

**Est., Var.,**
** ****Supp.,  **
**Code**

**Data**
** ****Files**

**Desk Top **
**Code**

**Publi-**
**cations**

- Data
- Entry

- Data
- Entry

- Data
- Entry

- Data
- Entry

- Data
- Entry

- Data
- Entry

- Data
- Entry

## Building a generalized processing system to process all surveys

- <date/time>

## Developing an Automated Collection and Processing System

-  What are the goals for the system?
- Design Quality
- Data Integrity, Accessibility, Timeliness
- What criteria will be applied?
- E.g. edit criteria
- What are the costs to develop and maintain the system?

**Notes:**
How accessible is the data base going to be.  Can survey analysts do queries on the data base.  Can only canned reports be done.  Can both survey designers and analysts access the data base.
Can you change the code, how accessible is that

How and where is the data base stored.  Is there just one data base, or multiple versions stored in separate places.  How are changes to the data tracked.  How do we know we are working with the final version or that all changes made it in.

Costs to develop
Do we need to write custom software is there any off the shelf software.  If you can use off the shelf software, are there any modules that may need cusomizing, e.g., estimation and variance formulas, etc.

Cost to maintain - Do need to be updating certain data elements all the time, what about adding/deleting a product, sales category, etc.

## Standard Economic Processing System (StEPS)

- U.S. Bureau of Census
- Developed to process over 100 economic surveys in the areas retail, wholesale, service industries, manufacturing, and construction.
- Written entirely in SAS and operates in a UNIX environment.

## Objectives for StEPS

- Reduce resources required for system maintenance
- Standardize survey procedures used in data analysis
- Provide greater staffing flexibility for analysts and programmers to process different surveys by providing a processing system common to all surveys.
- Make all survey data available to all users: both survey analysts and survey managers

## Objectives for StEPS - continued

- Provide a common structure to make it easier to implement improvements for all surveys in the system.
- Improve timeliness for new surveys by eliminating analyst retraining and the development of custom survey processing software.
- Minimize the need for system maintenance

## StEPS Generalized Design features

- Design a set of standard data structures that remain the same, regardless of the survey and its data
- Use parameters (stored in general data structures) to drive the survey-specific processing requirements.
- Standardize field names and possible values for similar concepts (control data)

## StEPS Design features (cont)

- StEPS _Design features (cont)_

- Interactive SAS/AF screens permit users to:
- Specify parameters
- Select processing options
- Review and change data
- Monitor batch processing
- Configuration to individual surveys via specification files and processing scripts
- Does not include Frames development, sample selection, actual data collection and dissemination

## Interactive

- StEPS

- Batch
- StEPS

-         Data
-    dictionaries

- Script Files
- (SAS
- Macros)

- Microdata

- Macrodata

- Job
- submission

- Listing
- requests

## StEPS Modules

- _USER SETUP_:
- Allows the user to select a survey and stat period to process, set up default printers, or change the font size of the screen display

## StEPS Modules (continued)

- _SURVEY SPECIFICATIONS_
- Allows users to set parameters for various processing activities (i.e. edits, derived items, imputation, estimation, and outliers); define data dictionaries; define the survey-specific line displayed in the Data Review and Correction screens

## StEPS Modules (continued)

- _COLLECTION ACTIVITIES_
- Allows users to perform activities associated with data collection, including creation of the label files, logging in respondents, submitting the reported data through batch jobs

## StEPS Modules (continued)

- _REVIEW AND CORRECTION_
- Allows users to view and edit survey data
- Review all item data for a specified ID
- Review ID data for a specified item
- Review historical data for selected items
- View item totals for specific stat periods
- View different versions of the data, including reported, edited, adjusted, weighted-adjusted
- View current-to-prior ratios (between 2 stat periods) for a specified item

## StEPS Modules (continued)

- _TOOLS_
- Provides users with the capability to do:
- Analyze data files by accessing SAS tools (SAS/ASSIST, Insight, EIS)
- Download data to the PC
- Query survey data sets
- Request various lisitings to review survey data

## StEPS Modules (continued)

- _RUN_
- Allows users to run processes as defined in survey specifications module.  Such processes include edits, estimation, imputation, and derived items.
- _VIEW RESULTS_
- Allows users to view the results from the run processes.

## StEPS Modules (continued)

- _MIS_
- Provides management information reports, including response rates, imputation rates, and edit summaries.

## Standard Data Structures for every survey

- Use of dictionary data sets
- _Control-data dictionary_ - information about variables (numeric or character) describing processing options for individual reporting units
- _Item-data dictionary_ - information about numeric variables containing data from questionnaire or from other sources
- e.g. annual textile sales, amount of fuel used,

## Use standard Libnames

- Libnames point to different physical locations or directories within a data set called CENTRAL.SURVEYS. Once a user selects a survey, a libname called SURVLIB is set up.
- CENTRAL.SURVEY data set
- SURVEY		Char	Survey identifier
- SURVNME		Char	Survey name
- SURVDIR		Char	Directory of top-level 					survey info: SURVLIB

## Use of Standard Libnames cont.

- Once a user selects a survey and SURVLIB is set up, a data set SURVLIB.VSTATPS is opened.
- User selects the statistical period of data to access from a list of stat periods available for that particular survey.
- StEPS is able to treat any stat period as the “base” (or current) stat period and any stat period other than the “base” as it relates to the base.

## SURVLIB.VSTATPS data set

- SURVEY		Char	Survey Identifier
- SURVNME	Char	Survey name
- STATP		Char	Statistical Period
- DATASDIR	Char	Stat period specific data
- PARMSDIR	Char	Specific parmeter
- SPRGDIR		Char	Directory for survey-					specific programs

## Use of Standard Libnames cont.

- Non-stat period related libnames: DATALIB and PARMLIB are then set up.  The base stat period is assigned the following libnames: DATA00 and PARM00.
- This design sets up standard libnames that simply point to different physical locations based on what is stored in these data sets, regardless of what survey or stat period is used.

## Microdata Storage (Survey Parameters or specifications

- Control-data files -  (name and address info)
- A record for each set of standard variables + control type variables specific to a survey
- Master control file
- Stat-period control file
- Item-data files (numeric info for each ID)
- A record for each item in the survey, along with various processing fields (weighted, corrected)
- Skinny file - separate record for each ID/item
- Fat file - all data relating to an ID

## Elements of StEPS

- SAS data sets:
- Data dictionaries
- Micro/macro data
- Processing parameters
- SAS/AF screens for specifying parameters, submitting batch jobs, and requesting results listings
- SAS macros and estimation scripts for batch calculations

## EFFORTS AT EIA FOR BUILDING A COMMON DATA COLLECTION AND PROCESSING SYSTEM 

- COMMON COLLECTION AND PROCESSING SYSTEM (CCAPS) being developed to handle up to 70 different energy surveys
- In pilot stage
- Development of a Master Universe Database as a frame file for all companies reporting on a EIA survey across all energy sectors

## COMMON COLLECTION AND PROCESSING SYSTEM (CCAPS)	Objectives

- Minimize costs for maintenance, long term development, and user operations
- Common database storage system to minimize data redundancy, common tools for data management and access
- Consolidate all EIA name and address systems into one comprehensive system    -- Master Universe Database (MUD)

## Common Collection and Processing System (CCAPS):	Data Entry

- Initialize new data collection period
- Data collection
- No change in data collection instruments
- allows for multiple collection modes within a survey
- Data input to CCAPS
- Direct entry by keystroke or fast key
- Import electronic data file (PEDRO, e.g.)

## Common Collection and Processing System (CCAPS):	Data Processing

- Perform all data edits
- Set flags when edit fails
- Provide mechanism for flag resolution
- Maintain historical log of all data changes by cell
- Provide reports to facilitate/evaluate edits

## 2 Levels of Automated Edits	

- FIRST LEVEL edits are performed on data when the data are first entered from the survey and saved into the data base.
- SECOND LEVEL edits are performed using the current week/month aggregate data.

## First Level Automated Edits

- Include summing across cells within a form
- Utilize previous historical information, inclusive of imputed values on that respondent for previous period (t-1), (t-2), ...(t-n).
- Can make use of previous aggregate information for that data cell for previous periods (t-1), ...(t-n).
- Occurs during the Edit and Save process for entering data.

## Second Level Automated Edits

- Requires an aggregation of current period (t)’s data.
- Any aggregation of current period’s data may be used, not just publication aggregation level.
- E.g., A rule based on the change in respondent’s market share from period (t-1) to period (t).

## Editing Goals and Criteria

- Need the ability to view data and edit flags and correct errors online after the Edit and Save process, as well as, after 2nd level edits.
- Performance measures are required to measure both the quality of the data and Editing processing itself.

## Performance Measures for Evaluating the Editing Process

- Counts of fatal, critical, and warning flags by type of edit
- Calculation of data failure rate by type of edit
- Count of critical and warning failures that result in changed data, calculation of hit rates by type of edit
- Aggregate performance measures: percent cells flagged and changed; percent of volume changed.

## Common Collection and Processing System (CCAPS):	Data Processing

- Perform imputations, estimations and aggregations
- Data suppression
- Reports for final data evaluation and drafts of publication reports
- Export files to publication system
- Export files to other composite publication systems (analysis, Monthly Energy Review)

## Form Selection Screen

## Sample Form

## Cell Flags/History

## Comments

## Reports Selection Screen

## Features Summary

- Common form screen structure
- Respondent data, form data, comments
- Right click
- Edits, history, flags
- Allows more than 1 form at a time
- more than 1 respondent
- more than 1 historical period
- Common screen management functions
- Comments, Imputations

## The Need for Developing a Master Survey Frame Database

- Need to standardize information on Company name and addresses, Ids, etc.,  to allow transfer of information across survey systems.
- Avoids confusion and inconsistencies of name and address information contained in the various survey systems.

## The Role of a Master Frame Survey Database for a Common Survey Processing System

- The Master Universe Database contains all information concerning the businesses in the survey processing system.
- Stores all name and address information, the energy sector and activity they are in, contact persons, business affiliations.
- Contains historical records about companies no longer in business, past affiliations, and reporting history.

## MUD Filter/Search Screen

## Economic Unit Tab

## Contacts

## Relationships

## Attributes

## Comments

## CCAPS Architecture Goal

- <date/time>

**CCAPS Architecture Goal**

**Form**

**Code**

**Energy**
**data**

**Data**
**Publications**
**Analyses**

**Respondent**
**data**

## Advanced Architectural Features

- Cell based approach
- Layered Approach
- Dynamic Formulas
- Custom Controls

## Cell Approach - design system around data cells rather than the form

- <date/time>

- Cell Approach - design system around data cells rather than the form

- Form 1

- Table 1

- Table 2

- Table 3

- Table 4

- Form 2

- Table 1

- Table 2

- Table 3

- Table 4

- Form 3

- Table 1

- Table 2

- Table 3

- Table 4

**Old Approach **
**(Publication based)**

**New Approach (Cell Based)**

- Form 1

- Form 2

- Form 3

- Form 4

- Form 5

- Form 6

- Cell
- Table

## Single Cell Approach

- Survey Form Structure Stored As Data
- Cell Tracks
- Data
- Flags
- History
- Imputations

## Single Cell Data Management 

- Form EIA-x

- Historical
-  Values

- Edit
- History

- Past
- Comments

- Dimensional
- Attributes

- Location

- Time

- Resource

- Other

## Layered Approach

- Data bases

## Dynamic Formulas

- Formula Types: Edit, Impute, Aggregate, Estimate
- Data Driven Features
- Assignable to cells
- Formula arguments are in database
- Arguments are modifiable by the user
- Formulas are applicable to cell groups
- Formula application rules are determined at runtime

## Custom Controls

- <date/time>

**Common Custom Controls**
**    ****Edit**
**    ****Grid**
**    ****Check Box**
**    ****Buttons**
**    **

**Common Code**
**    ****Load**
**    ****Edit**
**    ****Save**
**    ****Display**
**        ****Historical Data**
**        ****Flags**
**        ****Comments**
**Common Features**
**    ****Color**
**    ****Font**

**Notes:**
There was a need to develop a graphical editing package for survey managers to edit data on line, make changes to the data base and re-calculate estimates.

## Interactive Graphical Editing capability

- Median editing cost is 40% of survey cost for economic surveys
- Over-identification of potential errors
- extensive manual review and man-hours
- unknown/little impact on survey results
- increased respondent burden
- risk of missing true errors
- Capitalize on efficiency of technology while acknowledging subject matter specialists expertise

**Notes:**
In Vol #2 of Statistical Data Editing, Methods and Techniques published by the UN Statistical Commission and Economic Commission for Europe,  graphical editing is defined as the exploitation of human visual perception ability to identify an anomaly, a pattern or an implied relationship in data that would take much more time and effort to find by analytic non-graphical means..experiences show that traditional editing on a case-by-case basis does not often allow to see clearly the impact of an individual data point on the aggregate estimate.
--In 1990, the Data Editing Subcommittee of the FCSM released  a WP that concluded....

## Combines Features of other Graphical Editing Systems

- From BLS: a top down approach, displays an anomaly map similar to ARIES--Summarizes anomalies at aggregate                                                                   levels through color    --indicates relationship of aggregates   --provides drill down capability to lower level aggregates and respondent level data

**Notes:**
--The Graphical Editing Analysis Query System (GEAQS) was developed by EIA to address the  over identification issue and reduce survey cost and improve survey quality.

--GEAQS combines features from 4 other graphical systems being designed or implemented at the time.............

## Combines Features of other Graphical Editing Systems

- From Census--EDA Techniques   -- Box-whisker graphs of change with query and drill down capability   --Time series and scatter graphs
- From Statistics Sweden: Windows Application  -- Anticipate client/server solution, object oriented, integrated with data processing  -- Graphics enable more effective editing rules, limits/thresholds or other parameters

**Notes:**
FROM CENSUS
Time series
depicts the seasonality of the data
Scatter graphs
this period vs. last period, respondent level, query and drill dwon capability

Box-whisker graphs
change since last period, aggregate level, distribution, query capability
FROM STATISTICS SWEDEN
Windows Applications - icons, tool bars,  dialogue boses, point and click, resize, scrolling
Integrated with data processing

## Combines Features of other Graphical Editing Systems

- From Federal Reserve Bd: PowerBuilder   -- Quicker development time, driven by iterative user  feedback/requirements
- Incorporate Visualization Techniques  2 or 3 colors/4 shades, enhances perception, regression line on scatter graph, select and regraph to uncluster data

**Notes:**
Quicker development time, less costly, easier to maintain or modify

Driven by iterative user, feedback/requirements

## Slide 117

**Notes:**
--Given the large number of edit failures common to most survey editing systems, edit failures are frequently prioritized according to some measure of importance.
--The difference function shown here is one of the most well known approach to prioritizing failures.
--This formula uses the difference in a response value between two reporting periods, times a sample weight, times an indicator  variable for edit-failed response values, times a relative importance for that response component, divided by the aggregate for that component for time t-1.  Note t-1 is used, and the aggregate for time t is not accounted for.  

## SCORE FUNCTION	

- <date/time>

- SCORE FUNCTION
- Marginal Change
- Respondent’s Contribution to Total Change
-       	For price aggregate,           P k,t = i[wi,t*Pi,k,t*Vi,k,t] 		     
- i (wi,t*Vi,k,t)
- A respondent i’s contribution is,
- 			MPi =    [wi,t*Pi,k,t*Vi,k,t]   -   [wi,t-1*Pi,k,t-1*Vi,k,t-1] 		     
-  (wi,t*Vi,k,t)              (wi,t-1*Vi,k,t-1)
- MVi =      [wi,t*Vi,k,t]   -   [wi,t-1*Vi,k,t-1] 	     
-  wi,t*              wi,t-1

**Notes:**
--Similarly, GEAQS prioritizes individual response values according to their change between reporting periods but takes into account the aggregates for both t and t-1.
--As shown here, for a price aggregate P computes as the ratio of the sum of volume and sample weighted prices to the sum of sample weighted volumes, the change in the aggregate P from time t-1 to t, can be decomposed into the contribution of each individual respondent.  Similarly, the change in aggregate volumes can also be decomposed.

## <date/time>

**Notes:**
--This example shows how an aggregate price change of 11.7 cents is decomposed into the contributions by the individual respondents to be used in prioritizing the response. 

## Slide 120

**Notes:**
--GEAQS provides the user a number of dialogue boxes including the graphic desired, the dimensions of the data, the time periods, etc.
--Geographic anomaly maps, product anomaly maps, sales category anomaly maps,scatter graphs, time series graphs, and box-whisker plots provide different ways of viewing the data
--Using a top down approach, we begin by selecting ADF, for aggregate data, a geo-graphical anomaly map, #2 distillate fuel, residential, total retail, and prices -- --After we click on ok the geographical anomaly map

## Slide 121

**Notes:**
--The geographical anomaly map shows the highest level aggregate, US, in the center with regions of the country, PADS 1-5, orbiting out from the center.  PAD 1 has 3 subpadds.  The states contained in each PAD or subpadd extend from the respective PADD.
--The legend activated, left top corner, indicates the anomalies and respective levels, determined by price ch. in the aggregate relative to the prev. time period, and for states, also relative to the PADD or subpadd for PADD   Blue--price incr., green--price decr.,  Dkst bl--  price incr.> 20 cents --As we see in this ex. US has highest level anom., track down thru P1 to P1B.  --the bottom toolbar gives more info, now drill down to product anomaly map, p icon at top toolbar 

## Slide 122

**Notes:**
--The product anomal map for the area I had marked, P1B, shows #2 distill anomaly stems from fuel oil, the lowest level product aggregate for #2 distill
--can also see in the shaded oval area, that the geographical area I had selected, also has anomalies of the highest severity for certain types of gasoline
--this gives us a simultaneous cross view of our data for that area
--can pursue the fuel anomaly and return to the gasoline anomaly afterwards
--now click on the sales type anomaly map to drill down thru the 3rd dimension of data

**Notes:**
--Sales type anomaly map for 1B, fuel oil, shows the components of retail sales, resid., com, ind, out, chm, other, as well as the wholesale components.
--can see that the residential sales are driving the retail anomaly
--from this point we are at the lowest level aggregate, the level that the data are reported at, and can drill down to the respondent level, micro data, to see which respondents contribute to the anomaly.
--1st show different drill down using B-W plot instead, return to dialogue box and select B-w for the US, #2, retail

## Slide 124

**Notes:**
--This graphic displays price change on y axis, and different product components on the x axis
--Individual circles in each plot are the geographical aggregates for that product for retail. Hour glass indic, midle 50% of data, waist is median, once click on point in a bw, bottom tool bar (left bottom) indicates product and median value
--In this ex. see 2nd bw has highest aggreg. Price increases, and when click on top node, corr. Row in spreadsheet highlights values assoc. with that point., such as current and previous pr., --across the BW can see the next BW with the lgst pr incr is the last, the one for #2 fuel, can uncluster data by drawing box and showing pts only in box, see the whiskers, pts outside whiskers are extreme values, return to drill down, were at fuel oil in 1B, residential.
--choose scatter plot to display current vs previous period prices for all respondents and imputed values

## Slide 125

**Notes:**
--So for that aggreg, the individual respondent values are plotted.
--current pr on y, previous pr on x
--CNP on y, PNC on x axis
--data points can be uncluster by clicking and holding the left mouse button, dragging the mouse to draw a box around the data points to be unclustered, and clicking the in the box icon at the top

## Increases

- Increase

- Decrease

**Notes:**
--The data points selected are unclustered
--As shown on the legend which has been activated, the circles--reported data, triang--imputed data, red indicates respondents with> 20% MS, yellow 5% MS, and <5% are blue.
--dotted line--no change in price, solid line--regression line, above price incr, below pr decr,--if I click on the red triangle (imputed, MS>20), the row in the spreadsheet with corresp data is highlighted, 
--last col show the score function, marginal contribution of the respond to the agg, in this case 108.9 cents to the cell which increase by 13.8 cents, scroll across see imputed last month, reported this month 1000 times the previous gallonage, other two large score function same respondent, 3 diff states same subpadd1B, same problem  

## Summary on Graphical Editing	

- Fast method of identifying outliers
- Allows you to test your editing rules and potential to test alternatives
- Reduces the amount of correctly reported data failing the edits
- Reduces the time and resources for validating data

## http://www.fas.harvard.edu/~stats/survey-soft/survey-soft.html#Packages

**Notes:**
If you go to this page, it contains the following Estimation, Imputation, and Analysis software, it contains a lot of hyper test links that take you to pages on the individual software

## http://www.analyse- it.com/info/genmod.htm