# Presentation: 502565

## Survey Self-Assessments

**Tom Broene **
**Statistics and Methods Group**
**Energy Information Administration**
- tbroene@eia.doe.gov
**Thursday October 20, 2005 at 3:35 pm**

## Goal and Outline

- Survey self-assessments are part of several quality programs in Europe and are well received
- EIA decided in 2003 to try something similar as one part of our work on quality
- Summary of similar projects in Europe
- Audits at Statistics Sweden
- Eurostat’s DESAP and cobwebs
- Quality program at Portugal’s INE
- EIA approach, progress, and Future plans
- Most European material from FCSM talk by Nancy Kirkendall, with information from Conference in Mainz, Germany May 2004 – thanks to David Marker, WESTAT

## Quality Audit Program - Statistics Sweden 

- Started November 2002; planned to audit 200 products in 5 years.  36 done by May 2004, 69 by Aug 2005.
- Four steps
- Survey staff assembles documentation and completes 25 page self assessment questionnaire
- Audit (peer review) by three person team of staff from other offices
- Report and action plan
- Action and follow-up

## Follow-up at Statistics Sweden

- Have staff working part-time on scheduling, follow-up, and sharing good examples
- Average 10 recommendations per review, and 2 good examples to share
- Track progress (taken care of, working on it, actions planned, nothing yet, or other)
- Categorize recommendations by process
- IT systems were most frequently recommended (65%), but seldom resolved
- Documentation second most frequent, and the highest percent resolved or underway

## Development of a Self-Assessment Program (DESAP) at Eurostat (Cobweb Diagram)

- Coherence

- Relevance

- Accuracy

- Accuracy

- Accuracy

- Timeliness &
- Punctuality

- Comparability

## «

- Internal Quality Audits at the Portuguese National Statistical Institute

**Maria João Zilhão, Quality Management Unit**
**subset of material presented at JSM 2005, August 2005, Minneapolis**

## August 22-29, 2007

- _[http://www.isi2007.com.pt](http://www.isi2007.com.pt/)_
- _[isi2007@ine.pt](mailto:isi2007@ine.pt)_

**August 22-29, 2007**

## 800 employees, 70 years of activity

Quality Management System
Framework
Involvement activities
Quality Documentation System
Internal Quality Audits

- Quality Management System
- Framework
- Involvement activities
- Quality Documentation System
- Internal Quality Audits

- Portuguese National Statistical Institute

## Internal Sessions
Monthly 1 hour presentation
Everyone is invited
Good for involvement !
Internal suggestions
Anyone may propose a suggestion about anything
The Quality Committee chooses the 3 best Suggestions on a yearly basis - Prizes
Top Management answers every suggestion
Quality award

- Monthly 1 hour presentation
- Everyone is invited
- Good for involvement !
**Internal suggestions**
- Anyone may propose a suggestion about anything
- The Quality Committee chooses the 3 best Suggestions on a yearly basis - Prizes
- Top Management answers every suggestion
**Quality award**

- Involvement activities

## Objectives – Are you processing according to your process description?

To verify the degree of procedures implementation (handbooks procedures; Checklists)

To verify if procedures are properly designed

Details
14 auditors, with 2 or 3 on each team

Improvement actions must be published, carried out by the audited departments, and followed by the audit teams

Auditing involves people in the quality program (auditors and audited teams)

Auditor / “Consultant”

- To verify the degree of procedures implementation (handbooks procedures; Checklists)
- To verify if procedures are properly designed
**Details**
- 14 auditors, with 2 or 3 on each team
- Improvement actions must be published, carried out by the audited departments, and followed by the audit teams
- Auditing **involves** people in the quality program (auditors and audited teams)
- Auditor / “Consultant”

- Internal Quality Audits at INE

## Development at EIA

- When we started in 2003, we didn’t have details from others, such as the questions asked
- Formed a team with staff from the EIA offices running surveys and hired a consulting firm
- Developed a list of questions and data to collect
- Reviewed with ASA Energy Committee in April 2004
- Considered all ongoing aspects of survey operations, but not design or start-up issues

## EIA Goals for the Self-assessments

- Completed report for each survey with the survey manager’s assessment
- Inventory of data quality measures (matrix) for all surveys
- Survey-specific goals for improvement  (focused on one-year goals for now)

- This is one approach to emphasizing the quality of our information products, which is part of our strategic plan

## Format

- Advance group meeting to explain project and distribute questionnaire
- Individual meeting for each survey between survey personnel and 1 or 2 staff from Statistics and Methods Group
- Not an audit – not demanding proof
- Had follow-up questions, so not self-conducted
- Most interviews were 75 to 90 minutes
- Interviewers drafted reports and survey staff reviewed and modified; most about 5 pages; posting internally

## Progress

- Interviews completed, but still writing summary reports and have not discussed with EIA management
- Had good cooperation and discussions
- Interviewed some IT staff, but not contractors
- Had interviewer learning curve
- Preparation and writing took a few days for each survey since not familiar with most surveys
- Best if addressing only one or two surveys in an interview
- Some managers are easier to interview
- Some have many interruptions
- No downtime anymore

## Preliminary Common Themes

- Survey respondents vary a lot
- Changes to data collected are time-consuming for us and our respondents
- Wide variety in our automated data collection
- Obtained a list of successes
- Summer interns work great for annual surveys
- Old-fashioned customer service works for dealing with respondents, at least for some surveys

## Preliminary Thoughts on Process

- Preliminary discussions with management
- Some managers already know much of what we found
- Not enough detail to answer specific questions
- Comment at FCSM that questions in a self-assessment should be tough
- Some of our questions may have been too soft. Survey managers did contradict each other on questions on costs and training for new staff.
- Goals being selected are reasonable; most are projects already underway or planned. Not all selected a goal.
- Took less time than a quality profile, but still can go out-of-date while being written

## Questions for the Committee

- EIA is interested in suggestions for the future on any aspect of this project.
- Frequency – how often should we conduct a review?  (We don’t think that we can review all surveys every year.)
- Does it make sense to review all surveys in one program area during the same year? (Should we do reviews by program?)
- Should we ask survey staff to complete the questionnaire, and then have a more rigorous review?
- Do we need a greatly expanded list of questions, such as the DESAP?
- Who should be on the review team?
- Other?