# Presentation: 273000

## Session 20: Data Collection in the Field

**Ina F. Wallace**
**RTI International**

- IES/NCER Summer Research Training Institute, June 2007

**Notes:**
You have already learned about setting up a study in terms of questions to be asked, outcomes to be measured, design features, sampling considerations, and recruiting samples.
This session will cover data collection and the kinds of considerations you will need to make prior to beginning a study.
Although most of the examples to be used will be from school-based applications, these concepts are general and apply to a variety of settings.

## Data Collection in the Field  Topics Covered

- Methods of Data Collection
- Use of State-of-the-Art Data Collection Procedures
- Ensuring Data Quality
- Conducting Small-Scale Validation Studies
- Ethical Issues in Data Collection

**Notes:**
Methods will include the various kinds of interviews, self-administered questionnaires, assessments, and observations.  We will discuss advantages and disadvantages of each as well as practical considerations
State of the art Data Collection Procedures will include computerized and web-based procedures
Ensuring Data Quality entails both training and monitoring procedures
Conducting Small scale validation studies are pre-tests and pilot studies that enable you to examine whether your collection procedures will be appropriate for your studies
Ethical Issues include obtaining informed consent and Institutional Review Boards

## Methods of Data Collection

**Interviewer Administered**
- In Person
- Surveys
- Assessments
- Telephone
**Self-Administered**
- Paper and Pencil
- Mail-in
- In-person
- Computer Assisted Self Interview
- Web procedures

**Notes:**
Will Discuss
When you would use these procedures
What the measurement errors are with such procedures
Advantages and disadvantages with each method

## Data Collection Mode Characteristics

|  | High Data Collector
 Involvement |  | Low Data Collector Involvement |  |
| --- | --- | --- | --- | --- |
|  | Paper | Computer | Paper | Computer |
| Direct Contact with Respondent | Face-to-Face Paper and Pencil Interviewing/
Assessment  (PAPI) | Face-to-Face
Computer Assisted Personal Interview/
Assessment  (CAPI) |  | (Audio) Computer Assisted Self-Interview (CASI, 
A-CASI) |
| Indirect Contact with Respondent | Telephone
PAPI | Telephone Computer Assisted Telephone Interview (CATI) | Mail, Fax, E-mail questionnare | Touch-tone Data Entry (TDE), Web, Electronic Mail Survey (EMS) |
| No Contact with Respondent | Direct Observation | Computer Assisted Data Entry (CADE) | Administrative
Records | Electronic Data Interchange (EDI) |

- Adapted From Biemer & Lyberg (2003).  *Introduction to Survey Quality*

**Notes:**
Degree of Contact with Respondent – varies from direct in face to face interviewing or assessment to no contact in which the data collector observes but does not interact; telephone is mid-way.
Degree of data collector involvement varies from high (when data collector is responsible for all of the data collection activities) to very low when the burden is either on the respondent or other systems.
Finally, the degree of computer assistance is also another characteristic.

## Face-to-Face Interviewing

**Advantages**
- Maximum degree of communication and interaction
- Can utilize PAPI or CAI
- Allows for long, complex interviews
- Variety of tactics can be used to gain cooperation
- Response rates usually high

**Disadvantages**
- High cost due to travel
- Can generate social desirability for sensitive questions
- Interviewers can affect respondents
- Can generate interviewer variance
- Presence of others can affect responses
- Falsification of interview

**Notes:**
Face-to Face Interviewing oldest interview mode
Face-to-Face Interviewing is a mode in which the data collector is physically present and interacting with the respondent.  The data collector can use either paper and pencil or computer assisted interviews
Provides maximum degree of communication and interaction makes possible
Use of many tactics to gain cooperation and motivate respondents 
Can capitalize on body language
Can make observations that can help interpret a survey
Can provide probes
Is flexible in that modes can be changed when using face-to face-interviewing—can provide CASI or A-CASI for sensitive parts and thus capitalize on best features of two different modes
We will discuss ways to overcome some of the disadvantages through training and monitoring

## Telephone Interviewing

**Advantages**
- Cost lower than face to face
- More easy to provide training, supervision, and feedback
- Social desirability bias less than with face to face
- Interviewer variance less

**Disadvantages**
- Less flexible
- No ability to use visual aids
- Must be shorter
- Questions must be less complicated
- Less capacity for motivating the respondent
- No coverage of non-telephone units
- Response rates lower

**Notes:**
Only in the last 30 years has telephone survey been acceptable
Primary benefit is that it is lower in cost than in person, yet there is some involvement of an interviewer to provide clarifications and support
Telephone surveys reduce social desirability factors (i.e., respondents wanting to please the interviewer) and are more impersonal resulting in less difficulty getting responses to sensitive questions.
When telephone interviewers are centralized, administrative tasks such as  training, supervision, and provision of feedback are streamlined.
Telephone surveys must be shorter (< 30 minutes) and far less complicated.  The upper limit for response options is 6, but fewer is better.  Also, visual aids such as response sheets are not possible with telephone interviews unless they are mailed ahead of time.
Because the pace in telephone interviews is generally faster, one tends to get more  “I don’t know responses” and shorter replies.  Also, tend to get more extreme responses—first or last response. In general, responses are less considered. 

## Mail Surveys

**Advantages**
- Lowest cost
- Low social desirability
- Can utilize visual aids
- Allows respondent to take time and look up information (if necessary)
- No interviewer variance

**Disadvantages**
- Little control over who completes the survey
- Item non-response high
- Response rates lower
- Require a long field period to obtain an acceptable response rate
- Respondents must have good literacy skills
- Limit to closed-format questions

**Notes:**
Mail in surveys use a self-administered mode since there is no interaction with a data collector.  Self-administered questions can also be given to the respondent and completed without any help but collected after completing. Self-administered questionnaires are often given in school settings.
These surveys are very inexpensive to administer and have the advantage of not introducing interviewer error
They also have reduced social desirability since there is no interviewer to please and are suitable for asking sensitive questions since there is privacy
Such surveys must be self-explanatory since there is no one to clarify questions (unless given and a proctor is available).  
No control over response process—whether intended respondent answers, why a survey is not returned (not received or person not interested).
Quality of the data depends on how good the questionnaire is since the respondent has nothing else to guide him/her. The researcher must be careful to construct questions at an appropriate literacy level, to use questions that are closed-format since open answers often do not provide useful data without an interviewer to probe incomplete responses—open-format responses are often difficult to code.
Mail in surveys have lower response rates than interviewer-assisted surveys and often require follow-up surveys that are conducted by interviewers. 
Can improve response rates for mail-in surveys by using advance letters, multiple follow-ups, real stamps on return envelopes, personalized mailings and quality of the data by designing the questionnaire to ensure completing—Dillman method.

## Computerized Procedures

- Computer Assisted Personal Interview (CAPI)
- Computer Assisted Telephone Interview (CATI)
- Computer Assisted Self Interview (CASI)
- Audio Assisted Self Interview (ACASI)
- Telephone Audio Assisted Self Interview (T-ACASI)
- Computer Assisted Data Entry (CADE)
- Web Surveys

**Notes:**
CAPI is an in-person interview in which the interviewer uses the computer to administer the questions.  Reads from the computer and enters directly onto the computer
CATI is the same kind of interview as CAPI but administered over the telephone
Both CAPI and CATI enable the interviewer to combine the data collection activities with case management activities.  Thus, as after an interview is completed and “sent” to the home office, the case management system will register it as being completed.  Updated information collected can become a part of the case control system.
CASI is a computerized self interview in which the respondent reads questions on the computer and responds.  ACASI has the respondent listening to the interview and responding. In T-ACASI the interviewer initiates the call and then turns the interview over to the computer and the respondent .  
T-ACASI is almost identical to TDE or Touch Tone Data Entry- respondent calls a computer and responds of questions asked by the computer—often used for ordering and routing of telephone calls.  Used in surveys that are very short. There is no interviewer involvement with TDE.

## Computer Assisted Procedures

**Advantages**
- Complex skip patterns and branching are easier to follow
- Previously acquired information can be programmed
- Ranges, inconsistencies, and illegitimate values  can be checked
- Help screens can be included
- Data are available for analysis more quickly

**Disadvantages**
- Time consuming to develop and program computer
- Must be adequately tested
- Can’t change things easily once fielded
- Risk of computer crashing
- Costly

**Notes:**
Computer assisted techniques offer other advantages
 can switch languages for those who are not English speaking
A-CASI is very useful for self-administration for those with limited literacy levels, including young children 
CASI and A-CASI are very effective methods to be used for obtaining sensitive information such as personal, risky or illegal behaviors.  Turner and colleagues have shown that response rates and truthfulness increase as when surveys are presented using self-administered computerized 
Data can be added to a data file ready for immediate analysis.
Disadvantage
Lead time needed often several months and must include adequate testing using a variety of different scenarios 
No quality control over data entry- no checks on data entry or coding decisions that interviewers make except to be sure that entries are acceptable and consistent.  When open-response items are needed, often verbatim recordings are made and are coded later so that quality is not compromised.

## Direct Assessment

- Used to obtain psychological and educational information about an individual
- Individually administered
- Research driven NOT for clinical or diagnostic purposes
- Follows a standardized protocol

## Observational Techniques

- Involve a researcher making observations within a natural setting
- Direct Observation - individuals know they are being observed
- Continuous Monitoring recording as much behavior as possible
- Time Allocation randomly visiting the sample to observe at different times
- Unobtrusive Observation – individuals do not know they are being observed

**Notes:**
Observations are often used in school settings for the following purposes:
Examining social behavior such as on the playground
Examining teacher behaviors in the classroom
Observing classroom environments

Unobtrusive Observations concern is that the observer may change the subject’s behavior.  Replication difficult.  Ethical issues abound unless your observations are in a public place and individuals are anonymous (e.g. observing how many times in an hour people throw trash out of their car).

## Observational Techniques

**Advantages**
- No or limited participation needed
- May be good for generating hypotheses
- Sensitive social issues can be obtained more accurately
- High in validity because of the depth of information collected

**Disadvantages**
- Variance associated with observers is similar to that found with interviewers
- Can be very costly as they are often labor intensive
- Time consuming
- Reliability can be low
- Generalizability can be low

## Summary: Issues in Choosing a Strategy

- Type of Population
- Nature of Research Issue
- Question Form
- Question Content
- Response Rates
- Costs
- Length of Data Collection

## Methods of Administration

- Standardized Interviewing Approach
- A protocol in which interviewers interact with the respondent in a manner which is consistent across all interviewers
- Conversational Interviewing
- A protocol in which interviewers modify and adapt questions to the respondent’s situation
- Hybrid Interviewing** **
- Combination of the** **two approaches

**Notes:**
Irrespective whether the interview is in person or by telephone, uses paper and pencil or is computerized, these kinds of interviewing approaches have to be considered.
Standardized Interviewing Approach is a protocol requiring interviewers to ask questions as worded, to probe, provide feedback, and interact with the respondent in a manner which is consistent across all interviewers. It attempts to standardize the behavior of interviewers to reduce the measurement error in responses due to interviewers
Assumes the questionnaire is well designed—works well in all situations and questions will not need to be rephrased—and that the situations that interviewers encounter will map well to definitions of concepts used in the questionnaire .
Assumes that exceptions can be anticipated in the design of the questionnaire.
Conversational interviewing allows interviewers to interact freely with the respondent, to modify and adapt questions to the respondent’s situation, and to assist respondents in formulating a response.  Attempt to obtain the most accurate response by  minimizing all measurement error sources.
Assumes interviewers can be trained to understand researchers’ intent
Hybrid Combination may be best under certain conditions—standardized best for most and conversational approach for those questions where a standardized approach would be awkward such as household roster.  No firm research
One study showed that using natural, person-oriented behaviors (e.g., comments such as That is nice for you, I have similar feelings) resulted in more adequate responses relative to a more formal style.  Important to train interviewers not to add the personal views to the questions.
My personal view is that standardized interviewing, with perhaps some personal touches is the ideal way to go.

## Factors that Can Explain Interviewer Variance

**Interviewer**
- Characteristics
- Appearance
- Motives
- Beliefs/attitudes
- Perceptions
- Expectations
- Skills/Knowledge

**Respondent**
- Characteristics
- Knowledge
- Interest/motivation
- Confidence
- Strength of convictions
- Expectations

**Interviewer Effects**

**Questionnaire**
- Definition clarity
- Terminology/jargon
- Question form
- Instructions
- Question wording
- Question topic

**Survey Conditions and Setting**
- Mode of interview
- Standardization
- Interviewer training
- Interviewer supervision
- Monitoring/observation

- From Biemer & Lyberg (2003).  *Introduction to Survey Quality*

**Notes:**
The dark lines around the three boxes indicate sources of variance that contribute to “interviewer error” that can be controlled (at least in part) by careful design.  Although the respondent also contributes to non-sampling error.
Unless you will rely totally on self-administered procedures, you will need to contend with variance due to the interviewer, observer, or assessor.  
Some of them individually explain measurement error with self-administered procedures, but the focus here will be on the interaction between interviewer and design factors such as the instrument, respondent, and data collection conditions and setting.
Characteristics of Interviewers and Respondents
Demographics:  Matching respondent and interviewer on characteristics such as race, gender, and age can lead to a different pattern of results than when not matched but matters most when topics are sensitive.  
Interviewer Expectations not much research evidence supporting a relationship between interviewer expectations and respondent difficulty.  No research examining the relationship between expectations, interviewer behavior, and question subject matter. Some evidence that more experienced interviewers have higher item non-response—experienced interviewers had more missing data for income questions—they may have skipped these questions or didn’t convert a refusal because they knew many individuals objected to them.
Experienced, confident interviewers generally have higher response rates—obtain more cooperation.  

## Factors that Can Explain Interviewer Variance

**Interviewer**
- Characteristics
- Appearance
- Motives
- Beliefs/attitudes
- Perceptions
- Expectations
- Skills/Knowledge

**Respondent**
- Characteristics
- Knowledge
- Interest/motivation
- Confidence
- Strength of convictions
- Expectations

**Interviewer Effects**

**Questionnaire**
- Definition clarity
- Terminology/jargon
- Question form
- Instructions
- Question wording
- Question topic

**Survey Conditions and Setting**
- Mode of interview
- Standardization
- Interviewer training
- Interviewer supervision
- Monitoring/observation

- From Biemer & Lyberg (2003).  *Introduction to Survey Quality*

**Notes:**
Interviewer Effects and the Design of the Questionnaire
Rapport that the interviewer has with the respondent and the interviewer-respondent interaction can influence answers
Standardized instruments and procedures were developed to neutralize the effect of interviewer attitudes, beliefs, behaviors, and mannerisms on survey response.
Primary cause of interviewer variability is the questionnaire design, content, and instructions.  Questionnaires that are not “straight-forward,”  where the response options are not clear, where instructions are confusing, and particularly situations that have not been specified in advance can lead the interviewers to deviate from the script. Some interviewers do it well, some do it poorly, and some do not do it at all following the practice of having the respondent interpret the question.  One study showed that conversational interviewing led to more accurate responses.
Interviewer Effects and Survey Setting
Mode of interview – Paper has greater opportunity for error than CAI which eliminates skip errors.  Mis-keying remains a potential problem with CAI and may be affected by degree of computer proficiency  However, the research indicates that recording errors are less for CAPI interviewers
With CAPI need to be in a quiet place with a seat (no “doorstop interviews” and can have individuals who refuse
Training effects—one study comparing length of training showed that the greatest interviewer variance was associated with the shortest (1/2 day) and longest (10 days) training (overconfidence).  But overall on other quality measures, those with short training were much less likely to be rated as excellent or satisfactory (41%)
We will discuss ways that this errors can be minimized through design, training, and monitoring

## Role of the Interviewer/Assessor

- Gaining Cooperation
- Converting reluctant participants
- Recognizing and dealing with refusals
- Training and Motivating Respondents
- Interviewers implicitly and explicitly teach respondents how to behave
- Obtaining Quality Data
- Ensuring Confidentiality

**Notes:**
Gaining cooperation
Locating respondents and being available for an interview when they are 
Qualities that help with converting a reluctant participant or one who has refused include:
Confident assertiveness (You will want to participate).  They have no doubt in their voices
Engaging people personally – their interactions are focused on and tailored to the respondent—responsive to the respondent
Be very familiar with the study so you can convey it
Listen carefully to the respondents questions
Gaining cooperation with a young child entails many of the same skills but also must take into account the age and needs of the child
Be aware that child may feel anxious going with a strange adult ; ensue that the interviewer/assessor is introduced to the child by a familiar person
The examiner should elicit general information to give the child time to relax and become comfortable with the t environment
Explain the purpose of the assessment and the activities that will take place
Approach it as a pleasurable undertaking
Training respondents
Showing by example how to give thoughtful responses
Provide encouragement to reinforce
Obtaining Quality Data and Ensuring Confidentiality are our next topics

## Assessing Children

- Assessor Characteristics
- Flexibility
- Vigilance
- Self-awareness
- Eye contact
- Establishing Rapport
- Allaying apprehensions
- Helping children feel at ease
- Maintaining limits on permissible behavior
- Arranging the Physical Environment

## Ensuring Quality Data

**Recruiting interviewers/assessors**
- Characteristics of all data collectors
- Characteristics of interviewers
- Characteristics of assessors
- Training on standardized procedures
- Certification as data collectors
- Monitoring performance

**Notes:**
Recruiting Data Collection Staff
Characteristics that are independent of the kind of data collection activity
Good reading and writing skills, Good computer skills, 
Tolerance for part-time, intermittent work
Often hired through temporary agencies in which no benefits are provided
Need for flexible work hours
Need for bilingual staff who can interview and assess in English and other languages as needed.
Characteristics for survey interviewers
Already described the interaction between interviewer and respondent characteristics.  Based on this research, when a survey concerns a topic that is related to the respondent characteristic (e.g., race, religion, age), it is better to match interviewer and respondent characteristics
Trained interviewers with no special background are usually the best choice
Characteristics for assessors
Do need specialized skills, particularly when assessing children – experience working with young children.  And if working in school settings, some experience in schools is also a plus
Previous experience with the instrument being used or with similar instruments is also a plus

## Training

**Content of Training**
- Presenting the study
- Contacting participants
- Administering the questionnaire/ assessment
- Probing
- Recording responses
- Confidentiality
- Interpersonal relations
- Administrative procedures

**Procedures for Training**
- Written materials
- Lecture and presentation
- Computer-based tutorials
- Planned exercises
- Role Playing
- Observation
- Certification

**Notes:**
Content of Training
Presenting the Study – include purposes of the project, sponsorship, research goals, uses of research to help data collector answer potential questions 
Details regarding purposes of questions – the kind of information they are designed to elicit will help interviewer answer potential questions from respondents
Contacting respondents – how to get case information.  Computerized v. paper.  Who sends lead letters?  In a longitudinal study, how long since they were last contacted and information already available about them
Administering the questionnaire/assessment includes:
Materials needed: show cards, computer, test kits
Conventions w/respect to wording, skip patterns
Instructions in tests (e.g., basals, ceilings)
Probing procedures that are non directive
Recording answers – paper and pencil, on computer, verbatim or coding
How to use CAI programs and specifics of the computer
How to work with respondents to motivate them and what to do in specific situations so that interviewer remains in charge
Administrative procedures needed such as transmitting or mailing data, updating information about respondents
Procedures for Training
Manuals should include all of the information about the study, materials, procedures
Lectures, live demonstrations, and videotapes are all important ways to convey content and procedures of the interview or assessment
Computer-based tutorials are needed when computers will be used
Round-robin and paired practice are vital means of learning procedures and when the trainer is present, questions can be answered
Certification procedures – can be a combination of written and observation of mock or live assessment/interview

## Interviewer Training Agenda

- Introduction to the Study
- Role of the Interviewer
- Confidentiality Procedures
- Review of Questionnaire and Protocols
- Standardized Interviewing Techniques
- Round Robin Interview
- Gaining Cooperation
- Presentation of Scenarios
- Round Robin Demonstrations
- Paired Mocks
- Certification

**Notes:**
This was a 1.5 day training for a telephone interivew in which the interviewers were given materials a head of time
Certification involved conducting a telephone interviewer and having the supervisor listen over the phone
Refresher training was held 6 weeks after the first training and lasted 1⁄2 day

## Assessor Training Agenda

| Day 1:

Large group 
Introduction to Project
Role as an Assessor
Best practices with children

Small groups 
Pre-training quiz
Introduction to all materials
Learn each assessment
Observation
Round Robins
 | Day 2:

Small groups 
Individual practice with assessments
Paired practice with assessments

Large group
Building rapport and gaining cooperation
Administrative responsibilities 
Data security 
Informed Consent | Day 3:

Small groups  
Review of assessments
Paired practice
Post-training quiz

Large group 
Concluding remarks
Certification |
| --- | --- | --- |

**Notes:**
This was a 3-day training for an assessment with young children.
Assessors received their manual ahead of time and they were to spend one day reviewing information about the study
Certification involved conducting an assessment with another trainee who used scripted responses.  The assessment was observed and graded.
Following the training, each assessor was told to practice with a child at home.
Assessors were given videotapes of an assessment with a child that they were able to review.
Any assessor who didn’t pass certification, had to be observed again with a child.

## Monitoring Performance

**What to Monitor**
- Detection and prevention of falsified information
- Compliance with rules and guidelines about administering the questionnaire or assessment
- Performance on non-interview/assessment tasks

**How to Monitor**
- Telephone verification with a sample of respondents
- Direct observation – in-person or by telephone
- Audio-recording – CARI
- Review of materials
- Performance and production measures

**Notes:**
Falsified information can include fabrication of entire interviews, deliberate skipping of some questions, or falsifying other information—usually only a problem with personal interviews.  Reasons for doing so may include pressure to get a high response rate, fears about going to see respondents in a dangerous neighborhood, not wanting to ask a sensitive question or an interviewer thinking that s/he “knows” how a respondent would answer.  Solution: Verification calls to respondent or computer audio recorded interviewing - CARI. This is generally not an issue with telephone interviewing in call centers since interviewers don’t know when they will be observed.
Compliance with rules for interviewing/testing as set forth in the training. includes the quality of the response recording or coding and following skip patterns (if PAPI), scoring of items, and presentation of questions and administration of items Sometimes interviewers are non-compliant because they view the procedures to be poorly designed and thus they try to “fix” them.  If this is known and can be corrected, the data collection will be improved. Solution: Review of paper forms will provide insight as to legibility (important if data are to be scanned), skip patterns, and scoring.  Not necessary for CAPI.  Observation will be necessary to see how the individual is asking the questions as written, whether correctly probing, handling interpersonal aspects of interview/assessment- can either do direct observation by accompanying or listening on phone OR use computer audio recorded interviewing – CARI  Particularly important with first interviews/assessments. In our work, we always try to observe a portion of assessments for reliability.
Performance on non-interview tasks such as recording the outcome of each data collection attempt, gathering information from the field.  Errors in these areas can affect determination of response rates (e.g., if ineligibles are incorrectly coded, the denominator will be higher deflating response rate) and costs. Solution: Examine production measures  such as response rates, distribution of work completed by time, 

## Human Subjects Issues

- Institutional Review Boards
- Informed Consent
- Who is carrying out research under whose sponsorship
- Description of purposes and procedures
- Duration of involvement
- Risks and Benefits
- Confidentiality
- Voluntary nature of research including stopping early and skipping questions
- Compensation
- Contact information
- Confidentiality Issues

**Notes:**
Organizations that conduct federally funded research have an IRB that oversees research involving human subjects
The DHHS regulations governing the conduct of research involving human subjects are provided in Title 45, Code of Federal Regulations, Part 46 - or 45CFR46. The criteria to be used in the evaluation process
The protocol must be evaluated to see if it is scientifically sound and worthwhile
Risks must be minimized to the extent possible
Subjects must be selected in an equitable manner
Informed consent is required (HAND OUT)
Privacy and confidentiality must be protected
The study must be adequately monitored. 
FDA also has regs
Conveyed via informed consent that should be at a language/reading level appropriate for the individual
These are the elements of consent
Certain regulations apply regarding waiver of signed consent
Special issues pertain for children.  They do not give consent but only assent and their parent/guardian provides permission.  If > minimal risk, both parents must give permission unless a waiver of elements of consent is appropriate
All of these regulations are specified in the Code of Federal Regulations and the Office for Human Research Protections at DHHS can offer guidance

## Human Subjects Issues

- Institutional Review Boards
- Informed Consent
- Who is carrying out research under whose sponsorship
- Description of purposes and procedures
- Duration of involvement
- Risks and Benefits
- Confidentiality
- Voluntary nature of research including stopping early and skipping questions
- Compensation
- Contact information
- Confidentiality Issues

**Notes:**
Confidentiality Procedures
All people who have access to the data or a role in the data collection should provide signed confidentiality agreements with penalties for failure to adhere to them
Minimize links between answers and identifiers (name, address).  
Use IDs rather than names 
If names are needed, separate identifiers from data as soon as possible
Any files that have links between IDs and names should be stored separately from data in a locked cabinet and only accessible to a limited number of individuals
Destroy identifying data as soon as possible (not always possible with longitudinal studies)
Individuals who can recognize respondents from their profile of responses should not be permitted to see survey responses
Need to disclosure risk analyses to be sure that very small categories of people are not identifiable; if so, will have to mask data
Because data can be subpoenaed by a court, researchers will sometimes request a Certificate of Confidentiality from a federal agency that is sponsoring their research

## Data Collection Evaluation

| Stage of Data Collection | Evaluation Method | Purpose |
| --- | --- | --- |
| Design | Expert review of instrument | Identify problems with instrument |
| Pretest | Behavior coding
Cognitive interviewing | Evaluate one or more stages of the response process |
| Pretest/Post-Data Collection | Respondent focus groups
Interviewer debriefings | Evaluate instrument and data collection procedures  |
| Pretest/Data Collection | Supervisor observation
Telephone monitoring
CARI | Evaluate performance of data collector  |
| Post-Data Collection | Experimentation
Internal Consistency
External validation
Re-interview/retest | Compare alternative methods of data collection
Evaluate components of MSE
Validate estimates |

- Adapted From Biemer & Lyberg (2003).  *Introduction to Survey Quality*

**Notes:**
Goal of evaluating survey error is to assess, control, and/or compensate for the errors arising from different sources. Purposes vary depending on the stage of the data collection
Design – Expert reviews of instruments both unstructured and structured reviews (using set criteria)
Pilot study may be done to examine which of competing processes one will use
Pretest examines the data collection process as it will be fielded to uncover potential problems use techniques 
Behavior coding classifies behaviors that interviewers/respondents exhibit (e.g., changing wording, ask for clarification) that have effect on quality
Cognitive interviewing – methods for studying cognitive processes of respondents that use expanded interview approaches conducted in controlled settings can be done immediately after giving a response or after entire survey. Useful in clarifying comprehension errors, problems in identifying an answer category or in recalling information
Debriefing or conducting focus groups of interviewers or respondents in which they are asked to discuss their experiences with the data collection procedures including operations, the instrument, respondents’ reactions to procedures.  These are helpful in modifying procedures.
During the data collection , evaluation efforts take the form of quality control/quality assurance techniques – ensure that the production data meet set data quality standards; some of these were described under monitoring techniques. Use indicators of data quality such as response rate, data consistency rate item non response rate.

## Post-Data Collection Evaluations

- Experiments
- Observational Studies
- Internal Consistency Studies
- External Validation Studies
- Administrative Record Check Studies
- Re-interview/Re-Test Studies

**Notes:**
Goal is to estimate the various non-sampling error biases, interviewer variance, and other sources of errors using experimental or measurement methods.
Experiments – randomly assigning sample members to various modes of interviews, methods of data collection
Example – in one pretest we embedded a study of whether an additional $25 incentive given to a father (in addition to the incentive given to the mother for the in-person interview) for completing a self-administered questionnaire would improve response rate.  Needed justification for the IRB about this procedure.
Observational Studies – examining effects that are not subject to randomization such as whether there are differences in response rates for respondents who vary in gender, race etc. 
Internal Consistency Studies – internal consistency concerns extent to which items in a scale or test measure the same concept. Can also compare two or more variables that are thought to be related (global question of income as compared to more detailed questions).  For the former, we tend to use coefficient alpha to measure internal consistency – correlation of each item with the total score.
External Validation Studies – comparing estimates from current data collection to some external source that is considered to be the gold standard.  In a recent study of children in contact with the child welfare system, we examined rates of self-reported (by A-CASI) violence perpetrated as compared to estimates obtained in a RDD survey and found our data underestimated the RDD
Hybrid – comparing results from different modes administered in the same study.  Example: comparing rates of LD as measured by direct assessment and through reports of parents and teachers
Administrative records – comparing results from survey with records.  Using previous example, can compare parents’ classifying children as have a learning disability or an IEP with school records.  School records of an IEP would be the gold standard – must be sure that the time frame and characteristic are the same.

## Post-Data Collection Evaluations

- Experiments
- Observational Studies
- Internal Consistency Studies
- External Validation Studies
- Administrative Record Check Studies
- Re-interview/Re-Test Studies

**Notes:**
Re-interview/Re-test study – respondents from the original study are revisited and re-asked the same question.  If it is an assessment, then the original individual is re-administered the same test (this is what is done when standardizing an instrument)