Data Collection for Program Evaluation

Presented on May 21, 2015
Presenter

Andrea Robles, Ph.D.

Research Analyst

Office of Research and 

Evaluation, CNCS
Learning objectives

By the end of this presentation, you will be able to:

• Understand key questions to consider prior to selecting a 

data collection method

• Understand the importance of selecting appropriate data 

collection methods

• Describe some of the advantages and disadvantages of 

different data collection methods 

• Understand the differences between quantitative and 

qualitative methods and their roles in process and 

outcome evaluations
PART 1

Key Questions to Consider Prior to Selecting 

an Appropriate Data Collection Method 
Key questions to consider prior to 

selecting a data collection method

• Q1: What is the purpose/objective of the evaluation?

• Q2: What are the research questions?

• Q3: What is the type of evaluation design?

• Q4: What resources are available for the evaluation?
Q1. What is the purpose of the 

evaluation?

• The stated purpose/objective of the evaluation drives the 

expectations and sets boundaries for what the evaluation 

is to deliver. 

• The data that are collected should provide the information 

stakeholders need or hope to gain from the evaluation.

• Examples: 

– Produce evidence that the program is meeting its intended 

outcomes

– Understand how to operate the program more efficiently or identify 

barriers to implementation

For an overview of evaluation plans, CNCS grantees can refer to the module, 

“How to Write an Evaluation Plan” located on the Knowledge Network.
Q1. What is the purpose of the 

evaluation?: Overview of a logic model

• A logic model is a graphic “snapshot” of how a program 

works (its theory of change); it communicates the intended 

relationships among program components.

– Inputs, activities, and outputs on the left side of the logic model depict 

a program’s processes/implementation

– Changes that are expected to result from these processes are called 

outcomes and are depicted on the right side of the logic model

For an overview of logic models, CNCS grantees can refer to the module, “How 

to Develop a Program Logic Model” located on the Knowledge Network.
Q2. What are the research questions?

Differences in research questions for process and 

outcome evaluations

Changes?

Effects?

Impacts?

Research questions 

for process-focused 

evaluations ask: 

Who?

What?

When?

Where?

Why?

How?

About: 

Inputs/resources

Program activities

Outputs

Stakeholder views

Research questions 

for outcome-focused 

evaluations ask about: 

In: 

(Short-term)

Knowledge

Skills

Attitudes

Opinions

(Medium-term)

Behaviors

Actions

(Long-term)

Conditions

Status
Q3. What is the type of evaluation 

design?

Process Evaluation

Who? What? When? Where? Why? How?

Outcome Evaluation

Changes? Effects? Impacts?

• Goal is generally to inform 

changes or improvements in the 

program’s operations

• Documents what the program is 

doing and to what extent and how 

consistently the program has been 

implemented as intended

• Does not require a comparison 

group

• Includes qualitative and 

quantitative data collection

• Goal is to identify the results or 

effects of a program

• Measures program beneficiaries' 

changes in knowledge, attitude(s), 

behavior(s) and/or condition(s)

that result from a program

• May include a comparison group 

(impact evaluation)

• Can include both quantitative and 

qualitative data but typically 

requires quantitative data and 

advanced statistical methods
Q3. What is the type of evaluation design? 

Evaluation designs and CNCS’s requirements 

 

Evaluation Study Designs

Meet Requirements

Large Grantees

(annual CNCS funds 

of >= $500k)

Small Grantees/

EAP Programs (annual 

CNCS funds < $500k)

Process Design 

(Non-Experimental Design Studies)

No Yes

Outcome Design 

(Non-Experimental Design Studies)

No Yes

Outcome (Impact) Design 

(Quasi-Experimental* or 

Experimental Design Studies) Yes Yes

*Fulfills CNCS evaluation design requirement for large, recompete grantees if a reasonable comparison 

group is identified and appropriate matching/propensity scoring is used in the analysis.
Q4. What resources are available for the 

evaluation?

Questions to consider:

• How much of your evaluation budget can be allocated for 

data collection?

• Are staff members available to assist in the data 

collection for the evaluation? What are their areas of 

expertise? 

• Will you hire an external evaluator?

For an overview of budgeting and managing an evaluation, CNCS grantees 

can refer to the module, “Budgeting for Evaluation” and “Managing an 

External Evaluation” located on the Knowledge Network.
Q4. What resources are available for the 

evaluation?

Questions to consider (con’t.):

• What data are you already collecting as part of routine 

program operations? 

• How can you continue building on your data collection 

efforts? 

For an overview of budgeting and managing an evaluation, CNCS grantees 

can refer to the module, “Budgeting for Evaluation” and “Managing an 

External Evaluation” located on the Knowledge Network.
Summary: Key questions to consider prior to 

selecting a data collection method

• Q1: What is the purpose/objective of the evaluation?

• Q2: What are the research questions?

• Q3: What is the type of evaluation design?

• Q4: What resources are available for the evaluation?

Any questions?
PART 2

Data Collection Methods 
Data collection

What type of data meets your evaluation needs?

• Existing data (i.e., secondary data)

– Internal program data (e.g., participant records, program 

logs, performance measurement data)

– External datasets / administrative data (e.g., student 

records, test scores, medical records, test scores, Census 

data, unemployment insurance claims)

• New data (i.e., primary data)

– Data from surveys, assessments, interviews, and 

observations
Data collection

• Quantitative data 

– Numerical information that can be counted, quantified, 

and mathematically analyzed (e.g., test scores, ratings)

– Quantitative data are systematically collected, recorded, 

and analyzed

• Qualitative data 

– Narrative information that describes the study subject(s) 

and context (e.g., transcripts of interviews and focus 

groups, field notes from observation of certain activities)

– Qualitative data are systematically collected, recorded, 

and analyzed

• Individual anecdotes and testimonials are not qualitative data 

unless systematically collected, recorded, and analyzed
Data collection 

Quantitative Methods Qualitative Methods

Scope Less in-depth data across a larger number of 

study subjects

More in-depth data on fewer study subjects

Data 

collection

Standardized instruments with mainly 

closed-ended questions (i.e., questions with 

pre-defined response options) such as 

surveys and multiple choice 

assessments/tests

Standardized instruments and semi-structured 

interview guides mainly with open-ended 

questions (i.e., questions with no pre-defined 

response options) and can be used for 

interview, focus group, and observation 

protocols

Data 

format

Numeric Narrative

Data 

analysis

Statistical approaches are used to 

summarize the data (frequencies, means, 

crosstabs, regression ) 

Content analysis is often used in which 

themes/patterns in the data are identified, 

categorized, coded, and summarized

Results Results can be compared, summarized, and 

generalized to a larger population; May 

provide statistical evidence of program 

impact

Results provide meaning, illustrative 

explanation, and views of study subject(s); NOT 

able to provide statistical evidence of program 

impact
Common quantitative data collection 

methods

• Surveys 

– Standardized instruments that collect data from a targeted group

– Generally comprised of well-specified, closed-ended questions 

– Administered via mail, email/online, in-person, or by telephone

• Assessments/tests 

– Instruments used to assess knowledge, skill, or performance

– May be administered on paper, electronically, or via observation

– Some are commercially available or have been independently validated for 

accuracy at measuring the concept, topic or subject of interest (e.g., math 

achievement)

– Programs may choose to develop their own internal assessments/tests that are 

tailored to their program model.
Common qualitative data collection 

methods

• Qualitative Interviews 

– Collect information by talking with and listening to people 

– Performed either face to-face or over the telephone

– Rely on open-ended questions

• Focus groups

– Collect information through a guided small-group discussion

– Discussion centers around a small number of topics directed by a 

facilitator

– Often used to collect information on topic(s) that benefit from group 

discussion 
Common qualitative data collection 

methods

• Participant observation/field notes 

– Observe study participants in their “natural” settings

– May be structured or unstructured 

– Involve the researcher taking lengthy and descriptive notes of what is 

occurring

• Document review 

– Uses content analysis and other techniques to analyze and summarize 

printed material and existing written information

– Examples: Meeting minutes, program logs, training materials/manuals, 

annual performance reports, etc.
Questions??
Quantitative data collection methods: 

Advantages and disadvantages

Method Advantages Disadvantages 

Surveys • Quick and efficient

• Can cover a wide range of topics 

• Obtain responses from large 

number of people

• Can be completed anonymously

• Easy to compare and analyze

• High response rates are important 

but may be difficult to achieve

• May lack in-depth information on 

a topic/subject

• Bias responses/misunderstanding 

questions

• Potential desire of respondents to 

respond favorably

Assessments 

/Tests

• Provides objective information 

on knowledge and skills of 

participants

• Easy to compare and analyze

• May be oversimplified

• May be biased against some 

groups of test takers
Qualitative Data collection methods: 

Advantages and disadvantages

Method Advantages Disadvantages 

Qualitative 

Interviews

• Ability to explore a range and 

depth of topics

• Yields rich data

• Opportunity for interviewer to 

explain or clarify questions

• May be difficult to analyze and 

compare data

• Requires trained interviewers

• Potential desire of respondents 

to respond favorably

Focus groups • Ability to efficiently obtain 

varying opinions and 

perspectives in a short time

• Respondents build off of each 

others ideas

• Requires a skilled facilitator

• Time consuming to transcribe 

and analyze responses 

• May be difficult to schedule 

meeting with many respondents
Qualitative data collection methods: 

Advantages and disadvantages

Method Advantages Disadvantages 

Participant

observation/ 

Field notes

• Opportunity to view program 

operations in action

• Provides direct information

about behavior of individuals 

and groups

• Data occurs in a natural setting

• Requires experienced and welltrained observers

• Observer’s selective perception 

may influence data

• May be difficult to interpret 

behaviors 

Document

review

• Information is easily accessible

• Provides program history and 

development

• Opportunity to study historical 

trends

• Records may be incomplete or 

difficult to locate/access

• Analysis is limited to data 

previously collected
Mixed Methods Data Collection

• Using multiple methods helps validate findings 

and provide a more thorough assessment of your 

program (i.e., triangulating)

• See handout - Example data collection for the 

AmeriCorps homelessness program: 

– Member activity logs

– Observation 
PART 3

Considerations in Choosing a Data Collection 

Method
Considerations in choosing a data 

collection method

• Research Ethics

– Rights of human subjects, including privacy, confidentiality, and 

respect 

– Primary concern should be the safety of evaluation participants

• Institutional Review Board (IRB)

– A committee that has been formally designated to approve, monitor, 

and review research involving human subjects.

• Data Use Agreements

– Contractual documents used for the transfer and use of non-public 

use data
Additional considerations unique to 

outcome evaluations

• Reliability 

– Ability to yield consistent results under the same conditions

– Determines whether results are reproducible

– Determines the precision of measurement

• Validity 

– Ability to accurately measure the underlying concept

Reliable

Not Valid

Low Reliability

Low Validity

Both Reliable

and Valid

by Experiment-Resources.com

from https://explorable.com/statistics-reliability
Additional considerations unique to 

outcome evaluations

• Sampling and generalizability

– Selecting a representative subset of individuals from a population

– Important for ensuring findings can be generalized to an entire 

population 

• Statistical power 

– Likelihood of detecting significant effects when there is a real 

difference in the population

• Covariates 

– Outside variables that may be related to the outcome under study

– Can affect the findings of a statistical analysis
PART 4

Examples
Data for Process and Outcome 

Evaluations

• Existing Data (i.e., Secondary Data)

– Internal program records (participant records, member logs, 

performance data, etc.)

– External datasets (administrative data, etc.)

• New Data (i.e., Primary Data)

– Surveys

– Assessments/tests

– Interviews

– Focus groups

– Participant observation/field notes
Example data collection for a process 

evaluation

Process Evaluation of a Homelessness Prevention Program for Low-income Families

Research question Indicators What is collected and how? From whom / data sources? When collected and by whom?

Is the program’s 

activity – 

educational 

workshops - 

being 

implemented as 

designed? 

a) Duration of 

workshops

b) Participant 

workshop attendance 

rates

c) Topics covered by 

member

d) Members delivery 

of program curriculum 

during workshops

a, b, and c) Members 

report details about 

workshops in logs with 

pre-defined categories of 

reporting

d) observations of 

workshops

a, b, and c) Member logs

d) Evaluator observes 

members delivery of 

curriculum

a, b, and c) Evaluator 

collects the workshop logs 

quarterly

d) Quarterly observations by 

the evaluator(s) using 

structured observation 

guides
Example data collection for an impact 

evaluation: Exercise #2

Impact Evaluation of a Homelessness Prevention Program for Low-income Families

Research question Outcome of interest What is collected and 

how?

From whom / data sources? When collected and by whom?

What impact does the 

homelessness 

prevention program 

have on beneficiaries’ 

ability to hold a stable 

tenancy relative to a 

comparison group?

Tenancy status of lowincome families at risk 

of homelessness

1. Treatment group

2. Control/comparison

group
Example data collection for an impact 

evaluation: Exercise #2

Impact Evaluation of a Homelessness Prevention Program for Low-income Families

Research question Outcome of interest What is collected and 

how?

From whom / data sources? When collected and by whom?

What impact does the 

homelessness 

prevention program 

have on beneficiaries’ 

ability to hold a stable 

tenancy relative to a 

comparison group?

Tenancy status of lowincome families at risk 

of homelessness

Low income families’ 

housing stability is 

measured with a 

survey. 

1. Low-income families 

participating in the 

program serve as the 

treatment group.

2. Low-income families 

facing an immediate 

housing crisis that do not 

participate in a 

homelessness prevention 

program serve as the 

comparison group. 

The evaluator administers 

the survey at two time 

points:

- before the homelessness 

prevention program 

begins

- 1 year after the 

homelessness prevention 

program is implemented 
Important points to remember

• The evaluation’s purpose, research questions, type of 

design, and available resources should determine the data 

collection method(s) that is most appropriate. 

– CNCS has different evaluation requirements for large and small 

recompeting grantees 

• There are two general types of data collection methods – 

quantitative and qualitative – that can be used in any 

evaluation.

• Each data collection method has advantages and 

disadvantages. A mixed methods approach helps to 

overcome the weaknesses that come from a single method. 

• Process and outcome evaluations often demand different 

types of data collection methods.
Resources

• CNCS’s Knowledge Network

– http://www.nationalservice.gov/resources/americorps/evaluationresources-americorps-state-national-grantees

• The American Evaluation Association

– http://www.eval.org 

• The Evaluation Center 

– http://www.wmich.edu/evalctr/ 

• American Statistical Association

– http://www.amstat.org/sections/srms/pamphlet.pdf

• National Science Foundation

– http://www.nsf.gov/pubs/2002/nsf02057/nsf02057_4.pdf
Questions and Answers