1 

Handout on Best Practices for Writing an Evaluation Plan   

Example write‐ups for evaluation plan components 

Component  Sample text 1. Theory of change  When a natural disaster strikes a community, residents can find their homes destroyed an

livelihoods put on hold. Our target population‐‐low‐and‐moderate income residents‐‐is of

disaster, yet has the fewest resources to return to safe, sanitary, and secure living conditio

these residents suffer long‐term negative effects, including loss of income, housing instabhealth problems from the stress and trauma of the disruption. Our program provides such

deploying AmeriCorps members to operate a one‐stop‐shop for residents affected by disa

members coordinate volunteers, manage logistics such as material and tool delivery, and issues such as permitting and scheduling. We posit that providing a one‐stop shop for thes

low‐and‐moderate income residents obtain safe and stable housing after a disaster, result

stability, mental and emotional well‐being, and a sense of community engagement.  

2. Outcomes of interest  The outcome of interest underlying this process evaluation is that disadvantaged individua

knowledgeable of the civil legal system (including their eligibility for services, legal optionsnavigating the system) as a result of the services provided by AmeriCorps members. This p

study the preparation and ongoing member support for delivering high quality legal servic

individuals using our program’s two service delivery models. 

This evaluation will focus on measuring the intended short‐term outcomes for members oSpecifically, increases in: English language skills; professional skills; relevant professional &self‐efficacy; and social connections / decreased social isolation  

3a. Research questions (process) 

This process evaluation will address the following questions:  

(1) Are the legal services delivery models being implemented with fidelity across sites? Ha

enhancements to the service models based on the prior evaluation results been implemenwhy not.  (2) Does the training provided by the program and its host sites adequately prepare memb

deliver legal services via our program’s service delivery models?  

(3) Do the sites’ member management practices (including supervision) adequately suppo

effectively deliver legal services via the program’s service delivery models? 

3b. Research questions (impact) 

The primary questions guiding this impact evaluation is:  1. Among parents with prior CPS history, were parents who received program services les

without such services to experience new allegations of maltreatment? 
2 

Component  Sample text 

2. Which factors predict lower likelihood of recurrence, such as prior contact with the childemographic characteristics, home visiting program dosage, baseline risk factors as measureduction of risk factors as measured by AAPI change scores, or program completion? 

3c. Research questions (non‐experimental) 

The non‐experimental evaluation will address the following question:  

1.) Have host sites achieved a significant increase in watershed restoration after hosting A

for one service year? 

4a. Evaluation design (process/implementation) 

The design selected is an internal implementation evaluation using mixed methods. The st

implementation evaluation is that it will yield information to help improve the effectivene

by helping initiate or modify initial activities, provide insight into why certain goals or outc

being accomplished, and help program staff make decisions.  

4b. Evaluation design (quasi‐experimental) 

A quasi‐experimental design will be used to identify what would happen in the absence of

score matching technique will be used to identify a matched comparison group of student

the tutoring or mentoring services within each school. Students will be matched one‐to‐onneighbor matching algorithm with a .25 standard deviation caliper with exact matching onfollowing variables: grade, sex, race/ethnicity, English Language Learner status, special ed

free/reduced price meal eligibility and school.   

5a. Sampling methods (process & non‐experimental) 

Due to the small numbers, all supervisors and members will be included in data collection

interviews and/or pre/post‐tests of training. All client satisfaction surveys will be reviewed

method is needed. 

5b. Sampling methods (process) 

Sampling  for  this  study  is  based  on  the  number  of  AmeriCorps  members  serving,  the

supervisors, and the number of host sites served.  All 25 AmeriCorps members submit mo

their service requirements and the AmeriCorps Program Director reviews and follows up a

the data are complete.  The number of host site supervisors in the sample is based on the toeach  program  year,  chosen  through  a  competitive  request  for  proposal  process.  The

Manager is charged with following up with all host sites to ensure timely submittal of data

numbers table to create a random group of AmeriCorps service members and a group oour focus group. We will aim to have seven representatives in the focus group which is a cogroup allowing all members to have a chance to comment and share their thoughts and hin the conversation.   

5c. Sampling methods (quasi‐experimental) 

A power analysis was calculated to ensure that the study is adequately powered to detect

between the intervention and comparison groups. Furthermore, this power analysis enab

program teams to identify whether a sufficient number of children who meet the eligibilit

served by our program. Based on literature from interventions among similar populations
3 

Component  Sample text 

size (f=0.25) at an alpha of .05 was used to calculate the minimum sample size required to.80. The power analysis results indicated that a minimum total sample size of 130 total chapproximately 65 children per group, is required to achieve moderate power. Program maconfirmed that the available pool of children to include in the intervention and comparisoand meeting the minimum sample size required to properly power the study is feasible. Efsample sizes larger than the required minimum will ensure that attrition among students study’s power. 

6a. Data (Non‐experimental & Process) 

Data Sources This study will utilize a Pre‐ and Post‐Service Capacity Assessment Survey to measure cha

capacity and goal achievement. Both surveys will be administered online via SurveyMonk

and comparison group. ‐ The  Pre‐Survey will  consist  of approximately  20  questions, and will include  both  op

question  types. This survey will serve as a baseline measure of local government caclimate  action  domains,  including:  (1)  overall  agency  capacity,  (2)  understanding  oprogram  and  service  delivery  capacity,  (4)  stakeholder  support,  (5)  policy  awaren

awareness. Under each of these domains, respondents will rate themselves on variouLikert  scale.  The  Pre‐Survey  will  also  capture  qualitative  data  about  local  governmclimate action, and descriptive characteristics about the respondent and the agency.  

‐ The Post‐Survey will serve as an endpoint measure of local government capacity and

Pre‐Survey. The Post‐Survey will also include questions about  the agency’s achievem

staffing, project, and knowledge sharing goals.  

‐ For qualitative data collection, the evaluator will develop a semi‐structured interview

beneficiary  interviews.  Potential  interview  topics  may  include:  experience  in  the  

contributions of the member, outcomes observed at the agency level, perceived imp

the agency and community level, and program feedback. 

Data Collection Procedures Quantitative data for the study will be collected at two points in time during the 2019/20

and  Post‐Service  Capacity  Assessment  Survey  will  be  administered  online  via  Survey

intervention and comparison group before the program’s service commences  (April‐Julythe service term concludes (June‐July 2020). At each administration, agencies will receiveemails and two follow‐up calls to help ensure a reasonable response rate of 70% or highe

the post‐survey. Past administrations of the online pre‐ and post‐surveys among beneficia

to a  100%  response  rate, which indicates a  high  response  rate  can  be achieved  using a
4 

Component  Sample text 

population. The response rate will likely be lower  for comparison group agencies. Becauadministered online, data entry will be performed by the respondent at the time of survedata housed in SurveyMonkey will be exported into a statistical software package for dataQualitative data for the study will be collected at one point in time at the conclusion of theBased  on a  list  of  suggested  interviewees  developed  by  the evaluator,  the evaluation  tschedule one‐on‐one telephone interviews with up to 12 beneficiary agencies. It is expectthe interviews will be between 30 and 60 minutes each. 

6b. Data (Quasi‐experimental) 

The evaluation will include two data sources: 1. Treatment group data from 9 program sit

files from the CPS case records for both treatment and comparison groups. Our program hdatabase that includes family demographics, dosage of program participation, completionassessment data. CPS has data on prior contact, demographics, subsequent CPS contact a

such contact (substantiated, out of home placement). Our evaluator has an agreement wi

identified data for this evaluation from their records.  1. Extract a data set from our program’s database that contains the necessary independen

variables described above for families that were referred by CPS. Send dataset to CPS.  

2. CPS isolates a population in its data system within specified time parameters, and withi

the program clients. CPS will attach the program service data that was submitted for those

as simply program group cases, and export a de‐identified data set to the evaluator whichboth the treatment and comparison group regarding CPS priors and subsequent contacts.

7a. Analysis (non‐experimental) 

For all quantitative data sources, we will perform a visual analysis on the results, building 

summary tables to gain an initial understanding. This step will also involve disaggregating following ways: 

 Members that meet key criteria of our target population vs. members who do not

 Members who engaged fully in activities vs. members who did not

 Members broken out by their year of service (e.g., first / second / third year) 

We will leverage an appropriate statistical program (e.g., R or SPSS) to perform analyses avisualizations. Given the number of expected responses, we anticipate that only tests of stthat include all respondents will be relevant – comparisons between different subgroups onot be statistically significant due to the small sample size. For comparisons of pre and po

anticipating using the Wilcoxon Signed Rank test to assess statistical significance (this will appropriate for the sample as a whole, not subsets of members).

7b. Analysis (process)  For qualitative results obtained from focus groups, we will transcribe participant commen

results across focus groups thematically with supporting quotes. Given the limited budget
5 

Component  Sample text 

the number of focus group participants anticipated (10 – 15), we will not be leveraging a sanalysis software program or developing a comprehensive codebook.  

7c. Analysis (quasi‐experimental) 

Our evaluators will use multiple regression analyses to compare post‐measures between t

comparison groups to detect any significant differences. Retrospective pre‐measures, as w

such as the types of services received, will be included as predictor variables within regres

for any pre‐existing differences between the groups. The evaluators will assess baseline eq

treatment and comparison groups using ANOVA statistical tests. Baseline equivalence will

outcome variables and for other variables such as types of services received. We anticipatprocess for selecting comparison group members any baseline differences will be minimal

that are detected will be controlled for within our regression analyses. 

8. Evaluator qualifications  We plan to engage ABC Consulting as our external evaluator. ABC Consulting is a global re

firm, and is qualified to be our external evaluator because it has experience with: impact e

service, and disaster services  

We will recruit a lead evaluator and graduate student with the qualifications and skills necaccurate data, ensure effective data management, and statistically analyze the results of tThey will have the program director and program staff at their disposal, with the programinitial program overview and evaluation expectations at the start of the process. 

9. Timeline  ‐ Results from a pre‐survey will be collected in September and October 2019. We antici

survey into the existing pre‐service activities all members are required to complete. Danalyzed in November 2019, and discussed internally prior to January 2020.  

‐ Focus groups will be held near the end of the service year ‐ ideally in April 2020 at the

Conference.  

‐ Results from the post‐survey survey will be collected in May 2020.  

‐ Results from this survey will be analyzed in June 2020. ‐ A final report for Year 1 will be drafted during the summer of 2020.  

10. Budget  The budget for this evaluation is $x to cover the costs of the evaluator to perform data co

analysis, and a completed evaluation report.