How to Manage an External Evalua
Learning objectives

By the end of this presentation, you will be • Understand the importance of managing an exter

evaluation

• Plan for an evaluation, including identifying roles

resources, and evaluation needs 

• Develop a solicitation to hire an external evaluato• Oversee and collaborate on the evaluation proce• Recognize potential challenges
What is an external evaluation? 

• An evaluation of a program that is conducted by 

individual(s) who is external to the program

• External implies that the evaluator:

– Has no conflicts of interest related to the program or the – Is objective and impartial to the evaluation results

• Potential options for an external evaluator include

– Consulting firm– College or university personnel

• Faculty members• Graduate students

– Independent consultant
Internal or external evaluation? 

• Consider what expertise is needed

– The scope and complexity of the evaluation may demand 

outside of your program staff

• Check your funder’s requirements 

– Funders may require an external evaluation to ensure ob

(e.g., CNCS requires AmeriCorps State and National grareceiving >$500K annually to conduct an external evalua

• Consider financial and staff time implications

– Both types of evaluations have cost implications, whethe

or staff time. While using program staff to conduct an evamay be less costly, it also adds to staff workload.

• Consider taking a hybrid approach

– Program and external evaluator share responsibility base

respective expertise
Why does an external evaluation need 

managed? • An evaluation is like any activity taking place with

organization – it requires management from start 

• Program staff should ultimately be responsible fo

program’s evaluation.

• Building a good relationship with your evaluator a

having formal management practices in place en– The evaluation remains on track – The evaluation provides the information your program is 
Overview of key tasks in managing an 

external evaluation

• Task 1: Determine who will manage the external • Task 2: Define purpose, scope, and timing• Task 3: Determine budget and secure resources• Task 4: Solicit and select an external evaluator• Task 5: Establish a contract or consulting agreem• Task 6: Manage the evaluator – communicate, m

support
Task 1: Determine who will manage th

external evaluation• Who among program staff will be responsible for 

managing/overseeing the external evaluation act– Program Director – Another program staff member and/or evaluation departm– An evaluation committee or working group

• In deciding who will be responsible for managing 

evaluation, consider whether the individual(s) ha– Basic knowledge of standard evaluation terms and resea

practices

– Strong communication skills– Authority to make decisions about the evaluation– Supervisory skills– Capacity/time to assume additional responsibilities
Task 2: Define purpose, scope, and tim

• Define the evaluation’s purpose, scope, and timin

– Why is the evaluation being done? What questions do yo

answered? What type of evaluation should be conducted

– What component(s) of the program should be evaluated 

what time period?

– What requirements does the evaluation need to fulfill? (e

AmeriCorps State and National grantees receiving $500K annually are required to conduct an external impact eval

• Defining the purpose and scope helps with identi

evaluator who is well-matched for your program – What qualifications and skills are needed from the evalua

complete the evaluation tasks?

– What is the level of effort required from the evaluator?
Task 2: Define purpose, scope, and tim

(cont.)• Refer to your program’s logic model

– A program logic model is a detailed visual representation 

program and its theory of change.

– It communicates how a program works by depicting the i

relationships among program components. 

– For more information, review “How to Develop a Program 

Model” located on the Resource Page on Nationalservice.gov/resources 
Task 2: Define purpose, scope, and tim

(cont.)

Research questions Who? About:

for process-focused What? Inputs/resourcesevaluations ask: When? Program activities

Where? OutputsWhy? Stakeholder viewsHow?

In:

Research questions (Short-term) (Medium-term) (Lo

for outcome-focused Changes? Knowledge Behaviors Coevaluations ask about: Effects? Skills Actions Sta

Impacts? Attitudes

Opinions

CNCS’s evaluation requirements differ for slarge granteesFor more information, review “Overview of Evaluation Designon the Resource Page on Nationalservice.gov/resources
Task 2: Define purpose, scope, and tim

(cont.)

Optimal timing of external evaluation activlarge grantees during their second grant a• Year 1 

– Program staff define evaluation purpose and scope (by No– Hire external evaluator (by January)– External evaluator completes a written detailed evaluation 

and data collection instruments (by June)

• Year 2

– External evaluator conducts evaluation based on evaluatio

plan from year 1 (i.e., collect and analyze data) 

• Year 3

– External evaluator produces evaluation report in time to be 

with AmeriCorps State and National re-compete applicatio
Task 3: Determine budget and secure 

resources

1. Evaluator(s) time/labor

– Estimate cost of an evaluator (# of hours needed x average l

2. Travel 

– Consider travel expenses to attend meetings and collect data

3. Other direct costs (ODCs) 

– Estimate cost of expenses such as supplies, equipment, prin

4. Overhead costs and fees

– Evaluation firms often build these costs into “loaded“ labor ra

independent evaluators may account for overhead differently

5. Program costs necessary to support the evaluation 

– Consider the amount of staff and volunteer time needed to m

evaluation

For more information, review “Budgeting for Evaluationon the Resource Page on Nationalservice.gov/resourc
Task 4: Solicit and select an external 

evaluator

• Task 4.1: Develop a solicitation to which potentia

evaluators can respond (also called a request for proposals (RFP))

• Task 4.2: Solicit responses to your request• Task 4.3: Create criteria to assess and choose th

qualified evaluator

• Task 4.4: Evaluate the proposals and select an e
Task 4.1 Develop a solicitation

Key elements of a solicitation/RFP

1. Purpose and scope of the evaluation2. Program background: theory of change and suppo

research evidence, logic model

3. Detailed description of the work plan (project tasks

requirements, and deliverables)

4. Timelines (project’s period of performance, key mi

and due dates for deliverables)

5. Minimum eligibility requirements (skills, knowledge

experience required of the evaluator)

6. Resources and/or data to be made available to the 7. Estimate of the funds available for the work (option8. Contract vehicle
Task 4.1 Develop a solicitation (cont.)

Provide instructions for responding to the so• Format of proposal response

– Is there a required template or structure to follow? Page li

• Categories of information requested

– E.g., technical approach to the work, resumes of key staff

budget, management plan, past work samples 

• Proposal submission deadline• Point of contact for the proposal• Evaluation and selection
Task 4.2 Solicit responses to your req

• Determine method(s) of attracting applicants:

– Post the solicitation on your program/organization’s webs– Share solicitation or advertise the work in other evaluation

outlets

– Contact prospective evaluator(s) directly

• Potential sources for identifying evaluators:

– Professional organizations that have a network of evaluat

AERA and AEA)

– Local colleges and universities– Evaluation divisions of state or local agencies– Personal networks
Task 4.3 Create criteria to assess and cho

most qualified evaluator

• Establish concrete criteria on which to assess eac

applicant. Criteria that may be used include:– Evaluation plan/approach– Evaluator qualifications– Communication style– Project management– Cost 

• Develop a method to apply your criteria

– Scoring/rating scale– Checklist– Weighting 
Task 4.4 Evaluate the proposals and se

evaluator

• Evaluate the proposals using the assessment crite• Choose the highest quality proposal that matches 

criteria.

– If there is more than one high quality proposal, consider ga

additional information on top applicants (if allowed):

• Ask them to respond to questions • Ask them to participate in a “best and final” meeting• Conduct interviews• Check references

• Make a final decision about who to hire
Task 5: Establish a contract or consul

agreement

Standard elements of a contract:

1. Scope of work2. Payment/invoicing3. Point of contact for both parties4. Product ownership and rights5. Other special terms or conditions (e.g., modific

termination of contract)
Task 6: Manage the evaluator –

communicate, monitor, and support

• Communicate• Monitor• Support
Task 6: Manage the evaluator - comm

Maintain communication throughout the pro• Project kick-off meeting

– Discussion of tasks outlined in the contract– Details about upcoming deliverables– Project timeline– Questions or challenges to the work

• Regular, ongoing meetings to keep the evaluation 

in a timely and efficient manner 

• Ad hoc meetings to discuss specific topics

– e.g., Debrief on preliminary findings
Task 6: Manage the evaluator - monito

Continually monitor progress on the evaluatthe evaluator’s work:• Review and provide feedback on deliverables• Enforce the schedule for completing tasks and ma

adjustments as needed

• Assess the evaluator’s skills and performance thr

the evaluation – Ask program staff about their interactions with the evaluat– Find opportunities to observe the evaluator, while on-site 

meetings

• Keep up with invoicing/payments
Task 6: Manage the evaluator – monito

deliverables

• Evaluation design plan

– Written document that should detail all of the evaluation s

activities the evaluator plans to conduct

• Instruments

– Surveys or assessment tools to be used to obtain data

• Monthly or quarterly progress reports 

– Reports that provide a status update on the evaluation ac

• Interim/final reports

– Request to review an outline of the report prior to writing– Program staff may contribute to sections of report (e.g., b

• Other deliverables

– Short memos and/or research briefs
Task 6: Manage the evaluator - suppo

Provide support and feedback as needed• Offer advice and guidance to help troubleshoot is

needed

• Ensure the evaluator has access to the informatio

required

• Provide continuous input and feedback on the eva

work
Potential challenges and strategies 

1. Evaluation of the program requires skills or expertise 

the original plan

2. The evaluator does not understand your program an

not make efforts to include your input/feedback 

3. Evaluation has limited time for analysis and reportin4. The evaluator leaves, terminates the contract, or do

meet contractual requirements

5. You are not satisfied with the evaluator’s findings6. The evaluator changes the scope of the evaluation m

without approval to make this change

7. Serious and persistent differences in evaluation app
Group Exercise

• Read the example scenario • Discuss the following questions in small g

– What could the evaluator have done differently?– How could the grantee have effectively commu

monitored, and supported the evaluation to avooutcome? 

– What could the grantee have done differently d

planning process?
Important points to remember: 

• Start the process early

– Build in time for unexpected delays to avoid potentially mis

deadlines.

• Do your homework

– The evaluation manager should have a basic understandin

evaluation concepts and methods to ensure that he/she ca

communicate your program’s evaluation needs to the evalu

• Ask questions

– Make sure to ask questions if you are unclear about the ev

approach or activities your evaluator is undertaking.

• Collaborate often

– An evaluation is best served when both program staff and 

evaluator are well-informed and working together.
Resources on evaluation

Go to the National Service Resource Page on Nationalservice.gov/resources for more information:https://www.nationalserviceresources.gov/evaluation-am• Resource Page on Nationalservice.gov/resources on 

an Evaluation Consultant or Staff Memberhttp://siflearningcommunity.ning.com/forum/topics/whlook-for-an-evaluation-consultant-or-perhaps-a-part-ti

• Hiring the Right Evaluator for your National Service P

http://www.nationalserviceresources.org/files/legacy/fer/download/performanceMeasurement/Hiring_the_Ruator.pdf

• Selecting an Evaluator

http://www.nationalserviceresources.org/files/legacy/fer/download/performanceMeasurement/III._Selectinguator.pdf
Questions?