NORC at the University of Chicago

How to Manage an External 

Evaluation

1
By the end of this presentation, you will be able to:

• Understand the importance of managing an external evaluation

• Plan for an evaluation, including identifying roles, resources, and 

evaluation needs 

• Develop a solicitation to hire an external evaluator

• Oversee and collaborate on the evaluation process

• Recognize potential challenges

Learning Objectives
• An evaluation of a program that is conducted by an individual(s) who is 

external to the program

• External implies that the evaluator:

• Can not have a formal or personal relationship with, or stake in, the 

administration, management, or finances of the grantee or of the program 

being evaluated

• An external evaluator is objective and impartial to the evaluation results

What is an External Evaluation? 
• An external evaluator can not be:

• An individual associated with the program, and is free of conflict of 

interest

• A stakeholder, such as board members, partners, or volunteer 

affiliates 

• For example, a professor from a University through which the 

program is operated may not qualify

What is an External Evaluation? 
• Potential options for an external evaluator include:

• Research consulting firm

• College or university personnel (not affiliated with the organization)

• Faculty members

• Graduate students

• Independent consultant

Identifying an External Evaluator
• Consider what expertise is needed

• Scope and complexity may demand expertise outside of your program staff

• Check your funder’s requirements 

• Funders may require an external evaluation to ensure objectivity (e.g., Required 

for AmeriCorps ASN grantees receiving >$500K annually)

• Consider financial and staff time implications

• Both types of evaluations have cost implications, whether financial or staff time

Internal or External Evaluation? 
• Certain types of AmeriCorps grantees (i.e., AmeriCorps State and National 

grantees) may be interested in achieving a specific evidence tier.

• Preliminary evidence- internal or external

• Moderate evidence- external

• Strong evidence- external

• Consider taking a hybrid approach

• Program and external evaluator share evaluation responsibilities

• For more information about these requirements and evidence levels, visit: 

https://americorps.gov/grantees-sponsors/evaluation-resources

Internal or External Evaluation? (cont.) 
• An evaluation is like any other activity taking place within your 

organization – it requires management from start to finish

• Program staff should oversee and monitor their program’s evaluation

• Building a good relationship with your evaluator and having formal 

management practices in place ensures:

• Evaluation remains on schedule 

• Evaluation provides information that is helpful to the program

Why Does an External Evaluation Need to be Managed? 
• Task 1: Determine who will manage the external evaluation

• Task 2: Define purpose, scope, and timing

• Task 3: Determine budget and secure resources

• Task 4: Solicit and select an external evaluator

• Task 5: Establish a contract or consulting agreement

• Task 6: Manage the evaluator – communicate, monitor, and support

Overview of Key Tasks in Managing an External 

Evaluation
• Who among program staff will be responsible for managing evaluation 

activities? 

• Program Director 

• Another program staff member and/or evaluation department

• An evaluation committee or working group

Task 1: Determine Who Will Manage the External 

Evaluation
• Considerations in selecting an evaluation manager: 

• Basic knowledge of standard evaluation terms and research practices

• Strong communication skills

• Authority to make decisions about the evaluation

• Supervisory skills

• Capacity/time to assume additional responsibilities

Task 1: Determine Who Will Manage the External 

Evaluation (cont.)
• Consider the evaluation’s purpose, scope, and timing

• Why is the evaluation being done? What questions do you want answered? What 

type of evaluation should be conducted? 

• What component(s) of the program should be evaluated and over what time 

period?

• What requirements does the evaluation need to fulfill? (e.g., ASN grantees receiving 

$500K or more annually are required to conduct an external impact evaluation)

Task 2: Consider Purpose, Scope, and Timing
• Defining the purpose and scope helps with identifying an evaluator who 

is well-matched for your program 

• What qualifications and skills are needed from the evaluator to complete the 

evaluation tasks?

• What is the level of effort required from the evaluator?

Task 2: Consider Purpose, Scope, and Timing (cont.)
• Refer to your program’s logic model

• A program logic model is a detailed visual representation of a program and its theory of 

change.

• It communicates how a program works by depicting the intended relationships among 

program components. 

• For more information, review “How to Develop a Program Logic Model” located on the 

Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 2: Define Purpose, Scope, and Timing (cont.)
Task 2: Define Purpose, Scope, and Timing (cont.)

Changes?

Effects?

Impacts?

Research questions 

for process-focused 

evaluations ask:

Who?

What?

When?

Where?

Why?

How?

About:

Inputs/resources

Program activities

Outputs

Stakeholder views

Research questions 

for outcome-focused 

evaluations ask about: 

In:

(Short-term)

Knowledge

Skills

Attitudes

Opinions

(Medium-term)

Behaviors

Actions

(Long-term)

Conditions

Status

AmeriCorps’s evaluation requirements differ for small and 

large grantees

For more information, review “Overview of Evaluation Designs” located on the 

Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources
Optimal timing of external evaluation activities for large AmeriCorps State 

and National grantees during their second grant award 

• Year 1 

• Program staff define evaluation purpose and scope (by November)

• Hire external evaluator (by January)

• External evaluator completes a written detailed evaluation design plan and data 

collection instruments (by June)

• The evaluation plan is formally approved by AmeriCorps (by August)

Task 2: Define Purpose, Scope, and Timing (cont.)
• Year 2

• External evaluator conducts evaluation based on evaluation design plan from year 

1 (i.e., collect and analyze data) 

• Year 3

• External evaluator produces evaluation report in time to be submitted with 

AmeriCorps State and National re-compete application

Task 2: Define Purpose, Scope, and Timing (cont.)
Considerations for when to bring on an evaluator may depend on the 

resources available or the timing of the evaluation planning. An evaluator 

could be brought in:

• As part of the initial evaluation plan drafting process

• If the organization has access or relationships with evaluation experts who can 

serve as consultants for developing an evaluation design

• After an evaluation plan has been drafted, where the evaluator could provide 

additional insights/recommendations for further improvement

Task 2: Define Purpose, Scope, and Timing (cont.)
1. Evaluator(s) time/labor

• Estimate cost of an evaluator (# of hours needed x average labor rate) 

2. Travel 

• Consider travel expenses to attend meetings and collect data

3. Other direct costs (ODCs) 

• Estimate cost of expenses such as supplies, equipment, printing, etc.

Task 3: Determine Budget and Secure Resources
4. Overhead costs and fees

• Evaluation firms often build these costs into “loaded“ labor rates; independent 

evaluators may account for overhead differently

5. Program costs necessary to support the evaluation 

• Consider the amount of staff and volunteer time needed to manage the 

evaluation

For more information, review “Budgeting for Evaluation” located on the Resource Page 

on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 3: Determine Budget and Secure Resources (cont.)
• Task 4.1: Develop a solicitation to which potential evaluators can 

respond (also called a request for proposals (RFP))

• Task 4.2: Solicit responses to your request

• Task 4.3: Create criteria to assess and choose the most qualified 

evaluator

• Task 4.4: Evaluate the proposals and select an evaluator 

Task 4: Solicit and Select an External Evaluator
Key elements of a solicitation/RFP

1. Purpose and scope of the evaluation

2. Program background

3. Detailed description of the work plan 

4. Timelines 

5. Minimum eligibility requirements 

6. Resources and/or data to be made available to the evaluator 

7. Estimate of the funds available for the work (optional)

8. Contract vehicle

Task 4.1 Develop a Solicitation
• Develop and include the evaluation plan with the solicitation

• An evaluation plan does the following: 

• Details the program model being evaluated

• Describes and justifies the evaluation approach selected

• Provides instructions for the evaluation / a guide for each step of the 

evaluation process

For more information, review “How to Write an Evaluation Plan” located on the 

Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 4.1 Develop a Solicitation (cont.)
• The evaluation plan can include a section to describe the 

characteristics of an evaluator. This could include:

• Expertise in the evaluation design being proposed 

• Familiarity with a particular topic or focus area

• Able to conduct an objective and unbiased evaluation (i.e., is 

external to the program and there is no conflict of interest)

Task 4.1 Develop a Solicitation (cont.)
Provide instructions for responding to the solicitation

• Format of proposal response

• Is there a required template or structure to follow? Page limit? 

• Categories of information requested

• E.g., technical approach to the work, resumes of key staff, detailed budget, management 

plan, past work samples, prior experience evaluating AmeriCorps or similar programs 

• Proposal submission deadline

• Point of contact for the proposal

• Evaluation and selection

Task 4.1 Develop a Solicitation (cont.)
• Determine method(s) of attracting applicants:

• Post the solicitation on your program/organization’s website

• Share solicitation or advertise the work in other evaluation/research 

outlets

• Contact prospective evaluator(s) directly

Task 4.2 Solicit Responses to Your Request
• Potential sources for identifying evaluators:

• Professional organizations that have a network of evaluators (e.g., 

AERA and AEA)

• Local colleges and universities

• CNCS Office of Research and Evaluation list

• Evaluation divisions of state or local agencies

• Personal networks

Task 4.2 Solicit Responses to Your Request (cont.)
• Establish concrete criteria on which to assess each applicant. Criteria that 

may be used include:

• Evaluation plan/approach

• Evaluator qualifications (Years of experience, capacity, references, etc.)

• Communication style

• Project management

• Cost 

Task 4.3 Create Criteria to Assess and Choose the Most 

Qualified Evaluator
• Develop a method to apply your criteria

• Scoring/rating scale

• Checklist

• Weighting

Task 4.3 Create Criteria to Assess and Choose the Most 

Qualified Evaluator (cont.)
• Evaluate the proposals using the assessment criteria. 

• Choose the highest quality proposal that matches your criteria.

• If there is more than one high quality proposal, consider gathering additional 

information on top applicants (if allowed):

• Ask them to respond to questions 

• Ask them to participate in a “best and final” meeting

• Conduct interviews

• Check references

• Make a final decision about who to hire

Task 4.4 Evaluate the Proposals and Select an Evaluator
Standard elements of a contract:

1. Scope of work

2. Payment/invoicing

3. Point of contact for both parties

4. Product ownership and rights

5. Other special terms or conditions (e.g., modifications or termination 

of contract)

Task 5: Establish a Contract or Consulting Agreement
• Communicate

• Monitor

• Support

Task 6: Manage the Evaluator – Communicate, 

Monitor, and Support
Maintain communication throughout the project

• Project kick-off meeting

• Discussion of tasks outlined in the contract

• Details about upcoming deliverables

• Project timeline

• Questions or challenges to the work

• Regular, ongoing meetings to keep the evaluation moving in a timely and 

efficient manner 

• Ad hoc meetings to discuss specific topics

• e.g., Debrief on preliminary findings

Task 6: Manage The Evaluator – Communicate
Continually monitor progress on the evaluation and the evaluator’s work:

• Review and provide feedback on deliverables

• Enforce the schedule for completing tasks and adjust as needed

• Assess the evaluator’s skills and performance throughout the evaluation 

• Ask program staff about their interactions with the evaluator

• Find opportunities to observe the evaluator, while on-site or in meetings

• Keep up with invoicing/payments

Task 6: Manage the Evaluator – Monitor
• Evaluation design plan

• Written document that should detail all evaluation steps and activities the evaluator plans to 

conduct

• Instruments

• Surveys or assessment tools to be used to obtain data

• Monthly or quarterly progress reports 

• Reports that provide a status update on the evaluation activities

• Interim/final reports

• Request to review an outline of the report prior to writing

• Program staff may contribute to sections of report (e.g., background)

• Other deliverables

• Short memos and/or research briefs

Task 6: Manage the Evaluator – Monitor Deliverables
Provide support and feedback as needed

• Offer advice and guidance to help troubleshoot issues, as needed

• Ensure the evaluator has access to the information required

• Provide continuous input and feedback on the evaluator’s work

Task 6: Manage the Evaluator – Support
1. Evaluation of the program requires skills or expertise outside the original plan

2. The evaluator does not understand your program and/or does not make efforts to 

include your input/feedback 

3. Evaluation has limited time for analysis and reporting in Year 3

4. The evaluator leaves, terminates the contract, or does not meet contractual 

requirements

5. You are not satisfied with the evaluator’s findings

6. The evaluator changes the scope of the evaluation midcourse without approval to 

make this change

7. Serious and persistent differences in evaluation approach

Potential Challenges and Strategies 
Read the example scenario 

• Discuss the following questions in small groups:

• What could the evaluator have done differently?

• How could the grantee have effectively communicated, monitored, and 

supported the evaluation to avoid this outcome? 

• What could the grantee have done differently during the planning process?

Group Exercise
• Start the process early

• Build in time for unexpected delays to avoid potentially missing critical 

deadlines.

• Do your homework

• The evaluation manager should have a basic understanding of evaluation 

concepts and methods to ensure that he/she can effectively communicate 

your program’s evaluation needs to the evaluator.

• Ensure that the evaluator has the appropriate expertise, i.e., if proposing a 

QED, they should have prior experience implementing that design

Important Points to Remember: 
• Ask questions

• Make sure to ask questions if you are unclear about the evaluation 

approach or activities your evaluator is undertaking. 

• Collaborate often

• An evaluation is best served when both program staff and the 

evaluator are well-informed and working together.

Important Points to Remember (cont.): 
Please visit the Evaluation Resources Page for more information on the following courses: 

• How to Write an Evaluation Plan

• How to Develop a Program Logic Model

• Overview of Evaluation Designs

• Budgeting for Evaluation

https://americorps.gov/grantees-sponsors/evaluation-resources

Hiring the Right Evaluator for your National Service Program:

https://www.americorps.gov/sites/default/files/document/2015_03_19_ManaginganExternalEval

uationHiringtheRightEvaluator_ORE.pdf

Frequently Asked Questions: AmeriCorps State and National Evaluations:

https://www.americorps.gov/sites/default/files/document/2019_10_01_AmeriCorps_State_and_N

ational_Evaluation_FAQs_ASN.pdf

Resources on Evaluation
AmeriCorps State and National grantees with evaluation requirements 

may request assistance with identifying an appropriate evaluator using 

the AmeriCorps’ Technical Assistance (TA) Portal: 

https://americorpsevaluationta.norc.org/

Resources on Evaluation
Questions?
Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and 

Evaluation: evaluation@cns.gov

Thank you!