NORC at the University of Chicago

How to Manage an External Evaluation

2023  |

1

1

By the end of this presentation, you will be able to:

Understand the importance of managing an external evaluation

Plan for an evaluation, including identifying roles, resources, and evaluation needs 

Develop a solicitation to hire an external evaluator

Oversee and collaborate on the evaluation process

Recognize potential challenges

Learning Objectives

Facilitator notes: For this presentation, we have identified a number of learning objectives.  

By the end of this presentation, you will be able to: 

Understand the importance of managing an external evaluation

Plan for an evaluation, including identifying roles, resources, and evaluation needs

Develop a solicitation to hire an external evaluator

Oversee and collaborate on the evaluation process

Recognize potential challenges in managing an external evaluation

2

An evaluation of a program that is conducted by an individual(s) who is external to the program

External implies that the evaluator:

Can not have a formal or personal relationship with, or stake in, the administration, management, or finances of the grantee or of the program being evaluated

An external evaluator is objective and impartial to the evaluation results

What is an External Evaluation? 

Facilitator notes: An external evaluation is an evaluation of a program that is conducted by an individual or a team of people who are external to the program. External implies that the evaluator does not have a formal or personal relationship with the administration, management, or finances of the grantee or its program and is thus considered objective and impartial to the evaluation results

3

An external evaluator can not be:

An individual associated with the program, and is free of conflict of interest

A stakeholder, such as board members, partners, or volunteer affiliates 

For example, a professor from a University through which the program is operated may not qualify

What is an External Evaluation? 

Facilitator notes: Since an external evaluator should be objective and impartial during the evaluation, this means that the evaluator cannot be an individual from the grantee’s own organization, including stakeholders such as board members, partners, or volunteer affiliates. For example, the evaluator should not serve on the program’s board or benefit financially from the program. They should also not be associated with the program, even if indirectly, such as a professor at the university through which the program is operated. 

An evaluation is considered independent if it uses an external evaluator who has no formal or personal relationship with, or stake in, the administration, management, or finances of the grantee or of the program being evaluated. 

4

Potential options for an external evaluator include:

Research consulting firm

College or university personnel (not affiliated with the organization)

Faculty members

Graduate students

Independent consultant

Identifying an External Evaluator

Facilitator notes: Potential sources for identifying an external evaluator include: a research consulting firm that specializes in offering services to people seeking reliable in-depth data, college or university personnel such as faculty members or graduate students that specialize in your program’s focus area, or an independent consultant who is an individual with expertise in evaluation methods that align to your needs, as well as your program’s focus area.

5

Consider what expertise is needed

Scope and complexity may demand expertise outside of your program staff

Check your funder’s requirements 

Funders may require an external evaluation to ensure objectivity (e.g., Required for AmeriCorps ASN grantees receiving >$500K annually)

Consider financial and staff time implications

Both types of evaluations have cost implications, whether financial or staff time

Internal or External Evaluation? 

Facilitator notes: Determining whether to use your own internal staff or to rely on an external evaluator to complete an evaluation of your program is a critical decision that will need to be made early on in the planning phase of your evaluation. Generally speaking, the following are key considerations when determining whether to employ an external evaluator/team for an evaluation:

Expertise: Some programs may not have internal staff with the type of technical expertise required to conduct an evaluation. Therefore, depending on the scope and complexity of the evaluation, an individual or team who is external to the program may be needed. This is particularly the case for impact designs in which there are more rigorous analyses (e.g., power analysis, baseline equivalency testing, etc.). It is considered best practice to use an external evaluator for impact evaluations as it helps provide credibility to the study findings; it also helps reduce the likelihood of bias compared to when a program staff person conducts this evaluation. For AmeriCorps State and National grantees, however, large grantees receiving more than $500,000 annually, are required to use an external evaluator. 

Funder requirements: Funders may require that the programs they fund conduct an external evaluation to ensure objectivity in the evaluation’s results. For example, as you may already know, AmeriCorps State and National grantees receiving annual funds of less than $500,000 are NOT required to conduct an external evaluation. However, AmeriCorps grantees receiving annual funds of $500,000 or more are required to conduct an external evaluation.

Cost: Both internal and external evaluations have cost implications. While using internal program staff to complete an evaluation of your program may be less costly, it also adds to staff workload. It is important to consider this trade-off and whether staff have the capacity to take on additional work to carry out the evaluation activities. Where personnel resources are more scarce than funding, an external evaluator may be the better choice.

6

Certain types of AmeriCorps grantees (i.e., AmeriCorps State and National grantees) may be interested in achieving a specific evidence tier.

Preliminary evidence- internal or external

Moderate evidence- external

Strong evidence- external

Consider taking a hybrid approach

Program and external evaluator share evaluation responsibilities

For more information about these requirements and evidence levels, visit: https://americorps.gov/grantees-sponsors/evaluation-resources

Internal or External Evaluation? (cont.) 

Facilitator notes: Another consideration is evidence tier. 

Evidence Tier: Some AmeriCorps grantees, specifically AmeriCorps State and National grantees, are required to use either internal or external evaluators in order to achieve certain evidence levels when their evaluations are reviewed as part of the Evidence Review Process that occurs at the end of their grant cycle. Internal or external-> preliminary, but to achieve the higher levels of evidence – Moderate or Strong – an external evaluator must be used in addition to conducting an impact evaluation. For more information about these requirements and evidence levels, please review the information in the Mandatory Supplement which is part of the funding announcement for AmeriCorps State and National Grantees. The 2023 version can be accessed here: https://americorps.gov/funding-opportunity/fy-2023-americorps-state-national-grants.

Hybrid approach: Another option that programs may consider is to take a hybrid approach by hiring an external evaluator to support the more technical aspects of the evaluation and have internal program staff carry out the non-technical aspects of the evaluation. For example, you may hire an external evaluator to design the evaluation, identify or develop your data collection instruments and processes, analyze the data, and write up the evaluation findings. Program staff may play a role in the evaluation by helping to collect data (e.g., administering surveys or tests) and enter or process the data to be used by the external evaluator. This may both help to defray costs and ensure that program staff will remain actively involved in the evaluation process. 

7

An evaluation is like any other activity taking place within your organization – it requires management from start to finish

Program staff should oversee and monitor their program’s evaluation

Building a good relationship with your evaluator and having formal management practices in place ensures:

Evaluation remains on schedule 

Evaluation provides information that is helpful to the program

Why Does an External Evaluation Need to be Managed? 

Facilitator notes: Deciding to use an external evaluator does not necessarily mean that program staff will not be involved in the evaluation. 

An evaluation is like any other activity taking place within your organization and therefore it must be managed. The extent and quality of the program staff’s involvement during the development and implementation of the evaluation will affect the overall progress of the evaluation and how satisfied both parties are with the end result.

Program staff should ultimately be responsible for the program’s evaluation. The evaluator is someone that your program has hired to do a particular job which is to plan for and execute the evaluation; therefore, the evaluator is someone your program oversees and monitors because they are reporting to you. 

Building a good relationship with your evaluator and having formal management practices in place, helps to ensure that your evaluation remains on track and that the evaluation provides information that is helpful to your program. 

8

Task 1: Determine who will manage the external evaluation

Task 2: Define purpose, scope, and timing

Task 3: Determine budget and secure resources

Task 4: Solicit and select an external evaluator

Task 5: Establish a contract or consulting agreement

Task 6: Manage the evaluator – communicate, monitor, and support

Overview of Key Tasks in Managing an External Evaluation

Facilitator notes: The process of managing an external evaluation breaks down into six key tasks:

Task 1: Determine who among program staff will manage the external evaluation

Task 2: Define the purpose and scope of the evaluation

Task 3: Determine budget and secure resources

Task 4: Solicit and select an external evaluator

Task 5: Establish a contract or consulting agreement

Task 6: Manage the evaluator - communicate, monitor, and support

The first three tasks – determine who will manage the evaluation, define the purpose and scope, and determine your program’s budget and secure resources – are activities that help to lay a strong foundation for your external evaluation and prepare you to effectively engage an external evaluator. It should be noted that while some order exists as to how programs generally approach these initial steps, the process largely depends on a project’s particular circumstances, so they may not occur in an ordered fashion but may be done simultaneously instead.  

It’s worth noting that these tasks, and the process in general that we describe in this presentation, are the same as those that AmeriCorps implements when managing its own program evaluations and research projects. Many of the points we will touch on are informed by our own processes and lessons learned, and reflect best practices that are used both in our own work and in the field of program evaluation. 

Please note that the tasks described on this slide do not necessarily need to follow this order- the steps are not always linear. For example, the first step, depending on circumstances could be: reach out for assistance from an evaluator first (e.g., existing relationship or consultant hired by State Commission), or grantees may develop an evaluation internally before moving onto the other steps, etc.

We will now discuss each of these tasks in more detail.

9

Who among program staff will be responsible for managing evaluation activities? 

Program Director 

Another program staff member and/or evaluation department

An evaluation committee or working group

Task 1: Determine Who Will Manage the External Evaluation

Facilitator notes: This first task involves determining who among program staff will be responsible for managing and overseeing the external evaluation activities, from the hiring of the evaluator, communicating with the evaluator, and monitoring and supporting the evaluator’s progress and quality of work. Whose role it is to lead this process really depends on the organizational context and structure of your AmeriCorps program. For example, a smaller program may have the program director fulfill this role as part of his/her overall management of the program at large. Alternatively, the program director may assign another staff member to take on this role in addition to his/her existing programmatic responsibilities. Larger programs, on the other hand, may have a separate evaluation department that manages evaluation activities for the program or may form an evaluation committee or working group comprised of key stakeholders to collectively manage the external evaluation process; however, it is recommended that one person from the committee or working group is designated as the lead point person for the evaluation. 

10

Considerations in selecting an evaluation manager: 

Basic knowledge of standard evaluation terms and research practices

Strong communication skills

Authority to make decisions about the evaluation

Supervisory skills

Capacity/time to assume additional responsibilities

Task 1: Determine Who Will Manage the External Evaluation (cont.)

Facilitator notes:

Program directors will likely be the one to choose who among their program staff is best suited to manage the external evaluation. This is an important decision that should be carefully considered as this role requires someone with the following background:

Basic knowledge about standard evaluation terms and research practices. The evaluation manager should at least be familiar with basic evaluation concepts and methods so that he/she understands the services that are being secured as part of a consulting contract. If the person who assumes the role of the evaluation manager lacks this basic evaluation knowledge, it is advised that he/she pursue training opportunities or consult the many evaluation resources, covering a range of evaluation topics, that are located on AmeriCorps’s Resource Page.

Strong communication and supervisory skills. The evaluation manager will essentially be responsible for supervising the work of the external evaluator. Thus, he/she should be able to clearly communicate (in writing and verbally) what is expected of the evaluator and provide honest and direct feedback on their work throughout the evaluation project. 

Authority to make decisions about the evaluation. The person assigned responsibility for the evaluation should either be authorized to make decisions about its design and execution or have a clear understanding of what decisions need to be communicated to other leadership staff in order to gain authorization to move forward.

Capacity or time to take on additional responsibilities associated with overseeing and making decisions about the evaluation. While the external evaluator will likely be doing the bulk of the evaluation work, managing an external evaluation takes time. 

11

Consider the evaluation’s purpose, scope, and timing

Why is the evaluation being done? What questions do you want answered? What type of evaluation should be conducted? 

What component(s) of the program should be evaluated and over what time period?

What requirements does the evaluation need to fulfill? (e.g., ASN grantees receiving $500K or more annually are required to conduct an external impact evaluation)

Task 2: Consider Purpose, Scope, and Timing

Facilitator notes: This next task is to define the purpose and scope of the evaluation which essentially establishes the general parameters for the evaluation. This involves determining: 

Why is the evaluation being done? What questions do you want answered? What type of evaluation should be conducted? 

What components of the program should be evaluated? What time period should the evaluation cover? and 

What requirements does the evaluation need to fulfill? Often times, a program’s funder has specific evaluation requirements that must be fulfilled and those requirements may drive the purpose and scope of an evaluation. For example, AmeriCorps State and National grantees that receive an average annual AmeriCorps grant of $500,000 or more must conduct an external evaluation that covers at least one program year. In addition, the evaluation must be designed to provide statistical evidence of the impact of the program compared to what would have happened in the absence of the program through the use of a comparison or control group. 

12

Defining the purpose and scope helps with identifying an evaluator who is well-matched for your program   

What qualifications and skills are needed from the evaluator to complete the evaluation tasks?

What is the level of effort required from the evaluator?

Task 2: Consider Purpose, Scope, and Timing (cont.)

Facilitator notes:

It is advised that programs start to define the purpose and scope of their evaluation as an internal activity before attempting to formally solicit an external evaluator. Having established parameters for your evaluation will be critical in helping you identify an evaluator that is best suited to carry out the type of evaluation you are seeking for your program. Specifically, knowing your evaluation’s purpose and scope should indicate what qualifications and skills are needed in an evaluator to complete the desired evaluation tasks and the level of effort that will be needed. Once hired, your evaluator should be able to use the parameters that your program has defined for the evaluation as a starting point for developing a detailed evaluation design plan that specifies how they plan to carry out the work. 

Next, we provide more information on defining the purpose and scope of an external evaluation, specifically within the context of AmeriCorps’s evaluation requirements.

13

Refer to your program’s logic model

A program logic model is a detailed visual representation of a program and its theory of change.

It communicates how a program works by depicting the intended relationships among program components. 

For more information, review “How to Develop a Program Logic Model” located on the Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 2: Define Purpose, Scope, and Timing (cont.)

Facilitator notes: As you begin to define the purpose and scope of your evaluation, it is helpful to refer to your program’s logic model. Many of you may already be familiar with what a logic model is or you’ve already developed a logic model for your program.  A program logic model is a detailed visual representation of your program and its theory of change that communicates how your program works, the resources you have to operate your program, the activities you carry out, and the outcomes you hope to achieve. Your program logic model should clearly communicate how your program works by depicting the intended relationships among program components. Key program components consist of:

Inputs or resources – which are considered essential for a program’s activities to occur. They may include any combination of human, financial, organizational, and community-based resources that are available to a program and used to carry out a program’s activities. 

Activities – which are the specific actions that make up your program or intervention. They reflect processes, tools, events, and other actions that are used to bring about your program’s desired changes or results.  

Outputs – what a program’s specific activities will create or produce, providing evidence of service delivery. 

Outcomes – the specific changes that may result from a program’s activities or intervention. A program’s outcomes fall along a continuum, ranging from short- to long-term results. 

Logic models are typically read from left to right, employing an if-then sequence among key components of a program. A generic example is shown here. It reads, if your program has these inputs or resources, then it can carry out these activities. If your program carries out these activities, then it can produce these outputs. If your program has produced these outputs, then it will achieve these outcomes. 

Additionally, we can think of a logic model as essentially having two “sides.” The process side focuses on how the program’s activities are carried out or its planned work – inputs, activities, and outputs (direct products). The outcomes side of the logic model describes the expected sequence of changes that the program hopes to accomplish, which can be short-term, medium-term, and/or long-term changes. The outcomes side reflects the changes, effects, or impact of the program.  For more information on developing a program logic model, we highly encourage you to review the webinar How to Develop a Program Logic Model located on AmeriCorps’s Resource Page on Nationalservice.gov/resources. 

14

Task 2: Define Purpose, Scope, and Timing (cont.)

Changes?

Effects?

Impacts?

Research questions for process-focused evaluations ask:

Who?

What?

When?

Where?

Why?

How?

About: 

Inputs/resources

Program activities

Outputs

Stakeholder views

Research questions for outcome-focused evaluations ask about: 

In:

(Short-term)

Knowledge

Skills

Attitudes

Opinions

(Medium-term)

Behaviors

Actions

(Long-term)

Conditions

Status

AmeriCorps’s evaluation requirements differ for small and large grantees

For more information, review “Overview of Evaluation Designs” located on the Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Facilitator notes: Keeping in mind that there are two “sides” to a logic model, a process and an outcomes side, first let’s consider AmeriCorps’s evaluation requirements for large grantees (e.g., State and National grantees receiving $500K or more annually). As mentioned earlier, large grantees are required to conduct an impact evaluation, that is, an evaluation designed to assess changes occurring in the program’s beneficiaries that can be attributed to the AmeriCorps program. Specifically, impact evaluations concern the outcomes that are depicted on the right side of a logic model as they seek to measure changes in outcomes that are directly attributable to a program. An example of an impact evaluation question is: What impact did the AmeriCorps program have on beneficiaries’ knowledge compared to those who didn’t participate in the program? Answering questions of this type, involves including a comparison or control group (which is needed to produce estimates of what would have happened in the absence of the program). This allows for a comparison to be made between what actually happened and what would have happened in the absence of the intervention.

On the other hand, a process evaluation focuses on the program components that are depicted on the left side of a logic model, answering questions about the program’s inputs, activities, and outputs. This type of evaluation examines how well the program’s implementation matches the theory behind its creation and confirms what the program actually does on the ground. Whereas conducting an impact evaluation fulfills requirements for all grantees, a process evaluation only fulfills the evaluation requirements for small grantees (i.e., grantees receiving funding of less than $500,000 annually). For more information on process and impact evaluations, please see the webinar Evaluation Design located on the AmeriCorps’s Resource Page on Nationalservice.gov/resources. 

For large grantees in particular, AmeriCorps’s evaluation requirements provide a framework for defining the purpose and scope of the evaluation as the requirements clearly specify the type of evaluation (i.e., impact evaluation) to be conducted and also the timeframe (one-year) to be covered. This means that large grantees should narrow in on the outcomes side of their logic model as they work to define the questions (in general) that they want or need answered about their program and which outcome(s) they intend for the evaluation to focus on. Having a clear sense of your evaluation’s purpose and scope will help you screen for an evaluator who is well-matched to your program and the evaluation you are planning to complete. As most programs have limited resources to put towards an evaluation, it is important to note that it is not necessary to evaluate every aspect of your program as depicted in your logic model. Your evaluation may have a narrow focus (e.g., address questions about one or a few of your program’s expected outcomes) or it can have a broader focus (e.g., address questions about your program’s processes and expected outcomes), depending on the information you hope to and the resources you have available. In general, it is better to conduct an effective evaluation of a single program component than to attempt an evaluation of several components or an entire program without sufficient resources.

15

Optimal timing of external evaluation activities for large AmeriCorps State and National grantees during their second grant award 

Year 1 

Program staff define evaluation purpose and scope (by November)

Hire external evaluator (by January)

External evaluator completes a written detailed evaluation design plan and data collection instruments (by June)

The evaluation plan is formally approved by AmeriCorps (by August)

Task 2: Define Purpose, Scope, and Timing (cont.)

Facilitator notes: Another step in defining the scope of your external evaluation is to determine the timeframe of the evaluation activities. Again, it is important to consider AmeriCorps’s evaluation requirements. Specifically for large State and National grantees, AmeriCorps requires an external evaluation to be conducted during the three-year period of their second grant award and submit a report of the findings from the evaluation when they re-compete for a third grant award. Additionally, the evaluation must cover at least one year of AmeriCorps-funded service activity. 

Given those requirements, this slide breaks down the three-year grant cycle (with each year starting in Sept and ending in the following August). It provides recommendations for the timing of key evaluation activities to ensure that an evaluation report is completed in time for submission with your AmeriCorps State and National re-compete application during year 3. 

Year 1: The first year of your grant cycle is viewed as a planning year. It is recommended that program staff decide on the external evaluation’s purpose and scope by November (or earlier), hire an external evaluator by January (or earlier), and have the external evaluator complete a written detailed evaluation design plan, including the development of any data collection instruments to be used (by June or earlier). Additionally, the plan should be formally approved by AmeriCorps by August of the first year, but preferably sooner. 

16

Year 2

External evaluator conducts evaluation based on evaluation design plan from year 1 (i.e., collect and analyze data) 

Year 3

External evaluator produces evaluation report in time to be submitted with AmeriCorps State and National re-compete application

Task 2: Define Purpose, Scope, and Timing (cont.)

Facilitator notes: Year 2: This second year of your grant cycle is when your external evaluator would ideally conduct the evaluation (i.e., collect and analyze data). Since AmeriCorps requires that your evaluation covers one year of AmeriCorps-funded service activity, the timing of your data collection should reflect this requirement. For an impact evaluation, this often requires collecting baseline data in the fall of year 2 and follow-up data in the spring/summer of year 2. It is recommended that the evaluator analyzes the data during the final months of year 2.

Year 3: This third year of your grant cycle is critical in that an evaluation report will need to be completed (and reviewed by your program staff) in time to be submitted with your AmeriCorps State and National re-compete application. With applications typically due during year 3, it is recommended that the initial months of year 3 are when the evaluator writes and finalizes the evaluation report, relying on input and feedback from the grantee. It is also important to note that the analysis will need to be completed prior to the next GARP. Therefore, the evaluator should complete their portion of the analysis and report by the fall or winter of the third year. 

Again, these are recommended milestones for the three-year grant cycle and should be tailored to fit your specific evaluation. Note that hiring an external evaluator early in your three-year grant cycle is important to keeping on schedule because it usually takes three to six months of planning and preparation before data collection can begin. By delaying the hiring process, your evaluator may not have enough planning time to begin collecting data for the evaluation in the beginning of year 2 (e.g., baseline or pre-intervention data). 	

17

Considerations for when to bring on an evaluator may depend on the resources available or the timing of the evaluation planning. An evaluator could be brought in:

As part of the initial evaluation plan drafting process

If the organization has access or relationships with evaluation experts who can serve as consultants for developing an evaluation design

After an evaluation plan has been drafted, where the evaluator could provide additional insights/recommendations for further improvement

Task 2: Define Purpose, Scope, and Timing (cont.)

Facilitator notes:

When to bring on an evaluator may depend on several factors, which contributes to the earlier point that the process isn’t necessarily linear. Here are some examples of things that can influence when a grantee may choose to bring on an evaluator:

Grantees or State Commissions may have long standing relationships with an evaluator who can support grantees when writing their initial plan draft for the application. 

State Commission may have evaluation experts or consultants but are not necessarily going to be the evaluators themselves, who can support the grantee with an initial draft or revisions. Then, after approval, grantees will hire an external evaluator.

Other grantees may not hire an evaluator until after their evaluation plan is submitted. In these instances, it is important to include the plan as part of the RFP for hiring an external evaluator. It will help determine whether an evaluator is a good fit for the design being proposed. Once hired, the evaluator can work with the grantee to make revisions to the plan – whether it is to help them obtain approval or further refine processes for an already approved plan.

We will discuss budgeting and securing resources for an evaluation on the next slide. 

18

Evaluator(s) time/labor 

Estimate cost of an evaluator (# of hours needed x average labor rate)  

Travel 

Consider travel expenses to attend meetings and collect data

Other direct costs (ODCs) 

Estimate cost of expenses such as supplies, equipment, printing, etc.

Task 3: Determine Budget and Secure Resources

Facilitator notes: This next task involves establishing a realistic budget for your evaluation and securing resources. Once you have a sense of your evaluation’s purpose and scope, it is important to estimate the cost of hiring an external evaluator and whether your program has the funds available to cover that cost. It may be that your program has already designated a certain amount of your overall program funds for evaluation purposes. In this case, you want to be sure that the purpose and scope of your evaluation is in line with what you have budgeted, otherwise, you will need to make adjustments to your program’s budget or raise additional money to cover the expected cost of the evaluation. 

While this presentation does not provide details on how to develop an evaluation budget or estimate the amount of money your program needs to conduct one, there are resources on AmeriCorps’s Resource Page on Nationalservice.gov/resources that cover this topic in more depth. Also, it may be helpful to talk with staff from similar programs or organizations in your community who have conducted evaluations. They may be able to tell you how much their evaluations cost and whether they were satisfied with the services that were provided. If you do not have experience developing budgets, it may be helpful to consult AmeriCorps’s core curriculum module, “How to Budget for an Evaluation” located on the Resource Page on Nationalservice.gov/resources.

In general, there are 5 main components of an evaluation budget:

Evaluator/consultant time (or labor): In deciding to use an external evaluator, a major component of your evaluation budget will be labor, or the cost of paying the evaluator’s fees to conduct the evaluation. When estimating the cost of external evaluator time, consider the evaluation’s purpose, scope, and methods and estimate the time (# of hours) the evaluator(s) will need to spend performing each of the tasks for which they will be responsible. To calculate an estimated cost, you’ll need to multiply the total number of hours needed by the evaluator’s labor rate. 

Travel: Consider whether your evaluation will require travel for the evaluator to visit program sites for data collection or to your organization’s office for meetings. These costs should be factored into your budget.

Other direct costs (sometimes referred to by the acronym ODCs) which are a cost category that covers expenses for supplies and equipment, communication, printing, and materials directly related to your evaluation project that are not labor or travel. Note that this category includes potentially high cost items such as purchasing survey instruments or data collection platforms, and purchasing datasets. 

19

Overhead costs and fees

Evaluation firms often build these costs into “loaded“ labor rates; independent evaluators may account for overhead differently

Program costs necessary to support the evaluation 

Consider the amount of staff and volunteer time needed to manage the evaluation

For more information, review “Budgeting for Evaluation” located on the Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 3: Determine Budget and Secure Resources (cont.)

Facilitator notes: Additional considerations (continued)

Overhead costs and fees: These include operating expenses such as the cost of office space, utilities, etc. Overhead costs for large evaluation firms are often built into their “loaded” labor rates, but be aware that university staff or independent consultants may account for overhead differently. Also, fees charged for evaluation will depend on the type of contract you issue when hiring the evaluator. For example, federal agencies like AmeriCorps typically issue contracts that are either cost plus fixed fee, firm fixed price, or time and materials. While we will not describe the details of each type of contract here, it is important to understand that contract types for evaluation services do vary.

Program costs necessary to support the evaluation – Importantly, you should account for the fact that there may be significant investments of program staff and volunteer time (spent on activities like quality control and monitoring) that raise the overall cost of the evaluation. Program staff and/or volunteers can support the external evaluation as long as there is a lead external evaluator that is clearly independent from the program and as long as these individuals are not responsible for managing data or conducting analysis. The contributions of staff and/or volunteers’ time to the evaluation may not appear on a spreadsheet of direct costs, but the investment needs to be considered, especially if you seek lower cost options. 

Again, it is important to note that if you estimate that your evaluation will cost more than the funds your program has available, you should try to reallocate your program budget, and/or work on securing additional funds and resources to avoid a cost overrun associated with the work. It is critical that you are certain you have the financial resources to cover the required costs before taking any steps towards implementation. 

[Pause for questions; time check: ~30 min.]

20

Task 4.1: Develop a solicitation to which potential evaluators can respond (also called a request for proposals (RFP))

Task 4.2: Solicit responses to your request

Task 4.3: Create criteria to assess and choose the most qualified evaluator

Task 4.4: Evaluate the proposals and select an evaluator 

Task 4: Solicit and Select an External Evaluator

Facilitator notes: At this point, after having completed a thoughtful assessment of the evaluation you are seeking for your program, the next task is to identify and hire an external evaluator. Whether you opt to use a competitive selection process or a less formal process for selecting an evaluator (e.g., sole source), the right evaluator for your AmeriCorps program should be able to demonstrate that their experience, training, and work style is a good match for your program and the type of evaluation you want completed. 

In general, there are four basic steps for soliciting and choosing an evaluator. 

Task 4.1: Develop a solicitation to which potential evaluators can respond (sometimes called a request for proposals (RFP)). An evaluation plan could be included as part of the RFP.

Task 4.2: Solicit responses to your request for bids

Task 4.3: Create criteria to assess and choose the most qualified evaluator

Task 4.4: Evaluate the proposals and select an evaluator

It’s important to note that there may be specific processes and restrictions related to procurement that your program must follow, especially if your organization is a public agency. These steps outlined here are intended to serve as a guide and should be considered alongside any procurement protocols established by your organization.

We’ll talk about each of these steps in more detail on the next slides.

21

Key elements of a solicitation/RFP

Purpose and scope of the evaluation

Program background

Detailed description of the work plan 

Timelines 

Minimum eligibility requirements 

Resources and/or data to be made available to the evaluator 

Estimate of the funds available for the work (optional)

Contract vehicle

Task 4.1 Develop a Solicitation

Facilitator notes: If you are plan to use an external evaluator, more than likely you will need to develop a solicitation or a Request for Proposals (RFP) to solicit proposals for the evaluation project. Your evaluation RFP should be detailed and specific regarding the activities and products you will pay for, schedules, responsibilities, and other specific needs. This is why it is important to have established a clear understanding and consensus about the evaluation scope, timeline, and budget because that information will form the basis of the solicitation/RFP for your evaluation. 

We’ll provide some guidelines for preparing an evaluation solicitation/RFP. 

A comprehensive solicitation/RFP contains the following elements:

A description of the purpose and scope of the evaluation. This provides an overview of the evaluation, what questions are to be answered, which aspects of your program the evaluation is to focus on, and over what time period. Unless you are suggesting that the applicant develop the entire evaluation plan, you should explain the type and level of evaluation that you desire, for example, process or outcome evaluation. If you have already selected a particular evaluation design and methodology, then it is important to include this information in the solicitation. Reference to similar evaluations might be wise. It may also be helpful to include a copy of your most recent evaluation plan in the solicitation; this will also give potential evaluators a sense of what you are interested in evaluating and what you hope to learn. Including the evaluation will also help determine whether an evaluator is familiar with the design of interest, the topic area, and also start conversations on how to further improve the evaluation. 

Background information on the program. It is important to provide a brief description of the context, history, and current status of the program. Include a description of the program’s theory of change, that is, the theory that underlies how the program is designed to work, and supporting research evidence.

Description of the work plan. This element is really the core component of your solicitation and is often written as a list of tasks with a description of each task that the evaluator is to complete. These tasks might entail developing an evaluation plan, developing data collection instruments, selecting the sampling procedures and drawing of the evaluation sample, recruiting study participants, collecting and analyzing data, writing a final report, and presenting evaluation results. It is important to include any requirements (e.g., meetings the evaluator is to participate in, data sharing requirements), and a list of deliverables or products to be developed (e.g., evaluation plan, final report). If there are specific details about your evaluation that you already know you want the evaluator to execute, be sure to include these details in the solicitation itself. For example, if you know in advance that you want the evaluator to conduct two rounds of data collection (one at the beginning and another at the end of the year) or design a quasi-experimental evaluation using a matched comparison group, this information should be provided to applicants. Again, this is one of the most important sections of your solicitation/RFP. The quality of the proposals you receive will depend on the quality of the information in this section. Please see the handout titled ”Sample Work Plan” for a sample work plan section of a solicitation for hiring an evaluator to conduct an impact evaluation of a hypothetical AmeriCorps program. 

Timelines. This section should detail the project’s period of performance, indicating key milestones and due dates for deliverables. The requirements of your evaluation most often determine its duration. As noted earlier, this holds true for AmeriCorps State and National grantees as AmeriCorps’s evaluation requirements largely determine the timeline of evaluation activities. (NOTE FOR FACILITATOR: May refer back to slide 15).

Minimum eligibility requirements. To identify an evaluator who is the “right fit” for your project, it is important to specify the key qualifications you are looking for. This is essentially the skills, knowledge, and experience required of the evaluator and any other criteria that applicants must meet in order to qualify for the work. For example, are there any geographic limitations (e.g., must be local) or tax or business requirements (for-profit/non-profit, private firm, individual, university institute or department, etc.)? Do you want the evaluator to have a certain number of years of experience and in a specific type of evaluation design (e.g., outcomes evaluation, process evaluation, etc.)? Should the evaluator have a certain level of familiarity with programs similar to yours? Does the evaluation require experience in collecting data, conducting certain types of analyses of data, or writing reports? Add criteria that will help you identify evaluators with past experience doing the specific evaluation design type (QED, or RCT) you require, plus have experience gathering and analyzing the relevant kind of data (qualitative, quantitative, both) the evaluation will generate, and have subject matter experience in your program’s focus area. [Give an AC related example]

Resources and/or data. Note any resources or data that will be made available to the evaluator in order for potential evaluators to factor this into their proposal. 

Estimate of the funds available for the work. While not required, it is helpful to provide prospective evaluators with information on the scale of funding you can support. Otherwise, applicants may propose evaluation projects that vary widely in scope and methods. If you wish to spur competitiveness in the solicitation/ RFP process without revealing how much money you plan to spend on evaluation, you may suggest a ceiling on the evaluation project costs and indicate the project cost will be a significant factor in the proposal evaluation. 

Contract vehicle. Specify what type of contract will be used to commission the work (e.g., time and materials, fixed price, or cost reimbursement). 

Most states and other jurisdictions have additional sections and subsections (e.g., financial disclosure statements, affidavits of various kinds, etc.) that are required for an RFP, so it is important to check with the appropriate procurement office in your state or local government to make sure your RFP contains all of the necessary information. You may also find it helpful to obtain a copy of a successful RFP that has passed the procurement office standards, if possible. 

[Time check: ~40 min.]

22

Develop and include the evaluation plan with the solicitation

An evaluation plan does the following: 

Details the program model being evaluated

Describes and justifies the evaluation approach selected

Provides instructions for the evaluation / a guide for each step of the evaluation process

For more information, review “How to Write an Evaluation Plan” located on the Resource Page on https://americorps.gov/grantees-sponsors/evaluation-resources

Task 4.1 Develop a Solicitation (cont.)

Facilitator notes: Every grantee is required to draft an evaluation plan which goes through an approval process, which includes your plans for an internal or external evaluation (more on this to come)

An evaluation plan: 

Details the program model being evaluated

Describes and justifies the evaluation approach selected

Provides instructions for the evaluation / a guide for each step of the evaluation process. 

Your evaluation plan is useful because it helps you to decide what information is needed to address the evaluation objectives

It also helps identify methods for getting the needed information, determine a reasonable and realistic timeline for the evaluation, and creates a shared understanding between stakeholders (e.g., the grantee staff, evaluator, AmeriCorps staff).

Please see the “How to Write an Evaluation Plan” course for more information at Nationalservice.gov/resources.

23

The evaluation plan can include a section to describe the characteristics of an evaluator. This could include:

Expertise in the evaluation design being proposed 

Familiarity with a particular topic or focus area

Able to conduct an objective and unbiased evaluation (i.e., is external to the program and there is no conflict of interest)

Task 4.1 Develop a Solicitation (cont.)

Facilitator notes: Your evaluation plan includes a section to describe your evaluator.

You will describe: 

How your evaluator is sufficiently qualified and meets AmeriCorps evaluation requirement for the applicant (large or small- large requires an external evaluator). 

Whether the evaluator is internal or external to the organization. 

If an external evaluator has been identified, the description must show that: 

The evaluator is independent from the program and 

The evaluator is qualified to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts). 

Please note, if the evaluator is not yet identified or hired, you will describe the required and/or preferred qualifications that you will seek in the hiring process.

24

Provide instructions for responding to the solicitation

Format of proposal response

Is there a required template or structure to follow? Page limit? 

Categories of information requested

E.g., technical approach to the work, resumes of key staff, detailed budget, management plan, past work samples, prior experience evaluating AmeriCorps or similar programs 

Proposal submission deadline

Point of contact for the proposal

Evaluation and selection

Task 4.1 Develop a Solicitation (cont.)

Facilitator notes: Lastly, the solicitation should include detailed instructions on how evaluators are to apply for the position, identifying all information and documentation they must provide to those heading the selection process. AmeriCorps recommends that grantees solicit a formal written proposal in response to the solicitation. The proposal provides an opportunity to review the evaluator’s approach and writing style and will ultimately serve as the basis for the contract, if you choose to hire them. 

Format of proposal response: Do you want to provide a template or a structure that needs to be followed? Do you want to set a page limit? Do you prefer any font or spacing requirements? 

Categories of information requested: You may request the types of information the proposal should include, such as a technical approach to the work, qualifications of key staff, a detailed budget, management plan, deliverable schedule, past work samples, etc. List the information from the applicant that will help you determine if they are a good match for the work. 

Proposal submission deadline: Include the date and time the proposal is due.

Point of contact for the proposal: Provide a point of contact to whom candidates may direct inquiries and submit their proposal.

Evaluation and selection: Provide applicants with information on how their applications will be evaluated. This section should outline the proposal evaluation process, the persons who will evaluate them, the evaluation criteria, and weights or mathematical calculations that will be applied (if any), and how and when the final selection will be announced. 

25

Determine method(s) of attracting applicants:

Post the solicitation on your program/organization’s website

Share solicitation or advertise the work in other evaluation/research outlets

Contact prospective evaluator(s) directly

Task 4.2 Solicit Responses to Your Request

Facilitator notes: After you have developed a solicitation or job description, the next step is to request responses. This involves determining the method(s) you will use to attract bidders. You may choose to: 

Post the solicitation on your program or organization’s website

Share the solicitation with other outlets where you might reach prospective evaluators. For example, advertising in your local paper, posting the position at a local college or university, or working with your local government's human resource department (if you are a public agency) are potential ways of soliciting applications. Agency newsletters, local and national meetings, and professional journals are additional sources for posting your solicitation. It is wise to advertise as widely as possible, particularly if you are in a small community or are undertaking an evaluation for the first time. Several advertising sources will ensure that you receive multiple proposals. 

Directly contact prospective evaluator(s) who you have identified to inform them of the opportunity and/or send them a copy of the solicitation.

26

Potential sources for identifying evaluators:

Professional organizations that have a network of evaluators (e.g., AERA and AEA)

Local colleges and universities

CNCS Office of Research and Evaluation list

Evaluation divisions of state or local agencies

Personal networks

Task 4.2 Solicit Responses to Your Request (cont.)

Facilitator notes:

Potential sources for identifying evaluators include the following:

Organizations that have a network of experienced evaluators such as the American Evaluation Association (AEA). The AEA website (www.eval.org) contains a searchable database of AEA members who are available for evaluation consulting or to serve on evaluation teams due to specific expertise in particular methodologies. You can search for a listing of individuals in your geographic area and/or your desired area of expertise.

Local colleges and universities. When looking for an evaluator in an academic setting, you may start by contacting the department that best aligns with your program’s service area (e.g., social work, sociology, education, or public administration), or see if there’s a research or evaluation center within the college or university. The CNCS Office of Research and Evaluation holds research grant competitions for institutions of higher education, and these universities could serve as local resources. A listing of current and previous grantees can be found at this link: https://nationalservice.gov/impact-our-nation/research-evaluation/cncs-research-grantprogram/grantee-profiles. 

Evaluation divisions of State or local agencies. State or local government agencies may have planning and evaluation departments and individuals from these agencies may be willing to work with you on your evaluation. Additionally, staff members from these divisions may be able to direct you toward other organizations or individuals that are interested in conducting external evaluations. 

Personal networks, including other agencies in your community that have used external evaluators. These agencies may be able to recommend a good evaluator, suggest methods for advertising, and provide other useful information.  

GROUP QUESTION: 

What other methods has your program used to attract applicants?

What are other potential sources for identifying evaluators?

27

Establish concrete criteria on which to assess each applicant. Criteria that may be used include:

Evaluation plan/approach

Evaluator qualifications (Years of experience, capacity, references, etc.)

Communication style

Project management

Cost   

Task 4.3 Create Criteria to Assess and Choose the Most Qualified Evaluator

Facilitator notes: As noted previously, the solicitation establishes the evaluation criteria upon which each applicant will be assessed. This informs applicants of not only the criteria that will be used, but also the relative importance of each criteria. Some criteria that are commonly used to assess proposals include:

Evaluation plan/approach: Does the applicant’s evaluation approach address the purpose, scope, and goals of the evaluation? Be mindful of the questions and issues the applicant believes should be the focus of the evaluation, the type of data to be collected to address those questions and issues, the method of data collection, and the presentation of the evaluation’s results to determine if they are a good fit for the type of evaluation you want done.   

Evaluator qualifications: For example, how many years of experience does the applicant have? Do they have the capacity and access to resources necessary for the evaluation? Does the applicant demonstrate that he/she has the knowledge, experience, and skills that align with the type of evaluation you wish to conduct? Is the applicant familiar with AmeriCorps or similar programs? Do they have references from previous clients?

Communication style: Is the applicant’s writing style and language clear and accessible? Proposals that are indecipherable or written in highly technical language may be an indication of the evaluator’s work products. The proposal should be written in a language that you can understand and present a coherent plan. If requested, check the applicant’s past work samples for clarity, organization, readability, and potential usefulness for decision makers. Applicants providing highly technical, poorly written, and/or disorganized products will likely develop similar products for your evaluation. 

Project management: Does the applicant demonstrate an ability to effectively lead and manage the evaluation project? Does the applicant propose a reasonable timeline and schedule of deliverables that aligns with the timeline specifications in your solicitation?

Cost: Does the applicant’s proposed costs seem reasonable and within your stated budget? Applicants must indicate that their proposed evaluation approach can be carried out for the available funds specified in the solicitation. You may find a proposal to be of high quality, but costs much more than your anticipated budget. In this case, you may want to ask the applicant to provide further explanation of their budget. It may be the case that your program receives multiple proposals with varying ranges in cost. It is important to take a close look at each of the applicant’s budgets to decide which one is most reasonably priced for the work that you want done. It may seem like an easy decision to choose the lowest cost proposal, however, after a closer look at the budget may show that the applicant has not factored in key activities that are in your solicitation. It may be the case that the cost associated with each of the proposals you receive is higher than your budgeted amount. In this case, you may need to rethink the scope of your evaluation to determine if there are any activities or approaches that could be scaled down. You may also consider whether additional funds for the evaluation could be generated. Additionally, some applicants may be willing to donate some of their services (e.g., offer in-kind services) upon request.

28

Develop a method to apply your criteria

Scoring/rating scale

Checklist

Weighting

Task 4.3 Create Criteria to Assess and Choose the Most Qualified Evaluator (cont.)

Facilitator notes:

Once you have established the criteria on which to assess each applicant, it is important to specify the relative importance of each criteria in the solicitation. Depending on program needs and resources, each criteria may have equal value (20% each in this case) or some criteria may have greater weight (such as the evaluation plan and evaluator qualifications) than others (communication, project management, and cost). 

For internal purposes (not to be shared in the solicitation), you will need to develop a method or tool to apply your criteria. The review form should parallel the specification and criteria you lay out in your solicitation and may be a scoring or rating scale or checklist. The issue of whether a weight should be assigned to each criteria also comes into play here. For example, one option is to assign a numerical value to each criterion, which reflect its relative importance in comparison with other criteria. Agreeing on standards for rating the applicants provides a consistent measurement strategy and promotes an objective evaluation of proposals. Once the scoring method is established, the task of reviewing the proposals can begin. 

In the handouts you will find a Sample Evaluator Assessment Form to help serve as a guide for developing and rating evaluation criteria. The assessment form is organized around the five criteria discussed here, along with more detailed criteria under each heading. For this example, we use a typical five-level Likert item to measure the extent to which the reviewer strongly agrees (“5”) or strongly disagrees (“1”) that the applicant meets the specified criteria. Scores for individual criteria can then be summed to create a total score upon which to evaluate and compare your applicants. 

29

Evaluate the proposals using the assessment criteria. 

Choose the highest quality proposal that matches your criteria.

If there is more than one high quality proposal, consider gathering additional information on top applicants (if allowed):

Ask them to respond to questions 

Ask them to participate in a “best and final” meeting

Conduct interviews

Check references

Make a final decision about who to hire

Task 4.4 Evaluate the Proposals and Select an Evaluator

Facilitator notes: As you receive proposals in response to your solicitation, the next step is to conduct a critical review of this information and then make a decision about who your program wants to ultimately hire as your evaluator. It is recommended that multiple individuals (not just the evaluation manager) review the proposals using the criteria established. Multiple scores should either be averaged or compared and discussed during the selection process. Including an outside consultant with strong methodological expertise on the review team may help provide an outside point of view on the quality of the evaluation proposals. 

After all proposals have been reviewed and critiqued, it is time to choose the proposal that best meets your criteria. There may be a few proposals that are close in quality, so you may consider gathering additional information on your top applicants that will help the review team reach a final choice. In this case, the following activities may be helpful; however, these are all optional: 

Ask your top applicants to respond to questions or participate in a “best and final” meeting to ensure that all bidders are given the same opportunity and to ensure an objective selection process.

Conduct interviews to get a better a sense of the evaluator’s approach, style, and personality which should help you determine whether you and the evaluator are compatible. The “right” evaluator on paper may turn out to be difficult to communicate with or may have a work style that differs from your own. Use the proposal as the beginning of a conversation about the nature and scope of the evaluation project. Ask about anything in the proposal that you did not understand or need clarification on. An interview is also a good opportunity to ask about the applicant’s capacity to do the work within your timeline. 

Contact the applicant’s references to find out more about the evaluator’s working style, approach to evaluation, and any issues they might have encountered in working with the evaluator. These references should be contacted for objective views of the candidates. Some questions you might ask their references include: 

Did the evaluation approach used by the evaluator address the needs and desires of your organization? 

Was the evaluation conducted in a timely fashion? 

How was the evaluator’s quality of work?

Was the evaluation conducted within your budget? 

Was the evaluation report useful to your organization? 

Were there any major challenges in conducting the evaluation and how did the evaluator address them?

Would you hire the evaluator to conduct another evaluation for you?

Note that some public agencies have procurement processes that do not allow for interviews or consideration of information beyond what’s contained in the proposal so it is important to check that these activities are permitted before engaging in them. 

At this point, after carefully considering each proposal, you should have enough information to make a final decision about who to hire.   

30

Standard elements of a contract:

Scope of work

Payment/invoicing

Point of contact for both parties

Product ownership and rights

Other special terms or conditions (e.g., modifications or termination of contract)

Task 5: Establish a Contract or Consulting Agreement

Facilitator notes: Once you have selected an evaluator, the next step is to draw up a contract or a formal consulting agreement for the proposed work. The contract will provide you with the authority to manage your external evaluation. It should clearly state each requirement and how it will be monitored. As you work to establish a contract, be sure to clarify expectations and responsibilities with the evaluator to ensure there is a mutual understanding about the work. Contracts are legally binding documents and can be challenged in a court of law; therefore, it is best to work with someone with a strong contracts background to ensure your contract meets the standards for contracts of its kind.

A list of the standard elements of an evaluation contract are listed here:

Scope of work which details all of the tasks and responsibilities the evaluator agrees to perform. In general, the scope of work should be as detailed and precise as possible—with full descriptions of the tasks to be executed, expected deliverables, timelines, and also expectations for communication with the evaluator. This ensures that everyone has the same expectations about the evaluator’s work, which will be critical if for some reason the project goes awry.  At the same time, evaluation is not a precise science and modifications in the evaluation approach may become necessary, so there should be enough flexibility in the scope of work to allow for minor mid-course corrections or changes in the evaluation strategy as needed. Note that the scope of work for your contract should largely reflect what was already written for the work plan section of your solicitation/RFP. 

Payment/invoicing - Your contract should also include details regarding the total contract value (i.e., the amount that will be paid to the evaluator), how the evaluator will bill for services rendered, and a payment schedule. Linking payments to major deliverables such as the evaluation plan, an interim report, final report, etc., can be a good strategy for monitoring quality and progress on the evaluation. This will help ensure timeliness and quality of key deliverables as it provides leverage if your evaluation is falling short of expectations. 

Point of contact for both parties – It should be made clear in the contract the one person from the program side who is the main point of contact for the evaluation. This individual should be responsible for communicating with the evaluator and monitoring the contract and the evaluator’s work. Likewise, the contract should specify the one individual who will be the main point of contact on the evaluator/consultant side. This person should be responsible for communicating with program staff about the evaluation progress and directing the evaluation work. 

Product ownership and rights – Your contract may address questions about who owns products that result from the evaluation (e.g., data gathered during the study, analysis files, final report) and who has rights to publish the evaluation’s results. Your program may want to retain all rights to the data and to publishing results, so it is important to include language in your contract that specifies these terms. 

Any special terms or conditions (modifications or termination of contract) – This involves language describing how changes in the scope of work or a termination of the contract will be handled in the event they arise. 

31

Communicate

Monitor

Support

Task 6: Manage the Evaluator – Communicate, Monitor, and Support

Facilitator notes: After the contract is signed, the final, ongoing task for the program will be to manage the external evaluation itself. This involves communicating with the evaluator and monitoring and supporting the evaluator’s work throughout the evaluation. 

While the contract’s scope of work will guide the evaluator’s activities, program staff’s involvement in the implementation of the evaluation is important for ensuring that the evaluation remains on track and that the program’s evaluation needs are met. Leaving an evaluator to work on his or her own risks producing an evaluation that does not demonstrate a clear understanding of your program, uses an inappropriate evaluation design, only partially addresses program needs, creates unexpected costs, and/or does not meet funder requirements, among others. A successful evaluation depends on closely and effectively overseeing the evaluation process. This means maintaining open and regular communication between program staff and the evaluator throughout the project, continually monitoring progress on the evaluation and the evaluator’s performance, and ensuring that the evaluator receives the guidance, information, and resources necessary to support the evaluation. We discuss each of these activities in more detail on the following slides.

32

Maintain communication throughout the project

Project kick-off meeting

Discussion of tasks outlined in the contract

Details about upcoming deliverables

Project timeline

Questions or challenges to the work

Regular, ongoing meetings to keep the evaluation moving in a timely and efficient manner 

Ad hoc meetings to discuss specific topics

e.g., Debrief on preliminary findings

Task 6: Manage The Evaluator – Communicate

Facilitator notes: Communication between the program staff and the evaluator is critical to the success of the evaluation, beginning with the kick-off meeting between key staff from both parties. The kick-off meeting serves to clarify expectations and discuss details about the following: 

Each of the tasks outlined in the contract;

Upcoming deliverables; 

The project timeline; 

Any potential question or challenges to the work

If your evaluator is responsible for developing a written evaluation plan, the kick-off meeting can be an important opportunity to collaborate and/or finalize details about the overall evaluation design. The evaluation design is essentially the backbone of an evaluation, so you want to be sure that you and your evaluator fully agree on this before they develop and finalize the evaluation plan. 

Following the kick-off meeting, subsequent meetings to discuss the evaluation’s progress, and any challenges and questions that arise should be regularly scheduled to keep the evaluation moving in a timely and efficient manner. The meeting schedule should strike a balance between the amount of contact you need with the evaluator to feel informed about their work and the number of meetings that are supported by the budget. Agendas (either created by program staff or the evaluator) are helpful in making sure the meetings are productive and have specific topics and goals, whether related to planning and implementing the evaluation, discussing the progress of the evaluation, troubleshooting issues that arise, or discussing the data or early findings. 

Additional meetings between program staff and the evaluator should be expected on an ad hoc basis to troubleshoot challenges that may arise, provide input on the evaluator’s approach to a particular activity, debrief on the evaluation findings (preliminary and final results), etc. For example, once data have been collected and analyzed, you may ask your evaluator for a debrief meeting to discuss preliminary findings. The debrief will provide you with an opportunity to discuss and provide feedback on the evaluation’s conclusions and implications for your program. This will be important for informing the content of the evaluation’s final report. 

Note that if you have difficulty communicating with the evaluator (e.g., ignoring e-mails and/or phone calls), this may be a sign that the evaluator is over-committed and is unable to keep up with his or her present tasks. As soon as you notice a pattern of non-response or difficulty understanding the evaluator’s approach to the evaluation, it is important to confront this issue directly. Your evaluator should give your evaluation the full attention you’ve contracted for and be responsive to your program needs. 

[Time check: ~50 min.]

33

Continually monitor progress on the evaluation and the evaluator’s work:

Review and provide feedback on deliverables

Enforce the schedule for completing tasks and adjust as needed

Assess the evaluator’s skills and performance throughout the evaluation 

Ask program staff about their interactions with the evaluator

Find opportunities to observe the evaluator, while on-site or in meetings

Keep up with invoicing/payments

Task 6: Manage the Evaluator – Monitor

Facilitator notes: In addition to communicating with the evaluator throughout the project, it is important to continually monitor progress on the evaluation and the work of the evaluator. This not only ensures that evaluation needs are being met but that products delivered are of high quality. This involves: 

Reviewing and providing feedback on deliverables: Reviewing multiple drafts of a project’s deliverables is important for ensuring that the evaluation is progressing as planned and that the work meets expected standards of quality. As noted earlier, it is best to include language in the evaluator’s contract which allows program staff to review major deliverables and request modifications if they are substandard.

Enforcing the schedule for completing tasks and adjust as needed: In addition to monitoring the progress and quality of deliverables, it is also important to ensure that deliverables are completed on time. This includes following up with the evaluator when deadlines for deliverables are missed or repeatedly late. Reliability is crucial. If your evaluator seems to lack the time, skill, commitment, or interest to keep up with the project schedule and deadlines, this is a sign that their work might be going off course. As soon as you notice that your evaluator is falling behind schedule, it is important to confront this issue directly, so that you and the evaluator can work together to get the evaluation back on schedule.  

Assessing the evaluator’s skills and performance throughout the evaluation: It is important that you continue to assess your evaluator's skills and performance in addition to how they perform on deliverables. This may entail asking program staff about their interactions with the evaluator and finding opportunities to observe the evaluator, while on site or in meetings. For example, you may want to observe the evaluator as they visit program sites to collect data to determine if they are properly prepared and organized, respectful of program staff and others, doing everything that they said they would and in a manner with which you are comfortable, etc. If you find problems, address them with your evaluator immediately and help develop a plan for remedying them. 

Keeping up with invoicing/payments: Throughout the evaluation, evaluators should submit invoices as scheduled, and your program should make payments on time. You will also want to check to see if the correct amount has been invoiced and that invoices are submitted in the appropriate format for your accounting or finance department.  

As the success of an evaluation is largely a reflection of the quality of the deliverables produced, we provide more information about monitoring common deliverables on the next slide.

34

Evaluation design plan

Written document that should detail all evaluation steps and activities the evaluator plans to conduct

Instruments

Surveys or assessment tools to be used to obtain data

Monthly or quarterly progress reports 

Reports that provide a status update on the evaluation activities

Interim/final reports

Request to review an outline of the report prior to writing

Program staff may contribute to sections of report (e.g., background)

Other deliverables

Short memos and/or research briefs

Task 6: Manage the Evaluator – Monitor Deliverables

Facilitator notes: As noted previously, reviewing and providing feedback on the deliverables produced by the evaluator is important for ensuring that the evaluation is progressing as planned and that the work meets expected standards of quality. Common deliverables for an evaluation include the following:

Evaluation Design Plan. The evaluation design plan is often the first deliverable that an evaluator submits for the project. As the road map for your evaluation, it is important that program staff and the evaluator collaborate on completing the plan. The project kick-off meeting will serve as an opportunity to ensure that all parties have a common understanding of the evaluation and subsequent follow up meetings will ensure that all aspects of the evaluation plan are addressed. The evaluation plan may need to be renegotiated during your collaboration as new information is revealed or as initial strategies prove challenging. For example, initial assumptions about the program’s ability to recruit individuals for study participation may not be realized and modifications may need to be made to the sample size calculations and/or analysis plan. However, the evaluation design plan should not be in constant flux. If you notice that the evaluation plan is changing often, you should discuss this with the evaluator.

Instruments. Some evaluations may require developing instruments, such as surveys or assessment tools, that will be used to obtain data from respondents. Because program staff are most knowledgeable about how their program works, the beneficiaries they serve, as well as any unique or specific terms used by the program, reviewing instruments developed by the evaluator will ensure that evaluation tools are relevant, reliable and understandable to program participants and beneficiaries. 

Monthly/Quarterly Progress Reports. In addition to participating in regular, scheduled meetings by phone or in-person, you may also request that evaluators submit progress reports on a monthly or quarterly basis. Routinely monitoring the evaluation through these deliverables keeps program staff informed about how the evaluation is progressing as well as help anticipate and quickly address any challenges that might be encountered. 

Interim/Final Reports. Before the evaluator drafts an interim or final report, it is important to request an outline to ensure that all parties agree on the report’s content and format. Reviewing the outline will also provide an opportunity to ensure that key topics are addressed and that the report is useful for program decision makers and stakeholders. Once the review has been completed, it is recommended that you request comments be incorporated into the report and/or hold a conference call with your evaluator to clarify any sections that may be unclear, misplaced, or confusing. In addition, programs should consider contributing to the development of sections of the report where they are more knowledgeable of the topic, such as the program background section. 

Other Deliverables. In addition to major deliverables, program staff may want to consider requesting some smaller interim deliverables from their external evaluator. Memos written by the evaluator on specific topics can provide additional forms of communication on the study. For example, staff may request a memo summarizing research design options prior to requesting the evaluator to draft a full evaluation design plan. Also, staff may request a written memo summarizing the major study findings prior to receiving the detailed final report to provide documentation and begin preparations for dissemination of findings. In addition to a final report, staff may wish to request that the evaluator provide a shortened summary of the study findings written for a wider audience, such as a research brief. This type of document can be particularly useful for the dissemination of study findings. 

35

Provide support and feedback as needed

Offer advice and guidance to help troubleshoot issues, as needed

Ensure the evaluator has access to the information required

Provide continuous input and feedback on the evaluator’s work

Task 6: Manage the Evaluator – Support

Facilitator notes: Lastly, it is important to provide your evaluator with the organizational support and feedback needed to facilitate the evaluation process. This includes:

Offering advice and guidance to help troubleshoot issues, as needed. 

Ensuring the evaluator has all of the programmatic information, background materials, and resources that you had agreed to provide. Check in with the evaluator periodically to make sure they have all of the resources needed to effectively conduct the evaluation of your program.

Providing continuous input and feedback on the evaluator’s work, including key deliverables.

It is important to keep in mind that it is in your best interest to facilitate the completion of your evaluator’s work. 

36

Evaluation of the program requires skills or expertise outside the original plan

The evaluator does not understand your program and/or does not make efforts to include your input/feedback 

Evaluation has limited time for analysis and reporting in Year 3

The evaluator leaves, terminates the contract, or does not meet contractual requirements

You are not satisfied with the evaluator’s findings

The evaluator changes the scope of the evaluation midcourse without approval to make this change

Serious and persistent differences in evaluation approach

Potential Challenges and Strategies 

Facilitator notes: Despite your best efforts to build a relationship with your evaluator and effectively manage the overall process, problems may arise. Listed here are a few potential problems that you may encounter and possible strategies for overcoming them.

Evaluation of the program requires skills or expertise beyond what the evaluator can provide. You may find that your evaluator agrees with your assessment and is willing to add another person to the evaluation team who has expertise and skills needed to undertake additional or different methods/approaches. Many times additional expertise can be added to the evaluation team by using a few hours of a consultant's time. For example, programmers, statisticians, and the like can augment the evaluation team without fundamentally changing the evaluation team's structure. 

The evaluator does not understand your program and/or does not make efforts to include your input/feedback. Ideally, your relationship with your external evaluator should be a partnership where you are providing critical information on your program and important input into the evaluation design and its implementation. While your external evaluator is responsible for leading the evaluation of your program, they also should be requesting clarification and feedback from your program staff throughout the process. For example, in the initial stages of an evaluation, the external evaluator should be spending significant time familiarizing him/herself with your program’s goals, logic model, processes, and existing data. If at a later point in the evaluation process, you observe that the evaluator still does not seem to understand your program’s theory of change, operations, evaluation requirements, etc., then it may be necessary to confront the evaluator about the amount of attention and time they are spending on your evaluation. Ultimately, a successful evaluation should be a partnership between the evaluator and the program. If at any point, you do not feel that the evaluation is a collaborative process and/or that your input and feedback is not welcome, then your concerns should be presented to the evaluator as soon as possible. 

The evaluation has a limited time for analysis and reporting in Year 3.  As discussed earlier, the third year of your grant cycle is meant to be used to complete the evaluation’s analysis and final report in time to be submitted with your AmeriCorps State and National re-compete application. For some grantees, applications are due early in year 3, reducing the amount of time available to write the evaluation report. It is recommended that you discuss with your evaluator this important time constraint on the evaluation as early as possible. The evaluator should be able to work with you to plan the production of the final report so you have it for your application submission.

The evaluator leaves, terminates the contract, or does not meet contractual requirements. If the evaluator leaves the area or terminates the contract, you will most likely be faced with recruiting a new one. In some instances, programs have successfully maintained their ties to evaluators who have left the area, but it is important to consider how this affects your evaluation and whether it will jeopardize any of the work. When your evaluator does not meet contractual requirements and efforts to resolve the dispute have failed, public agencies should turn the case over to their procurement office and private agencies should seek legal counsel. 

You are not happy with the evaluator's findings. Sometimes program managers and staff discover that the evaluator's findings are not consistent with prior research on their program or their own impressions of the program's effectiveness. In this situation, you may want to communicate your concerns with the evaluator and work together to understand the reasons for these findings. You may want to review aspects of the evaluation approach to confirm their reliability, such as the accuracy of data collection instruments, adequacy of sample sizes, and the quality of the data collected. You also may want to request that your evaluator further examine your program participants to determine whether there are any differences between those that are or are not benefiting from the program and why. However, be prepared to accept findings that may not support your perceptions. Not every program will work the way it was intended to and you may need to consider making program changes based on your findings. 

The evaluator changes the scope of the evaluation midcourse without gaining approval from program staff to make this change. Program staff should make it clear to the evaluator that the scope of the evaluation cannot be changed without discussion and written approval from you. For example, if you discover that the evaluator added a program site or conducted focus groups that were not part of the original evaluation plan, you must address this immediately. Changing the scope of the evaluation midcourse not only quickly expends already limited resources, but can jeopardize the validity of your findings. 

Program staff and the evaluator are unable to agree on the evaluation approach. As noted earlier, the evaluation should be a collaborative process between program staff and the external evaluator. Therefore, key features and activities of the evaluation should be regularly discussed and agreed upon by the two parties. If severe differences continue and many reasonable attempts to resolve them have been tried, this situation could jeopardize the quality of the evaluation. In this extreme situations, program staff may consider terminating the evaluation contract. However, this decision should be weighed carefully and discussed with your funder, as a new evaluator will need to be recruited and brought up to speed midcourse. 

The earlier you pick up on signs that there may be problems and confront the evaluator directly, the more likely you and the evaluator can work together to get things back on track. Nevertheless, if a poor relationship with a consultant seems intractable, or if the products are considerably deficient or delinquent, you may want to consider terminating the contract. 

GROUP QUESTION: 

What challenge(s) have you encountered in working with an external evaluator? How did you resolve them?

37

Read the example scenario 

Discuss the following questions in small groups:

What could the evaluator have done differently?

How could the grantee have effectively communicated, monitored, and supported the evaluation to avoid this outcome? 

What could the grantee have done differently during the planning process?

Group Exercise

Facilitator notes: Now that you are familiar with the main tasks involved in managing an evaluation, we would like to reinforce these ideas through a group exercise. You may gather in small groups or work individually if you choose. Read the vignette or example scenario in your handout that describes a hypothetical situation involving the evaluation of an AmeriCorps program. This brief story illustrates how the extent of a program’s involvement in and management of their evaluation can affect whether program evaluation needs are met. 

After your group has finished reading the example scenario, several questions are presented for your group to consider. Afterwards, we will open up the discussion to all the groups. 

[Read below after all groups have had a chance to discuss the questions.]

Let’s discuss together some of the questions listed in your handout.

What could the evaluator have done differently? 

Facilitator probe:	Did the evaluation team do everything they could to identify a comparison group?

How could the grantee have effectively communicated, monitored, and supported the evaluation to avoid this outcome? 

Facilitator probe:	What management practices should have been used?

What could the grantee have done differently during the planning process?

Facilitator probes:	What qualifications should the person overseeing the evaluation have had? 

			How could AmeriCorps ABC have supported the program coordinator?

			What other approaches could AmeriCorps ABC have used to identify an external evaluator? 

38

Start the process early

Build in time for unexpected delays to avoid potentially missing critical deadlines.

Do your homework

The evaluation manager should have a basic understanding of evaluation concepts and methods to ensure that he/she can effectively communicate your program’s evaluation needs to the evaluator.

Ensure that the evaluator has the appropriate expertise, i.e., if proposing a QED, they should have prior experience implementing that design

Important Points to Remember: 

Facilitator notes: A few points are important to remember when managing your external evaluation:

Start the process early. Previously, we discussed that delays are not uncommon in an evaluation and that additional time needs to be factored into your schedule. In addition to the evaluation itself, the time period for developing a solicitation and selecting an external evaluator may take longer than anticipated. For example, some programs may be required to engage in a formal competitive bidding process which involves additional staff time to undertake. Starting the process as early as possible will allow for some flexibility in the evaluation schedule so that you avoid missing critical deadlines in case certain activities end up taking more time than expected.

Do your homework. While your external evaluator will have the experience and expertise to conduct your evaluation, it is important that you also familiarize yourself with evaluation concepts and tools to help you effectively communicate your evaluation needs to your evaluator. It is also important that the evaluator you hire has the appropriate experience. That is, if you plan to conduct a QED, an impact design, the evaluator should have experience with that or other rigorous designs. If they are only familiar with process evaluations or qualitative research, then they may not be the right fit for your evaluation. In addition to consulting evaluation resources broadly, there are a number of evaluation resources specific to AmeriCorps that are available on the Resource Page on Nationalservice.gov/resources. These AmeriCorps-specific resources are important to review prior to and throughout the evaluation process. 

39

Ask questions

Make sure to ask questions if you are unclear about the evaluation approach or activities your evaluator is undertaking. 

Collaborate often

An evaluation is best served when both program staff and the evaluator are well-informed and working together.

Important Points to Remember (cont.): 

Facilitator notes: Other points to consider are to:

Ask questions. Do not hesitate to ask questions if you are unclear about the evaluation approach or activities your evaluator is undertaking. An evaluator is unlikely to know all of the details of your program and how specific program constraints may affect evaluation activities. Likewise, program staff may not have adequate knowledge of the technical details involved in evaluation. Asking questions and soliciting feedback from your evaluator as the study progresses will help avoid confusion, duplication of efforts, and unfounded assumptions and ensure that tasks are completed in a timely fashion.  

Collaborate often. An evaluation is best served when both the program staff and the evaluator are well-informed and working together on its design and implementation. 

40

Please visit the Evaluation Resources Page for more information on the following courses: 

How to Write an Evaluation Plan

How to Develop a Program Logic Model

Overview of Evaluation Designs

Budgeting for Evaluation

https://americorps.gov/grantees-sponsors/evaluation-resources

Hiring the Right Evaluator for your National Service Program:https://www.americorps.gov/sites/default/files/document/2015_03_19_ManaginganExternalEvaluationHiringtheRightEvaluator_ORE.pdf

Frequently Asked Questions: AmeriCorps State and National Evaluations:

https://www.americorps.gov/sites/default/files/document/2019_10_01_AmeriCorps_State_and_National_Evaluation_FAQs_ASN.pdf

Resources on Evaluation

Facilitator notes: For more information on evaluation, please go to the National Service Resource Page on Nationalservice.gov/resources located here. Also, additional resources are shared. 

[The FAQ pdf link may change once/if a new FAQ document is developed]

41

AmeriCorps State and National grantees with evaluation requirements may request assistance with identifying an appropriate evaluator using the AmeriCorps’ Technical Assistance (TA) Portal: https://americorpsevaluationta.norc.org/

Resources on Evaluation

Facilitator notes: For more information on evaluation, please go to the National Service Resource Page on Nationalservice.gov/resources located here. 

42

Questions?

Facilitator notes: Does anyone have any questions?

43

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

44

# image2.emf

# image5.png

# image9.jpeg

# image10.jpeg

# image3.png

# image4.emf

# image1.png

# image6.png